No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

A big part of where Microsoft Azure is going is being driven by template-defined environments that leverage the Azure Resource Manager (ARM) for deployment orchestration.

If you’ve spent any time working with ARM deployments you will have gotten used to seeing this pattern in your templates when deploying Virtual Machines (VMs):

The adminPassword property accepts a Secure String object which contains an encrypted string that is passed to the VM provisioning engine in Azure and is used to set the login password. You provide the clear text version of the password either as a command-line parameter, or via a parameters file.

The obvious problems with this way of doing things are:

  1. Someone needs to type the cleartext password which means:
    1. it needs to be known to anyone who provisions the environment and
    2. how do I feed it into an automated environment deployment?
  2. If I store the password in a parameter file (to get around 1.B) I risk leaking the password should someone access the file (typically from source control).

Thankfully, Microsoft now provides a way to help solve these challenges.

Say hello to Azure Key Vault

Key Vault can store two types of information: Keys (Certificates) and Secrets.

In our scenario we are interested in the storing of Secrets which fit perfectly with our Secure String requirements for passwords.

Firstly I’m going to provision a Key Vault instance – note that this doesn’t need to be in the same Region or Resource Group as your intended deployment. In fact, I’d argue that you wouldn’t provision your Vault in the same Region (or Resource Group) as your deployments to restrict access to the Key Vault as much as possible.

Once the Key Vault is provisioned we are ready to push our passwords into it as Secrets.

I think its worth noting that a Secret has a lifecycle – you can update a Secret once created (i.e. publish an updated password to the same Secret) which means you can request specific versions of Secrets over time. This is beneficial to our scenario because it allows us to support multiple environments at different stages of their lifecycle.

For example: need an environment from two years ago? No issue – just request it be provisioned with the password (Secret) that was valid at that point in time.

Let’s go ahead and add a password to our Vault.

I could, additionally, specify a validity date range for this Secret using the NotBefore and Expires arguments. For this blog we’ll just leave it to be valid until updated with a new value.

The above PowerShell will return the following output from which we can grab the ‘ID’ of the Secret which we will use later.

Vault Name   : provisionvault
Name         : LocalAdminPass
Version      : 2c585ddc60f54d048767924d0125ede5
Id           : https://provisionvault.vault.azure.net:443/secrets/LocalAdminPass/2c585ddc60f54d048767924d0125ede5
Enabled      : True
Expires      :
Not Before   :
Created      : 23/11/2015 01:58:26
Updated      : 23/11/2015 01:58:26
Content Type :
Tags         :

Using in ARM templates

Now we have our password stored as a Secret in a Key Vault we can go ahead and use it in an ARM template.

At present the way we supply Key Vault Secrets to ARM templates is via use of parameters file. If we use our original snippet for VM creation (shown below)

we can feed in the Secret we pushed to our Key Vault using the below snippet in our parameters file.

In order to deploy this template we must use the Azure CLI tools with the following command:

azure group deployment create -n {deploymentname} -g {resourcegrop} -f {template}.json -e {template-parameters}.json

Once the deployment is finished you will find that the provisioned VM(s) are now accessible with the password that had been securely held in the Key Vault!

Where can I use this?

I’ve tested in a range of scenarios from VM provisioning (Windows and Linux) through to Azure SQL Database Server setup, and they all support this technique.

I did find at least one scenario that didn’t work… which I’ll cover next.

Limitations

You probably spotted that I switched from PowerShell earlier in the blog to do the final deployment with the Azure CLI. At time of writing the PowerShell Cmdlets (v1.0.0 / 1.0.1) don’t support the feeding of a Key Vault reference to an ARM template so for your final deployment step (or all of your steps if you want) you need to use the CLI.

Additionally, some existing ARM deployment scenarios such as “deploy AD-domain joined VMs” that leverage extensions like the DSC one, don’t appear to support the feeding of Key Vault secrets into them (and a result fail). I’m still troubleshooting this one so if I find a resolution I’ll post an update here.

Finally, I also find the supplying of the Key Vault and Secret details in the parameters file a little restrictive. I tried changing the actual template to reference the Key Vault but it’s not an approach that is supported and your deployment fails.

Summary

Having been working with Azure for quite some time it’s interesting to see the transition of the platform to the new ARM-based model (also known as the ‘v2’ platform) and the possibilities it opens up such as the one shown here.

If you’re doing a lot of template-based deployments that include passwords stored in parameter files (or other insecure stores) then this approach should provide you with a lot of value (and comfort). Also, if as an IT Pro you are concerned about locking down environments so that only the right groups of users can login and modify configurations, this approach should be high on your list.

As always, leave any comments or questions below.

Simon.

Tagged , ,

36 thoughts on “No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

  1. Useful summary Simon and a great approach for deployment-time secrets.

    I’m grappling with another problem with RM templates which relates to provisioning-time secrets – I need to to supply my provisioning script (either custom script or DSC) with a client secret so it can get a token from WAAD to enable that provisioning script to authenticate to KeyVault to access secrets such as software licence keys.

    Would love to hear if you had come across a solution to that one!

    • Simon says:

      Julian, it sounds like you want to use a service principal that can obtain / manage secrets, which would be (if I understand it correctly) similar to how Azure Automation is updating entries in a Key Vault in this post from Dushyant Gill: http://www.dushyantgill.com/blog/2015/04/26/say-goodbye-to-key-management-manage-access-to-azure-storage-data-using-azure-ad/. Take a look and let me know how you get on.

      • Thanks for the link Simon.

        That’s broadly the approach, but it takes advantage of the azure automation variable storage to hide the service principal credentials. Thinking about the provisoning script running at first boot I still find myself in an “infinite regression of secrets”!

        The provisioning script (which might be customscript or dsc) needs the clientId and clientSecret of the service principal so it can authenticate to WAAD to get the token for KeyVault.

        The obvious way to inject these at runtime would be through environment variables, but I haven’t yet found how to pre-poulate environment variables on an Azure VM using either legacy or resource-manager approaches – this may of course be a failure of my googling!

  2. Brian says:

    Simon – when you create a keyVault for use this way the vault needs to have the EnabledForTemplateDeployment property set to true… You do *not* need to set the EnabledForDeployment property (line 6 of your provision-vault.ps1)

  3. Ana says:

    is it necessary to have an application to access secrets? Or is it possible to retrieve the secret value using a powershell script.?

  4. Robert Davies says:

    Hi Simon

    Thanks for this. It’s a shame it only works when providing the key vault secret details in the parameters file. I tried it in the template and as you say it fails. It’s a shame as I have an RM template to deploy some cloud services and associated certificate and was hoping to get the private key password from the key vault using this method. Be good to know if things change.

    Thanks anyway.

    Robert

    • Simon says:

      Thanks for the feedback Robert – I’ll be sure to update this post if / when this method of deployment changes to support this being embedded in the actual template itself and not simply in the parameters file.

    • Brian says:

      Robert – why doesn’t putting the ref in the param file work for you? It should work [technically] for the scenario you described…

      • Robert Davies says:

        Hi Brian, it’s because I need to pass some dynamically generated parameters (based on user selections) to provide the cloud service names, location and which certificate to use. As far as I’m aware, it is not possible to provide some parameters to New-AzureRMResourceGroupDeployment as a parameters object and others (i.e. the keyvault secret) via a parameters file, it’s either one or the other. Unless you know different?

      • Brian says:

        You can override individual parameters on the cmd line by simply providing them as a param to the deployment cmdlets. I don’t know if that’s [well] documented but that’s one option. The other would be creating a param object on the file from the file, a bit more involved but might fit your workflow better. If you need more detail, lmk we can try to work through it via email. (bmoore @ microsoft.com)

  5. Robert Davies says:

    Hi Brian, thanks for that. I will take another look tomorrow (UK time) and perhaps ping you if I get stuck.

  6. Hey Everyone,

    Is it normal to not see a resource in the resource group that has been created through an ARM template after you’ve created the key vault?

    • Simon says:

      Kyle – KeyVaults don’t yet show up in the portal. Your question is a little vague – are you suggesting a new resource isn’t showing in the right resource group or that it isn’t showing at all? Could you clarify a little?

      • Hello Simon,

        Sorry about that. What i am saying is i have created a resource group, and used an arm template to create a key vault resource within that resource group. But when i look at the resource that the key vault should be in i cannot see a key vault resource. From what you said in your reply the key vaults do not show up yet in the portal. I am assuming that is coming though.

        Thank you very much 🙂

      • Simon says:

        Correct – the KeyVault won’t show up at this stage (but it is there!)

  7. Hi Simon,

    We are managing the deploy time keys and secrets using the vaults. But once it is provisioned, to login to the server, we need to store the keys locally or secrets should be known to others right? In this case the utmost purpose of storing keys securely in vault and referencing in provisioning is defeated. Is there a nice way of handling this scenario?

    • Simon says:

      Hi Satheeshkumar – your scenario is pretty common and there’s a few ways to look at it:

      • Use keyvault for AD Domain-level admins, but use local Admins to grant Windows server access. This way your Domain Admin password isn’t shared.
      • Use cert-based RDP – while you need to share the certificate you can easily block access by rotating the cert in use. This applies to SSH as well.

      The ideal use case I see is for hands-off environment provisioning where you want non-Admin users to be able to create an environment that they don’t necessarily need admin-level access to.

      HTH.

  8. BB says:

    Hello Simon,

    i am using keyvault to assign password to my VM. here i use 2 files (Template & parameter) to make it work. now i am using a tool to deploy Arm template in Azure but here tool will allow to use only onefile. now i need this template and parameter to be used from single file. is it possible to make keyvault secret called from single template if possible can you help or guide me on getting it to work.

    • Simon says:

      Hi BB – at this stage the only way to provide a KeyVault secret reference is from a parameters file.

    • Brian says:

      BB – that seems like an odd restriction from the tool given the whole nature of using parameters. Does that tool allow you to pass in a json blob for parameters? You could supply the reference that way…

  9. Oliver says:

    Hey Simon,

    Do you happen to know if it’s possible to pass credentials from a vault into a DSC extension? For instance, I have a domain admin password that I need to feed into a DSC resource within my template that creates a second domain controller – now, that keyvault secret is defined in my parameters file, but it seems like it never gets to the DSC resource because it fails every time when looking for the domain, something I would expect if it never received domain credentials.

    Thanks!

  10. Oliver says:

    Hey thanks for your reply Simon, but I do have a DSC resource on my second DC (dc02); it’s under resources and starts at line 514. I have that resource depending on dc02 being up (which itself depends on having a nic, etc) and also depending on the domain join of server1, which is what you are talking about in point 2, using the non-DSC domain join. So to summarize, I have one domain controller coming up to create the domain, a generic windows server that joins the domain using the json domain join, and I am having issues with passing credentials into DC02.

    I have merged this branch back into another one, so I am leaving the branch “seconddc” alone in case you want to take a look – right now in that branch I have the credentials for the DSC resource for DC02 explicitly hard coded in plain text (lines 541, 542) and as-is this works. It’s when I use a parameter for username and password that it doesn’t work, even though previously I was referencing the admin name/pass for DC01 (since that’s the account used to bring up the first domain controller and create the domain).

    The lines I am referencing are here: https://github.com/oradcliffe/AzureRm-Windows-Domain/blob/seconddc/WindowsVirtualMachine.json

    Thanks!

    • Simon says:

      Oliver, I had a play (without using KeyVault) and can get most of the script to work – the piece that fails looks like it is due to the way you are using the xActiveDirectory Module in your CreateSecondDC.ps1 script to try and add the DC to an existing Domain.

      I’m not a DSC guru, and certainly less so in the Active Directory space, but there is an example script that shows how you go about creating a HA setup for AD that might be useful: https://github.com/PowerShell/xActiveDirectory

      I made a couple of minor tweaks to your template – you will find a PR with them pending for your branch on Github.

      One tip I’ll give you about troubleshooting this: don’t delete the Resource Group or even your VM. Make your DSC changes, upload them to Github at the asset URL you have in your template and then simply delete the DSC Extension from the VM and run your ARM template again. This will pull down whatever the latest pieces are that you put into your DSC modules.

      HTH.

      • Oliver says:

        Hey, in that example for HA AD, it looks like he’s allowing plaintext passwords in the MOF that gets created, which is something I am trying to avoid; if I hard code passwords this all works for me every time too. I actually have it all working, even pulling from a keyvault now, EXCEPT for in my DC-02 DSC script I am defining username and password outside of the private settings, which I would like to be using.

        Here’s kind of the latest of where I’m at, if you’re interested in all this:
        https://powershell.org/forums/topic/dsc-azure-templates-and-key-vaults/

        As far as your PR, I looked it over and it looks like you’re putting a dependency on the actual VM resources for the DC01 VM resource, which I am not sure I want to do. I am pretty new to this so please stop and correct me if I’m wrong, but if we put those dependencies up, then neither of those VMs will start to provision until DC01 is up, but since it’s just the DC01 VM that we’re waiting for, we don’t have an AD domain yet anyway.

        The way I see the workflow going is the 3 VMs come up, DC01 promotion happens, then Server1 joins the domain, and then I set the dependency on DC02 promotion on Server1 joining the domain, because for sure the domain is up at that point. I’m not sure that the order the VMs themselves come up is important, but please correct me if I am wrong.

        Thanks for your replies! And if you are interested, do check out that link to Powershell.org, I got a script resource in DSC to work and actually spit out some information about the credentials that are being passed in to the script, so that was pretty interesting! It is totally working the way my xActiveDirectory is set up in the second domain controller, but for whatever reason if I use the PrivateSettingsRef for the Admin password, the password gets lost somewhere.

      • Simon says:

        The domain has to be available before you try to join it. In theory DSC should try to correct “skew” in configuration by re-applying itself but that’s not how it works here. I think the dependency actually needs to be on the DSC extension of your primary DC as that’s what you need to succeed before trying to run the others – and that the dependency should actually be defined within the DSC extensions on the other hosts (i.e. the VMs will be created but the DSC won’t run until the primary DC’s DSC has succeeded). It might also be worth splitting this template into two – one for the Primary DC and one for the other VMs.

      • Oliver says:

        Real quick, if my DC02 DSC resource completes and does promote it to a domain controller, if I remove the DSC extension it will still be that, right? So it wouldn’t need to rerun the extension, so it might be easier to de-provision rather than go through a demotion process right?

      • Simon says:

        Correct – it won’t attempt to “uninstall” anything the DSC script configured.

  11. Oliver says:

    Oh I think I see what happened, you forked my master branch, and I haven’t merged the second domain controller piece with that branch yet. The other two branches, seconddc and activedir, are what I am working off of.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: