No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

A big part of where Microsoft Azure is going is being driven by template-defined environments that leverage the Azure Resource Manager (ARM) for deployment orchestration.

If you’ve spent any time working with ARM deployments you will have gotten used to seeing this pattern in your templates when deploying Virtual Machines (VMs):

The adminPassword property accepts a Secure String object which contains an encrypted string that is passed to the VM provisioning engine in Azure and is used to set the login password. You provide the clear text version of the password either as a command-line parameter, or via a parameters file.

The obvious problems with this way of doing things are:

  1. Someone needs to type the cleartext password which means:
    1. it needs to be known to anyone who provisions the environment and
    2. how do I feed it into an automated environment deployment?
  2. If I store the password in a parameter file (to get around 1.B) I risk leaking the password should someone access the file (typically from source control).

Thankfully, Microsoft now provides a way to help solve these challenges.

Say hello to Azure Key Vault

Key Vault can store two types of information: Keys (Certificates) and Secrets.

In our scenario we are interested in the storing of Secrets which fit perfectly with our Secure String requirements for passwords.

Firstly I’m going to provision a Key Vault instance – note that this doesn’t need to be in the same Region or Resource Group as your intended deployment. In fact, I’d argue that you wouldn’t provision your Vault in the same Region (or Resource Group) as your deployments to restrict access to the Key Vault as much as possible.

Once the Key Vault is provisioned we are ready to push our passwords into it as Secrets.

I think its worth noting that a Secret has a lifecycle – you can update a Secret once created (i.e. publish an updated password to the same Secret) which means you can request specific versions of Secrets over time. This is beneficial to our scenario because it allows us to support multiple environments at different stages of their lifecycle.

For example: need an environment from two years ago? No issue – just request it be provisioned with the password (Secret) that was valid at that point in time.

Let’s go ahead and add a password to our Vault.

I could, additionally, specify a validity date range for this Secret using the NotBefore and Expires arguments. For this blog we’ll just leave it to be valid until updated with a new value.

The above PowerShell will return the following output from which we can grab the ‘ID’ of the Secret which we will use later.

Vault Name   : provisionvault
Name         : LocalAdminPass
Version      : 2c585ddc60f54d048767924d0125ede5
Id           : https://provisionvault.vault.azure.net:443/secrets/LocalAdminPass/2c585ddc60f54d048767924d0125ede5
Enabled      : True
Expires      :
Not Before   :
Created      : 23/11/2015 01:58:26
Updated      : 23/11/2015 01:58:26
Content Type :
Tags         :

Using in ARM templates

Now we have our password stored as a Secret in a Key Vault we can go ahead and use it in an ARM template.

At present the way we supply Key Vault Secrets to ARM templates is via use of parameters file. If we use our original snippet for VM creation (shown below)

we can feed in the Secret we pushed to our Key Vault using the below snippet in our parameters file.

In order to deploy this template we must use the Azure CLI tools with the following command:

azure group deployment create -n {deploymentname} -g {resourcegrop} -f {template}.json -e {template-parameters}.json

Once the deployment is finished you will find that the provisioned VM(s) are now accessible with the password that had been securely held in the Key Vault!

Where can I use this?

I’ve tested in a range of scenarios from VM provisioning (Windows and Linux) through to Azure SQL Database Server setup, and they all support this technique.

I did find at least one scenario that didn’t work… which I’ll cover next.

Limitations

You probably spotted that I switched from PowerShell earlier in the blog to do the final deployment with the Azure CLI. At time of writing the PowerShell Cmdlets (v1.0.0 / 1.0.1) don’t support the feeding of a Key Vault reference to an ARM template so for your final deployment step (or all of your steps if you want) you need to use the CLI.

Additionally, some existing ARM deployment scenarios such as “deploy AD-domain joined VMs” that leverage extensions like the DSC one, don’t appear to support the feeding of Key Vault secrets into them (and a result fail). I’m still troubleshooting this one so if I find a resolution I’ll post an update here.

Finally, I also find the supplying of the Key Vault and Secret details in the parameters file a little restrictive. I tried changing the actual template to reference the Key Vault but it’s not an approach that is supported and your deployment fails.

Summary

Having been working with Azure for quite some time it’s interesting to see the transition of the platform to the new ARM-based model (also known as the ‘v2’ platform) and the possibilities it opens up such as the one shown here.

If you’re doing a lot of template-based deployments that include passwords stored in parameter files (or other insecure stores) then this approach should provide you with a lot of value (and comfort). Also, if as an IT Pro you are concerned about locking down environments so that only the right groups of users can login and modify configurations, this approach should be high on your list.

As always, leave any comments or questions below.

Simon.

40 thoughts on “No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

  1. Useful summary Simon and a great approach for deployment-time secrets.

    I’m grappling with another problem with RM templates which relates to provisioning-time secrets – I need to to supply my provisioning script (either custom script or DSC) with a client secret so it can get a token from WAAD to enable that provisioning script to authenticate to KeyVault to access secrets such as software licence keys.

    Would love to hear if you had come across a solution to that one!

      1. Thanks for the link Simon.

        That’s broadly the approach, but it takes advantage of the azure automation variable storage to hide the service principal credentials. Thinking about the provisoning script running at first boot I still find myself in an “infinite regression of secrets”!

        The provisioning script (which might be customscript or dsc) needs the clientId and clientSecret of the service principal so it can authenticate to WAAD to get the token for KeyVault.

        The obvious way to inject these at runtime would be through environment variables, but I haven’t yet found how to pre-poulate environment variables on an Azure VM using either legacy or resource-manager approaches – this may of course be a failure of my googling!

  2. Simon – when you create a keyVault for use this way the vault needs to have the EnabledForTemplateDeployment property set to true… You do *not* need to set the EnabledForDeployment property (line 6 of your provision-vault.ps1)

  3. is it necessary to have an application to access secrets? Or is it possible to retrieve the secret value using a powershell script.?

  4. Hi Simon

    Thanks for this. It’s a shame it only works when providing the key vault secret details in the parameters file. I tried it in the template and as you say it fails. It’s a shame as I have an RM template to deploy some cloud services and associated certificate and was hoping to get the private key password from the key vault using this method. Be good to know if things change.

    Thanks anyway.

    Robert

    1. Thanks for the feedback Robert – I’ll be sure to update this post if / when this method of deployment changes to support this being embedded in the actual template itself and not simply in the parameters file.

    2. Robert – why doesn’t putting the ref in the param file work for you? It should work [technically] for the scenario you described…

      1. Hi Brian, it’s because I need to pass some dynamically generated parameters (based on user selections) to provide the cloud service names, location and which certificate to use. As far as I’m aware, it is not possible to provide some parameters to New-AzureRMResourceGroupDeployment as a parameters object and others (i.e. the keyvault secret) via a parameters file, it’s either one or the other. Unless you know different?

      2. You can override individual parameters on the cmd line by simply providing them as a param to the deployment cmdlets. I don’t know if that’s [well] documented but that’s one option. The other would be creating a param object on the file from the file, a bit more involved but might fit your workflow better. If you need more detail, lmk we can try to work through it via email. (bmoore @ microsoft.com)

  5. Hi Brian, thanks for that. I will take another look tomorrow (UK time) and perhaps ping you if I get stuck.

    1. Kyle – KeyVaults don’t yet show up in the portal. Your question is a little vague – are you suggesting a new resource isn’t showing in the right resource group or that it isn’t showing at all? Could you clarify a little?

      1. Hello Simon,

        Sorry about that. What i am saying is i have created a resource group, and used an arm template to create a key vault resource within that resource group. But when i look at the resource that the key vault should be in i cannot see a key vault resource. From what you said in your reply the key vaults do not show up yet in the portal. I am assuming that is coming though.

        Thank you very much 🙂

  6. Hi Simon,

    We are managing the deploy time keys and secrets using the vaults. But once it is provisioned, to login to the server, we need to store the keys locally or secrets should be known to others right? In this case the utmost purpose of storing keys securely in vault and referencing in provisioning is defeated. Is there a nice way of handling this scenario?

    1. Hi Satheeshkumar – your scenario is pretty common and there’s a few ways to look at it:

      • Use keyvault for AD Domain-level admins, but use local Admins to grant Windows server access. This way your Domain Admin password isn’t shared.
      • Use cert-based RDP – while you need to share the certificate you can easily block access by rotating the cert in use. This applies to SSH as well.

      The ideal use case I see is for hands-off environment provisioning where you want non-Admin users to be able to create an environment that they don’t necessarily need admin-level access to.

      HTH.

  7. Hello Simon,

    i am using keyvault to assign password to my VM. here i use 2 files (Template & parameter) to make it work. now i am using a tool to deploy Arm template in Azure but here tool will allow to use only onefile. now i need this template and parameter to be used from single file. is it possible to make keyvault secret called from single template if possible can you help or guide me on getting it to work.

    1. BB – that seems like an odd restriction from the tool given the whole nature of using parameters. Does that tool allow you to pass in a json blob for parameters? You could supply the reference that way…

  8. Hey Simon,

    Do you happen to know if it’s possible to pass credentials from a vault into a DSC extension? For instance, I have a domain admin password that I need to feed into a DSC resource within my template that creates a second domain controller – now, that keyvault secret is defined in my parameters file, but it seems like it never gets to the DSC resource because it fails every time when looking for the domain, something I would expect if it never received domain credentials.

    Thanks!

    1. Hi Oliver,

      I haven’t tried that specific scenario so can’t advise specfically on it – it may be that the protectedsettings node for the DSC extension is expecting a PowerShell-type secure string which it would not be receiving from KeyVault.

      I know since the start of this year it’s been possible to domain join without using a DSC script which can be a bit of a nightmare to debug. See https://blogs.msdn.microsoft.com/igorpag/2016/01/25/azure-arm-vm-domain-join-to-active-directory-domain-with-joindomain-extension/.

      If you want to persist with DSC there are local log files on the machine that will give you debug information (some of which is displayed in the Portal as well) for the extension when it fails.

      HTH,
      Simon.

      1. Hey Simon,

        Thanks for the reply! I am actually using that joindomain extension already in my template – my scenario is that I have one domain controller come up, then afterwards two more VMs are created – one to be a generic server and one to be a secondary DC. The server I have joining the domain with the joindomain extension from my template, but using the xADDomainController DSC extension is where I am having issues.

        Even removing the keyvault as a variable, it doesn’t seem to want to accept my credentials from my ARM template. The debug information, or at least the information that I have found on the machine and in the portal, unfortunately doesn’t tell me what it thinks it has for credentials, which makes this really tough to troubleshoot.

      1. Hey, thanks for the reply! I have seen that, and as much other documentation as I can dig up after like two weeks of searching. Could you possibly take a look here: https://github.com/oradcliffe/AzureRm-Windows-Domain/tree/seconddc and see if there’s anything I am doing wrong? For the life of me, two weeks at least, I haven’t been able to figure this out.

        I just get an error saying that the second DSC extension fails because it can’t find the domain, which I believe is due to the fact that the extension isn’t getting domain credentials, but I am having a tough time seeing what could possibly be wrong with the way I am passing these credentials around.

      2. Oliver, I had a play with your template and removed the keyvault dependency to test it out. It provisions fine, however the one in Github has a fairly big issue – apart from your primary DC you don’t run the DSC extension on any other VM (perhaps you uploaded an old version of the template? Based on your comment it sounds like it). Apart from this I’d recommend looking at:

        1. Dependency management in ARM templates (this makes sure your DC is provisioned before trying to Domain join other machines to it). See: https://blogs.msdn.microsoft.com/brian_farnhill/2015/09/08/cross-server-dsc-dependency-options-with-azure-resource-manager-templates/
        2. Consider using the non-DSC based Domain Join as shown in this quick start: https://github.com/Azure/azure-quickstart-templates/blob/master/201-vm-domain-join/

        Hope this helps.

  9. Hey thanks for your reply Simon, but I do have a DSC resource on my second DC (dc02); it’s under resources and starts at line 514. I have that resource depending on dc02 being up (which itself depends on having a nic, etc) and also depending on the domain join of server1, which is what you are talking about in point 2, using the non-DSC domain join. So to summarize, I have one domain controller coming up to create the domain, a generic windows server that joins the domain using the json domain join, and I am having issues with passing credentials into DC02.

    I have merged this branch back into another one, so I am leaving the branch “seconddc” alone in case you want to take a look – right now in that branch I have the credentials for the DSC resource for DC02 explicitly hard coded in plain text (lines 541, 542) and as-is this works. It’s when I use a parameter for username and password that it doesn’t work, even though previously I was referencing the admin name/pass for DC01 (since that’s the account used to bring up the first domain controller and create the domain).

    The lines I am referencing are here: https://github.com/oradcliffe/AzureRm-Windows-Domain/blob/seconddc/WindowsVirtualMachine.json

    Thanks!

    1. Oliver, I had a play (without using KeyVault) and can get most of the script to work – the piece that fails looks like it is due to the way you are using the xActiveDirectory Module in your CreateSecondDC.ps1 script to try and add the DC to an existing Domain.

      I’m not a DSC guru, and certainly less so in the Active Directory space, but there is an example script that shows how you go about creating a HA setup for AD that might be useful: https://github.com/PowerShell/xActiveDirectory

      I made a couple of minor tweaks to your template – you will find a PR with them pending for your branch on Github.

      One tip I’ll give you about troubleshooting this: don’t delete the Resource Group or even your VM. Make your DSC changes, upload them to Github at the asset URL you have in your template and then simply delete the DSC Extension from the VM and run your ARM template again. This will pull down whatever the latest pieces are that you put into your DSC modules.

      HTH.

      1. Hey, in that example for HA AD, it looks like he’s allowing plaintext passwords in the MOF that gets created, which is something I am trying to avoid; if I hard code passwords this all works for me every time too. I actually have it all working, even pulling from a keyvault now, EXCEPT for in my DC-02 DSC script I am defining username and password outside of the private settings, which I would like to be using.

        Here’s kind of the latest of where I’m at, if you’re interested in all this:
        https://powershell.org/forums/topic/dsc-azure-templates-and-key-vaults/

        As far as your PR, I looked it over and it looks like you’re putting a dependency on the actual VM resources for the DC01 VM resource, which I am not sure I want to do. I am pretty new to this so please stop and correct me if I’m wrong, but if we put those dependencies up, then neither of those VMs will start to provision until DC01 is up, but since it’s just the DC01 VM that we’re waiting for, we don’t have an AD domain yet anyway.

        The way I see the workflow going is the 3 VMs come up, DC01 promotion happens, then Server1 joins the domain, and then I set the dependency on DC02 promotion on Server1 joining the domain, because for sure the domain is up at that point. I’m not sure that the order the VMs themselves come up is important, but please correct me if I am wrong.

        Thanks for your replies! And if you are interested, do check out that link to Powershell.org, I got a script resource in DSC to work and actually spit out some information about the credentials that are being passed in to the script, so that was pretty interesting! It is totally working the way my xActiveDirectory is set up in the second domain controller, but for whatever reason if I use the PrivateSettingsRef for the Admin password, the password gets lost somewhere.

      2. The domain has to be available before you try to join it. In theory DSC should try to correct “skew” in configuration by re-applying itself but that’s not how it works here. I think the dependency actually needs to be on the DSC extension of your primary DC as that’s what you need to succeed before trying to run the others – and that the dependency should actually be defined within the DSC extensions on the other hosts (i.e. the VMs will be created but the DSC won’t run until the primary DC’s DSC has succeeded). It might also be worth splitting this template into two – one for the Primary DC and one for the other VMs.

      3. Real quick, if my DC02 DSC resource completes and does promote it to a domain controller, if I remove the DSC extension it will still be that, right? So it wouldn’t need to rerun the extension, so it might be easier to de-provision rather than go through a demotion process right?

  10. Oh I think I see what happened, you forked my master branch, and I haven’t merged the second domain controller piece with that branch yet. The other two branches, seconddc and activedir, are what I am working off of.

    1. Hi AK,

      You can specify the version you want by added another property “secretVersion” in the parameters file. I created an updated version of the above sample to show you how:

      HTH,
      Simon.

    1. You should also take a look at the new Managed Service Identity that App Service now supports as it looks to be a cleaner option than what I have here. Once I get some hands-on time I will write up about it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s