Tag Archives: Azure Key Vault

Moving from Azure VMs to Azure VM Scale Sets – Runtime Instance Configuration

In my previous post I covered how you can move from deploying a solution to pre-provisioned Virtual Machines (VMs) in Azure to a process that allows you to create a custom VM Image that you deploy into VM Scale Sets (VMSS) in Azure.

As I alluded to in that post, one item we will need to take care of in order to truly move to a VMSS approach using a VM image is to remove any local static configuration data we might bake into our solution.

There are a range of options you can move to when going down this path, from solutions you custom build to running services such as Hashicorp’s Consul.

The environment I’m running in is fairly simple, so I decided to focus on a simple custom build. The remainder of this post is covering the approach I’ve used to build a solution that works for me, and perhaps might inspire you.

I am using an ASP.Net Web API as my example, but I am also using a similar pattern for Windows Services running on other VMSS instances – just the location your startup code goes will be different.

The Starting Point

Back in February I blogged about how I was managing configuration of a Web API I was deploying using VSTS Release Management. In that post I covered how you can use the excellent Tokenization Task to create a Web Deploy Parameters file that can be used to replace placeholders on deployment in the web.config of an application.

My sample web.config is shown below.

The problem with this approach when we shift to VM Images is that these values are baked into the VM Image which is the build output, which in turn can be deployed to any environment. I could work around this by building VM Images for each environment to which I deploy, but frankly that is less than ideal and breaks the idea of “one binary (immutable VM), many environments”.

The Solution

I didn’t really want to go down the route of service discovery using something like Consul, and I really only wanted to use Azure’s default networking setup. This networking requirement meant no custom private DNS I could use in some form of configuration service discovery based on hostname lookup.

…and…. to be honest, with the PaaS services I have in Azure, I can build my own solution pretty easily.

The solution I did land on looks similar to the below.

  • Store runtime configuration in Cosmos DB and geo-replicate this information so it is highly available. Each VMSS setup gets its own configuration document which is identified by a key-service pair as the document ID.
  • Leverage a read-only Access Key for Cosmos DB because we won’t ever ask clients to update their own config!
  • Use Azure Key Vault as to store the Cosmos DB Account and Access Key that can be used to read the actual configuration. Key Vault is Regionally available by default so we’re good there too.
  • Configure an Azure AD Service Principal with access to Key Vault to allow our solution to connect to Key Vault.

I used a conventions-based approach to configuration, so that the whole process works based on the VMSS instance name and the service type requesting configuration. You can see this in the code below based on the URL being used to access Key Vault and the Cosmos DB document ID that uses the same approach.

The resulting changes to my Web API code (based on the earlier web.config sample) are shown below. This all occurs at application startup time.

I have also defined a default Application Insights account into which any instance can log should it have problems (which includes not being able to read its expected Application Insights key). This is important as it allows us to troubleshoot issues without needing to get access to the VMSS instances.

Here’s how we authorise our calls to Key Vault to retrieve our initial configuration Secrets (called on line 51 of the above sample code).

My goal was to make configuration easily manageable across multiple VMSS instances which requires some knowledge around how VMSS instance names are created.

The basic details are that they consist of a hostname prefix (based on what you input at VMSS creation time) that is appended with a base-36 (hexatrigesimal) value representing the actual instance. There’s a great blog from Guy Bowerman from Microsoft that covers this in detail so I won’t reproduce it here.

The final piece of the puzzle is the Cosmos DB configuration entry which I show below.

The ‘id’ field maps to the VMSS instance prefix that is determined at runtime based on the name you used when creating the VMSS. We strip the trailing 6 characters to remove the unique component of each VMSS instance hostname.

The outcome of the three components (code changes, Key Vault and Cosmos DB) is that I can quickly add or remove VMSS groups in configuration, change where their configuration data is stored by updating the Key Vault Secrets, and even update running VMSS instances by changing the configuration settings and then forcing a restart on the VMSS instances, causing them to re-read configuration.

Is the above the only or best way to do this? Absolutely not 🙂

I’d like to think it’s a good way that might inspire you to build something similar or better 🙂

Interestingly, getting to this stage as well, I’ve also realised there might be some value in considering moving this solution to Service Fabric in future, though I am more inclined to shift to Containers running under the control an orchestrator like Kubernetes.

What are you thoughts?

Until the next post!

Tagged , , , , , ,

No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

A big part of where Microsoft Azure is going is being driven by template-defined environments that leverage the Azure Resource Manager (ARM) for deployment orchestration.

If you’ve spent any time working with ARM deployments you will have gotten used to seeing this pattern in your templates when deploying Virtual Machines (VMs):

The adminPassword property accepts a Secure String object which contains an encrypted string that is passed to the VM provisioning engine in Azure and is used to set the login password. You provide the clear text version of the password either as a command-line parameter, or via a parameters file.

The obvious problems with this way of doing things are:

  1. Someone needs to type the cleartext password which means:
    1. it needs to be known to anyone who provisions the environment and
    2. how do I feed it into an automated environment deployment?
  2. If I store the password in a parameter file (to get around 1.B) I risk leaking the password should someone access the file (typically from source control).

Thankfully, Microsoft now provides a way to help solve these challenges.

Say hello to Azure Key Vault

Key Vault can store two types of information: Keys (Certificates) and Secrets.

In our scenario we are interested in the storing of Secrets which fit perfectly with our Secure String requirements for passwords.

Firstly I’m going to provision a Key Vault instance – note that this doesn’t need to be in the same Region or Resource Group as your intended deployment. In fact, I’d argue that you wouldn’t provision your Vault in the same Region (or Resource Group) as your deployments to restrict access to the Key Vault as much as possible.

Once the Key Vault is provisioned we are ready to push our passwords into it as Secrets.

I think its worth noting that a Secret has a lifecycle – you can update a Secret once created (i.e. publish an updated password to the same Secret) which means you can request specific versions of Secrets over time. This is beneficial to our scenario because it allows us to support multiple environments at different stages of their lifecycle.

For example: need an environment from two years ago? No issue – just request it be provisioned with the password (Secret) that was valid at that point in time.

Let’s go ahead and add a password to our Vault.

I could, additionally, specify a validity date range for this Secret using the NotBefore and Expires arguments. For this blog we’ll just leave it to be valid until updated with a new value.

The above PowerShell will return the following output from which we can grab the ‘ID’ of the Secret which we will use later.

Vault Name   : provisionvault
Name         : LocalAdminPass
Version      : 2c585ddc60f54d048767924d0125ede5
Id           : https://provisionvault.vault.azure.net:443/secrets/LocalAdminPass/2c585ddc60f54d048767924d0125ede5
Enabled      : True
Expires      :
Not Before   :
Created      : 23/11/2015 01:58:26
Updated      : 23/11/2015 01:58:26
Content Type :
Tags         :

Using in ARM templates

Now we have our password stored as a Secret in a Key Vault we can go ahead and use it in an ARM template.

At present the way we supply Key Vault Secrets to ARM templates is via use of parameters file. If we use our original snippet for VM creation (shown below)

we can feed in the Secret we pushed to our Key Vault using the below snippet in our parameters file.

In order to deploy this template we must use the Azure CLI tools with the following command:

azure group deployment create -n {deploymentname} -g {resourcegrop} -f {template}.json -e {template-parameters}.json

Once the deployment is finished you will find that the provisioned VM(s) are now accessible with the password that had been securely held in the Key Vault!

Where can I use this?

I’ve tested in a range of scenarios from VM provisioning (Windows and Linux) through to Azure SQL Database Server setup, and they all support this technique.

I did find at least one scenario that didn’t work… which I’ll cover next.

Limitations

You probably spotted that I switched from PowerShell earlier in the blog to do the final deployment with the Azure CLI. At time of writing the PowerShell Cmdlets (v1.0.0 / 1.0.1) don’t support the feeding of a Key Vault reference to an ARM template so for your final deployment step (or all of your steps if you want) you need to use the CLI.

Additionally, some existing ARM deployment scenarios such as “deploy AD-domain joined VMs” that leverage extensions like the DSC one, don’t appear to support the feeding of Key Vault secrets into them (and a result fail). I’m still troubleshooting this one so if I find a resolution I’ll post an update here.

Finally, I also find the supplying of the Key Vault and Secret details in the parameters file a little restrictive. I tried changing the actual template to reference the Key Vault but it’s not an approach that is supported and your deployment fails.

Summary

Having been working with Azure for quite some time it’s interesting to see the transition of the platform to the new ARM-based model (also known as the ‘v2’ platform) and the possibilities it opens up such as the one shown here.

If you’re doing a lot of template-based deployments that include passwords stored in parameter files (or other insecure stores) then this approach should provide you with a lot of value (and comfort). Also, if as an IT Pro you are concerned about locking down environments so that only the right groups of users can login and modify configurations, this approach should be high on your list.

As always, leave any comments or questions below.

Simon.

Tagged , ,