Category Archives: Security

Secure your VSTS Release Management Azure VM deployments with NSGs and PowerShell

One of the neat features of VSTS’ Release Management capability is the ability to deploy to Virtual Machine hosted in Azure (amongst other environments) which I previously walked through setting up.

One thing that you need to configure when you use this deployment approach is an open TCP port to the Virtual Machines to allow remote access to PowerShell and WinRM on the target machines from VSTS.

In Azure this means we need to define a Network Security Group (NSG) inbound rule to allow the traffic (sample shown below). As we are unable to limit the source address (i.e. where VSTS Release Management will call from) we are stuck creating a rule with a Source of “Any” which is less than ideal, even with the connection being TLS-secured. This would probably give security teams a few palpitations when they look at it too!

Network Security Group

We might be able to determine a source address based on monitoring traffic, but there is no guarantee that the Release Management host won’t change at some point which would mean our rule blocks that traffic and our deployment breaks.

So how do we fix this in an automated way with VSTS Release Management and provide a secured environment?

Let’s take a look.

The Fix

The fix is actually quite straightforward it turns out.

As the first step you should go to the existing NSG and flip the inbound rule from “Allow” to “Deny”. This will stop the great unwashed masses from being able to hit TCP port 5986 on your Virtual Machines immediately.

As a side note… if you think nobody is looking for your VMs and open ports, try putting a VM up in Azure and leaving RDP (3389) open to “Any” and see how long it takes before you start seeing authentication failures in your Security event log due to account enumeration attempts.

Modify Project Being Deployed

We’re going to leverage an existing Release Management capability to solve this issue, but first we need to provide a custom PowerShell script that we can use to manipulate the NSG that contains the rule we are currently using to block inbound traffic.

This PowerShell script is just a simple wrapper that combines Azure PowerShell Cmdlets to allow us to a) read the NSG b) update the rule we need c) update the NSG, which commits the change back to Azure.

I usually include this script in a Folder called “Deploy” in my project and set the build action to “Copy always”. As a result the file will be copied to the Artefacts folder at build time which means we have access to it in Release Management.

Project Setup

You should run a build with this included file so that it is available in your

Modify Release Management Defintion

Note that in order to complete this step you must have a connection between VSTS and your target Azure Subscription already configured as a Service Endpoint. Typically this needs to be done by a user with sufficient rights in both VSTS and the Azure Subscription.

Now we are going to modify our existing Release Management definition to make use of this new script.

The way we are going to enable this is by using the existing Azure PowerShell Task that we have available in both Build and Release Management environments in VSTS.

I’ve shown a sample where I’ve added this Task to an existing Release Management definition.

Release Management Definition

There is a reason this Task is added twice – once to change the NSG rule to be “Allow” and then once, at the end, to switch it back to “Deny”. Ideally we want to do the “Allow” early in the process flow to allow time for the NSG to be updated prior to our RM deployment attempting to access the machine(s) remotely.

The Open NSG Task is configured as shown.

Allow Script

The Script Arguments should match those given in the sample script above. As sample we might have:

-resourceGroupName MyTestResourceGroup -networkSecurityGroupName vnet01-nsg 
-securityRuleName custom-vsts-deployments -allowOrDeny Allow -priority 3010

The beauty of our script is that the Close NSG Task is effectively the same, but instead of “Allow” we put “Deny” which will switch the rule to blocking traffic!

Make sure you set the “Close” Task to “Always run”. This way if any other component in the Definition fails we will at least close up the NSG again.

Additionally, if you have a Resource Group Lock in place (and you should for all production workloads) this approach will still work because we are only modifying an existing rule, rather than trying to add / remove it each time.

That’s it!

You can now benefit from VSTS remote deployments while at the same time keeping your environment locked down.

Happy days 🙂

Tagged , , , , ,

Azure Functions: Access KeyVault Secrets with a Cert-secured Service Principal

Azure Functions is one of those services in Azure that is seeing a massive amount of uptake. People are using it for so many things, some of which require access to sensitive information at runtime.

At time of writing this post there is a pending Feature Request for Functions to support storing configuration items in Azure KeyVault. If you can’t wait for that Feature to drop here’s how you can achieve this today.

Step 1: Create a KeyVault and Register Secrets

I’m not going to step through doing this in detail as the documentation for KeyVault is pretty good, especially for adding Secrets or Keys. For our purposes we are going to store a password in a Secret in KeyVault and have the most recent version of it be available from this URI:

https://mytestvault.vault.azure.net/secrets/remotepassword

Step 2: Setup a Cert-secured Service Principal in Azure AD

a. Generate a self-signed certificate

This certificate will be used for our Service Principal to authorise itself when calling into KeyVault. You’ll notice that I’m putting a -1 day “start of” validity period into this certificate. This allows us to deal with the infrastructure running at UTC (which my location isn’t) and avoid not being able to access the certificate until UTC matches our local timezone.

b. Create Service Principal with Cert Authentication

This step requires you to log into an Azure Subscription that is tied to the target Azure AD instance in which you wish to register the Service Principal. Your user must also have sufficient privileges to create new users in Azure AD – if it doesn’t this step will fail.

At this point we now have a Vault, a Secret, and a Service Principal that has permissions to read Secrets from our Vault.

Step 3: Add Cert to App Service

In order for our Function App(s) to utilise this Service Principal and its certificate to access KeyVault we need to upload the PFX file we created in 2.a above into the App Service in which our Functions live. This is just as you would do if this App Service was running a Web App but without the need to bind it to anything. The official Azure documentation on uploading certs is good so I won’t duplicate the instructions here.

Watch out – Gotcha!

Once you’ve uploaded your certificate you do need to do one item to ensure that your Function code can read the certificate from store. You do this by adding an Application Setting “WEBSITE_LOAD_CERTIFICATES” and either specify just the thumbprint of your certificate or put “*” to specify any certificate held in the store.

Step 4: Function App KeyVault and Service Principal Setup

a. Nuget Packages

Accessing KeyVault with a Service Principal in Functions requires us to load some Nuget packages that contain the necessary logic to authenticate with Azure AD and to call KeyVault. We do this by adding the following to our Function App’s project.json.

b. KeyVault Client CSX
Now let’s go ahead and drop in our KeyVault “client” that wraps all code for accessing KeyVault in a single CSX (note that this is mostly inspired by other code that shows you how to do this for Web Apps).

Step 5: Use in a Function

As we’ve encapsulated everything to do with KeyVault into a CSX we can retrieve a secret from KeyVault in a Function using a single call once we’ve imported our client code.

Happy (Secure) Days!

Tagged , , ,

Azure Automation Runbooks with Azure AD Service Principals and Custom RBAC Roles

If you’ve ever worked in any form of systems administrator role then you will be familiar with process automation, even only for simple tasks like automating backups. You will also be familiar with the pain of configuring and managing identities for these automated processes (expired password or disabled/deleted account ever caused you any pain?!)

While the cloud offers us many new ways of working and solving traditional problems it also maintains many concepts we’d be familiar from environments of old. As you might have guessed, the idea of specifying user context for automated processes has not gone away (yet).

In this post I am going to look at how in Azure Automation Runbooks we can leverage a combination of an Azure Active Directory Service Principal and an Azure RBAC Custom Role to configure an non-user identity with constrained execution scope.

The benefits of this approach are two-fold:

  1. No password expiry to deal with, or accidental account disablement/deletion. I still need to manage keys (in place of passwords), but these are centrally managed and are subject to an expiry policy I define.
  2. Reduced blast radius. Creating a custom role that restricts actions available means the account can’t be used for more actions that intended.

My specific (simple) scenario is stopping all running v2 VMs in a Subscription.

Create the Custom Role

The Azure Resource Manager (ARM) platform provides a flexible RBAC model within which we can build our own Roles based on a combination of existing Actions. In-built Roles bundle these Actions into logical groups, but there are times we may want something a little different.

The in-built “Virtual Machine Contributor” Role isn’t suitable for my purposes because it provides too much scope by letting assigned users create and delete Virtual Machines. In my case, I want a Role that allows an assigned user to start, stop, restart and monitor existing Virtual Machines only.

To this end I defined a custom Role as shown below which allows any assigned user to perform the functions I need them to.

Let’s add this Role by executing the following PowerShell (you’ll need to be logged into your Subscription with a user who has enough rights to create custom role definitions). You’ll need to grab the above definition, change the scope and save it as a file named ‘vm-power-manager-customerole.json’ for this to work.

New-AzureRmRoleDefinition -InputFile vm-power-manager-customrole.json

which will return a result similar to the below.

Name             : Virtual Machine Power Manager
Id               : 6270aabc-0698-4380-a9a7-7df889e9e67b
IsCustom         : True
Description      : Can monitor, stop, start and restart v2 ARM virtual machines.
Actions          : {Microsoft.Storage/*/read, Microsoft.Network/*/read, Microsoft.Compute/*/read
                   Microsoft.Compute/virtualMachines/start/action...}
NotActions       : {}
AssignableScopes : {/subscriptions/c25b1c8e-1111-4421-9090-1a12d7012dd3}

that means the Role shows up in the Portal and can be assigned to users 🙂

VM Power Manager Role

Now we have that, let’s setup our Service Principal.

Setup Service Principal

Microsoft provides a good guide on creating a Service Principal on the Azure documentation site already so I’m not going to reproduce that all here.

When you get to “Assign application to role” hop back here and we’ll continue on without needing to dive into the Azure Portal.

For the purpose of the rest of this post, these are the parameters I used to create my Service Principal.

Name: Azure VM Start Stop SP
Sign-on URL / App URI: http://myvmautomation
Client ID*: c6f7c745-1234-5678-0000-8d14611e75f4
Tenant ID*: c7a48abc-1990-4fef-e941-a1cd55422e41

* Client ID is returned once you save your Application. Tenant ID comes from your Azure AD tenant ID (see Microsoft setup instructions referenced above).

Important: you will also have to generate and grab a key value that you will need to use as it is the password for the Service Principal. Don’t forget to grab it when it’s displayed!

Assign the Service Principal the necessary Azure Roles

# Assign our custom Role
New-AzureRmRoleAssignment -ServicePrincipalName http://myvmautomation `
                          -RoleDefinitionName 'Virtual Machine Power Manager' `
                          -Scope '/subscriptions/c25b1c8e-xxxx-1111-abcd-1a12d7012123'

# Assign the built-in 'Reader' Role
New-AzureRmRoleAssignment -ServicePrincipalName http://myvmautomation `
                          -RoleDefinitionName 'Reader' `
                          -Scope '/subscriptions/c25b1c8e-xxxx-1111-abcd-1a12d7012123'

We now have all the baseline mechanics out of the way – next it’s onto using this information in our Runbooks.

Asset Setup

Azure Automation has the concept of an Asset that can be one of six items: Schedules, PowerShell Modules, Certificates, Connections, Variables and Credentials.

These are shared between all Runbooks in an Automation Account and are extremely useful in helping you deliver generic re-usable Runbooks.

For this post we are going to create a new Credential using the following process.

Our Automation Account is called ‘Core-Services’ and is hosted in a Resource Group ‘rg-test-01’

$username = "c6f7c745-1234-5678-0000-8d14611e75f4"
$password = ConvertTo-SecureString -String "YOUR_SERVICE_PRINCIPAL_KEY" -AsPlainText -Force

$newCreds = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $username,$password

New-AzureRmAutomationCredential -Name "VMPowerServicePrincipal" `
                                -Description 'Service Principal used to control power state of VMs' `
                                -Value $newCreds `
                                -ResourceGroupName 'rg-test-01' `
                                -AutomationAccountName 'Core-Services'

This creates a Credential we can now use in any Runbook.

The sample PowerShell Runbook below shows how we do this using the Login-AzureRmAccount Cmdlet using the -ServicePrincipal switch.

I also specify a Tenant identifier (this is the Azure AD Tenant identifier from when you setup the Service Principal) and the Subscription identifier so we set context in one call.

The Tenant and Subcsription identifiers are held as Automation Account Variables which we read in at the start of execution (the pattern below allows you to override which Variables you pass in should you want to use different ones).

So there we have it – a way to perform VM power state management in an Azure Automation Runbook that uses a non-user account for authentication along with custom RBAC roles for authorisation.

Enjoy!

Tagged , , ,

Using Active Directory Security Groups to Grant Permissions to Azure Resources

Kloud Blog

The introduction of the Azure Resource Manager platform in Azure continues to expose new possibilities for managing your deployed resources.

One scenario that you may not be aware of is the ability to use scoped RBAC role assignments to grant limited rights to Azure AD-based users and groups.

We know Azure provides us with many built-in RBAC roles, but it may not be immediately obvious that you can control their assignment scope.

What do I mean by this?

Simply that each RBAC role (including custom ones you create) can be used at various levels within Azure starting at the Subscription level (i.e. applies to anything in the Subscription) down to a Resource (i.e. applies just to one particular resource such as a Storage Account). Role assignments are also cascading – if I assign “Owner” rights to a User or Group at the Subscription level then they have that role…

View original post 355 more words

Tagged ,

No More Plaintext Passwords: Using Azure Key Vault with Azure Resource Manager

A big part of where Microsoft Azure is going is being driven by template-defined environments that leverage the Azure Resource Manager (ARM) for deployment orchestration.

If you’ve spent any time working with ARM deployments you will have gotten used to seeing this pattern in your templates when deploying Virtual Machines (VMs):

The adminPassword property accepts a Secure String object which contains an encrypted string that is passed to the VM provisioning engine in Azure and is used to set the login password. You provide the clear text version of the password either as a command-line parameter, or via a parameters file.

The obvious problems with this way of doing things are:

  1. Someone needs to type the cleartext password which means:
    1. it needs to be known to anyone who provisions the environment and
    2. how do I feed it into an automated environment deployment?
  2. If I store the password in a parameter file (to get around 1.B) I risk leaking the password should someone access the file (typically from source control).

Thankfully, Microsoft now provides a way to help solve these challenges.

Say hello to Azure Key Vault

Key Vault can store two types of information: Keys (Certificates) and Secrets.

In our scenario we are interested in the storing of Secrets which fit perfectly with our Secure String requirements for passwords.

Firstly I’m going to provision a Key Vault instance – note that this doesn’t need to be in the same Region or Resource Group as your intended deployment. In fact, I’d argue that you wouldn’t provision your Vault in the same Region (or Resource Group) as your deployments to restrict access to the Key Vault as much as possible.

Once the Key Vault is provisioned we are ready to push our passwords into it as Secrets.

I think its worth noting that a Secret has a lifecycle – you can update a Secret once created (i.e. publish an updated password to the same Secret) which means you can request specific versions of Secrets over time. This is beneficial to our scenario because it allows us to support multiple environments at different stages of their lifecycle.

For example: need an environment from two years ago? No issue – just request it be provisioned with the password (Secret) that was valid at that point in time.

Let’s go ahead and add a password to our Vault.

I could, additionally, specify a validity date range for this Secret using the NotBefore and Expires arguments. For this blog we’ll just leave it to be valid until updated with a new value.

The above PowerShell will return the following output from which we can grab the ‘ID’ of the Secret which we will use later.

Vault Name   : provisionvault
Name         : LocalAdminPass
Version      : 2c585ddc60f54d048767924d0125ede5
Id           : https://provisionvault.vault.azure.net:443/secrets/LocalAdminPass/2c585ddc60f54d048767924d0125ede5
Enabled      : True
Expires      :
Not Before   :
Created      : 23/11/2015 01:58:26
Updated      : 23/11/2015 01:58:26
Content Type :
Tags         :

Using in ARM templates

Now we have our password stored as a Secret in a Key Vault we can go ahead and use it in an ARM template.

At present the way we supply Key Vault Secrets to ARM templates is via use of parameters file. If we use our original snippet for VM creation (shown below)

we can feed in the Secret we pushed to our Key Vault using the below snippet in our parameters file.

In order to deploy this template we must use the Azure CLI tools with the following command:

azure group deployment create -n {deploymentname} -g {resourcegrop} -f {template}.json -e {template-parameters}.json

Once the deployment is finished you will find that the provisioned VM(s) are now accessible with the password that had been securely held in the Key Vault!

Where can I use this?

I’ve tested in a range of scenarios from VM provisioning (Windows and Linux) through to Azure SQL Database Server setup, and they all support this technique.

I did find at least one scenario that didn’t work… which I’ll cover next.

Limitations

You probably spotted that I switched from PowerShell earlier in the blog to do the final deployment with the Azure CLI. At time of writing the PowerShell Cmdlets (v1.0.0 / 1.0.1) don’t support the feeding of a Key Vault reference to an ARM template so for your final deployment step (or all of your steps if you want) you need to use the CLI.

Additionally, some existing ARM deployment scenarios such as “deploy AD-domain joined VMs” that leverage extensions like the DSC one, don’t appear to support the feeding of Key Vault secrets into them (and a result fail). I’m still troubleshooting this one so if I find a resolution I’ll post an update here.

Finally, I also find the supplying of the Key Vault and Secret details in the parameters file a little restrictive. I tried changing the actual template to reference the Key Vault but it’s not an approach that is supported and your deployment fails.

Summary

Having been working with Azure for quite some time it’s interesting to see the transition of the platform to the new ARM-based model (also known as the ‘v2’ platform) and the possibilities it opens up such as the one shown here.

If you’re doing a lot of template-based deployments that include passwords stored in parameter files (or other insecure stores) then this approach should provide you with a lot of value (and comfort). Also, if as an IT Pro you are concerned about locking down environments so that only the right groups of users can login and modify configurations, this approach should be high on your list.

As always, leave any comments or questions below.

Simon.

Tagged , ,

Azure Security Fundamentals: Azure SQL Database

I’m continuing my Kloud blog series on the security fundamentals for various Microsoft Azure services with my most recent post being for Azure SQL Database.

Please go and have a read!

Tagged , ,