Tag Archives: VSTS

Secure your VSTS Release Management Azure VM deployments with NSGs and PowerShell

One of the neat features of VSTS’ Release Management capability is the ability to deploy to Virtual Machine hosted in Azure (amongst other environments) which I previously walked through setting up.

One thing that you need to configure when you use this deployment approach is an open TCP port to the Virtual Machines to allow remote access to PowerShell and WinRM on the target machines from VSTS.

In Azure this means we need to define a Network Security Group (NSG) inbound rule to allow the traffic (sample shown below). As we are unable to limit the source address (i.e. where VSTS Release Management will call from) we are stuck creating a rule with a Source of “Any” which is less than ideal, even with the connection being TLS-secured. This would probably give security teams a few palpitations when they look at it too!

Network Security Group

We might be able to determine a source address based on monitoring traffic, but there is no guarantee that the Release Management host won’t change at some point which would mean our rule blocks that traffic and our deployment breaks.

So how do we fix this in an automated way with VSTS Release Management and provide a secured environment?

Let’s take a look.

The Fix

The fix is actually quite straightforward it turns out.

As the first step you should go to the existing NSG and flip the inbound rule from “Allow” to “Deny”. This will stop the great unwashed masses from being able to hit TCP port 5986 on your Virtual Machines immediately.

As a side note… if you think nobody is looking for your VMs and open ports, try putting a VM up in Azure and leaving RDP (3389) open to “Any” and see how long it takes before you start seeing authentication failures in your Security event log due to account enumeration attempts.

Modify Project Being Deployed

We’re going to leverage an existing Release Management capability to solve this issue, but first we need to provide a custom PowerShell script that we can use to manipulate the NSG that contains the rule we are currently using to block inbound traffic.

This PowerShell script is just a simple wrapper that combines Azure PowerShell Cmdlets to allow us to a) read the NSG b) update the rule we need c) update the NSG, which commits the change back to Azure.

I usually include this script in a Folder called “Deploy” in my project and set the build action to “Copy always”. As a result the file will be copied to the Artefacts folder at build time which means we have access to it in Release Management.

Project Setup

You should run a build with this included file so that it is available in your

Modify Release Management Defintion

Note that in order to complete this step you must have a connection between VSTS and your target Azure Subscription already configured as a Service Endpoint. Typically this needs to be done by a user with sufficient rights in both VSTS and the Azure Subscription.

Now we are going to modify our existing Release Management definition to make use of this new script.

The way we are going to enable this is by using the existing Azure PowerShell Task that we have available in both Build and Release Management environments in VSTS.

I’ve shown a sample where I’ve added this Task to an existing Release Management definition.

Release Management Definition

There is a reason this Task is added twice – once to change the NSG rule to be “Allow” and then once, at the end, to switch it back to “Deny”. Ideally we want to do the “Allow” early in the process flow to allow time for the NSG to be updated prior to our RM deployment attempting to access the machine(s) remotely.

The Open NSG Task is configured as shown.

Allow Script

The Script Arguments should match those given in the sample script above. As sample we might have:

-resourceGroupName MyTestResourceGroup -networkSecurityGroupName vnet01-nsg 
-securityRuleName custom-vsts-deployments -allowOrDeny Allow -priority 3010

The beauty of our script is that the Close NSG Task is effectively the same, but instead of “Allow” we put “Deny” which will switch the rule to blocking traffic!

Make sure you set the “Close” Task to “Always run”. This way if any other component in the Definition fails we will at least close up the NSG again.

Additionally, if you have a Resource Group Lock in place (and you should for all production workloads) this approach will still work because we are only modifying an existing rule, rather than trying to add / remove it each time.

That’s it!

You can now benefit from VSTS remote deployments while at the same time keeping your environment locked down.

Happy days 🙂

Tagged , , , , ,

Per-environment config value tokenization for Azure Web Apps using VSTS Release Management

For the majority of the last ten years I’ve been working with delivery of solutions where build and deployment comes from some centralised location.

When Microsoft made InRelease part of TFS as Release Management, I couldn’t wait to use it. Unfortunately in its state at that time the learning curve was quite steep and the immediate value was outweighed by the effort to get up and running.

Roll forward to 2016 and we find Release Management as a modern, web-based feature of Visual Studio Team Services (VSTS). The cherry on the cake is that a lot of the learning curve has dropped away as a result.

In this post I’m going to look at how we can deploy a Web Deploy (or MS Deploy) packaged Web Application to an Azure Web Application and define different deployment environments with varying configurations.

Many people would apply configuration transformations at build time, but in my scenario I want to deploy the same compiled package to multiple environments without the need to recompile anything.

My Challenge

The build definition for my Web Application results in a package that allows it to be deployed to an Azure Web App by Web Deploy. The result is the web.config configuration file is in a zip file that is transferred to the server for deployment by Web Deploy.

Clearly at this point I don’t have access to the web.config file in the drop folder so I can’t transform it with Release Management. Or can I?!

Using Web Deploy Parameters

Thankfully the design of Web Deploy provides for the scenario I described above though use of either commandline arguments or a specially formatted input file that I will call the “SetParameters” file.

Given this is a first-class feature in the broader Microsoft developer toolkit, I’d expected that there would be a few Tasks in VSTS that I could use to get all of this up and running… I got close, but couldn’t quite get it functioning as I wanted.

Through the rest of this post I will walk you through the setup to get this going.

Note: I am going to assume you have setup Build and Release Management definitions in VSTS already. Your Build should package to deploy to an Azure Web App and the Release Management definition to deploy it.

VSTS Release Management Setup

The first thing to get all of this up and running is to add the Release Management Utilities extension to your subscription. This extension includes the Tokenizer Task which will be key to getting the configuration per-environment up and running.

You also need to define an “Environment” in Release Management for each deployment target we have, which will also be used as a container for environmental configuration items to replace at deployment time. A sample is shown below with two Environments defined

Environments

We’ll come back to VSTS later, for now, let’s look at the project changes you need to make.

Source Project Changes

For the purpose of this exercise I’m just worrying about web.config changes.

First of all, you need to tokenise the settings you wish to transform. I have provided a sample below that shows how this looks in a web.config. The format of two underscores on either side of your token placeholder is required.

The next item we need to do is to add a new XML file to our Visual Studio project at the root level. This file should be called “Parameters.xml” and I have included a sample below that shows what we need to add to if it we want to ensure we replace the tokens in the above sample web.config.

You’ll notice one additional item in the file below that isn’t related directly to the web.config above – the IIS Website name that will be used when deployed. I found if I didn’t include this the deployment would fail.

When you add this file, make sure to set the properties for it to a Build Action of “None” and Copy to Output Directory of “Do not copy”.

Note: if you haven’t already done so, you should run a Build so that you have Build Artifacts ready to select in a later step.

Add the Tokenizer to your Release Management Definition

We need now to return to VSTS’ web interface and modify our existing Release Management definition (or create a new one) that adds the Tokenizer utility to the process.

You will need to repeat this so all your environments have the same setup. I’ve shown how my Test environment setup looks like below (note that I changed the default description of the Tokenizer Task).

Release Management Definition

Configuration of the Tokenizer is pretty straight forward at this point, especially if we’ve already run a build. Simply select the SetParameters.xml file your build already produced.

Tokenizer setting

Define values to replace Tokens

This is where we define the values that will be used to replace the tokens at deployment time.

Click on the three dots at the top right of the environment definition and from the menu select “Configuration variables…” as shown below.

Variable Definition

A dialog loads that allows us to define the values that will go into our web.config for this environment. The great thing you’ll note is that you can obfuscate sensitive details (in my example, the key to access the Document DB account). This is non-reversible too – you can’t “unhide” the value and see the plain-text version.

Token Values

We’re almost done!

Explicitly select SetParameters file for deployment

I’m using the 3.* (preview) version of the Deploy Azure App Service Release Management Task, which I have configured as shown.

App Service Task

At this point, if you create a new Release and deploy to the configured environment you will find that the deployed web.config contains the values you specified in VSTS and you will no longer need multiple builds to send the same package to multiple environments.

Happy Days! 🙂

Tagged , , , , , ,

Deploying to Azure VMs using VSTS Release Management

I am going to subtitle this post “the missing manual” because I spent quite a bit of time troubleshoothing how this should all work.

Microsoft provides a bunch of useful information on how to deploy from Visual Studio Team Services (VSTS) to different targets, including Azure Virtual Machines.

In an ideal world I wouldn’t be using VMs at all, but for my current particular use case I have to use VMs so the above (linked) approach worked.

The approach sounds good but I ran into a few sharp edges that I thought I would document here (and hopefully the source documentation will be updated to reflect this in due course).

Preparing deployment targets

Azure FQDNs

Note: Please see my update at the bottom of this post before reading this section. While you can use IP addresses (if you make them static) it’s worth configuring test certs with the FQDN.

I thought I’d do the right thing by configuring the Azure IP of my hosts to have a full FQDN rather than just an IP address.

As I found out this is not a good idea.

The main issue you will run into is the generated certs on target hosts only have the hostname in them (i.e. azauhost01) rather than the full public FQDN (i.e. azauhost01.australiaeast.cloudapp.azure.com).

When the Release Management infrastructure tries to connect to a host this cert mismatch causes a fatal error. I didn’t spend much time troubleshooting so decided to revert to use of IP addresses only.

When using dynamic IP addresses the first Release Management action “Azure Deployment:Select Resource Group action” is important as it allows for discovery of all VMs and their IPs (i.e. no hardcoding required). This apprach does mean, however, you need to consider how you group VMs into Resource Groups to allow any VM in the Resource Group to be used as the deployment target.

Select Resource Group

Local permissions

I was running my deployments to non-Domain joined Windows 2012 R2 server instances with local administrative accounts and had opened the necessary port in the NSG rules to allow WinRM into the servers from the VSTS infrastructure.

Everything looked good on deployment until PowerShell execution on the target hosts resulted in errors due to permissions. As it turns out the error message was actually useful in resolving this problem 🙂

In order to move beyond this error I had to prepare each target host by running these commands at an admin command prompt on the host:

winrm quickconfig

Enable-PSRemoting

We could drop these into a DSC module and run that way if we wanted to make this repeatable across new hosts.

There is a good troubleshooting overview for this from Microsoft.

Wait! Where did my PowerShell script go?

If you follow the instructions provided by Microsoft you need to add a deployment Powershell script (actually a DSC module) to your Web App (their example uses “ConfigureWebserver.ps1” for the filename).

There is one issue with this approach – the build step to package the Web App actually ends up bundling the PowerShell inside of a deployment zip which means once the files are copied to your target VM the PowerShell can’t be invoked.

The fix for this is to add an additional build step that copies the PowerShell to the drops folder on VSTS which means the PowerShell will be transferred to the target VM.

Your final build definition should look like the below

Build definition

and the Copy Files task should be configured like this (note below that /Deploy is the folder in my solution that contains the PowerShell I’ve included for deployment purposes):

Build Step

Once you have done this you will find that the script is now available in the VSTS drops folder and can be copied to the VMs which allows you to execute it via the Release Management action.

Wrapping up

Once I had these changes in place and had made some minor path / project name tweaks to match my project I could run the process end-to-end.

The one final item I’ll call out here is the default deployment location of the solution on the target VM ends up being the wwwroot of your inetpub folder with a subfolder named ProjectName_deploy. If you map this to an Application in IIS you should be good to go :).

Update – 8 December 2016

After I’d been running happily for a while my Release started failing. It turns out my target VMs were cycled and moved to different public IP addresses. As the WinRM HTTP certificate had the old IP address in the remote calls failed.

I found a great blog post on how to rectify this situation though: http://www.dotnetcurry.com/windows-azure/1289/configure-winrm-execute-powershell-remote-azure-with-arm

Happy days!

Tagged , ,