Category Archives: IaaS

What happened with my Azure VM Scale Set Instance IDs?

In my last couple of posts (here and here) I've covered moving from Azure VMs to running workloads on VM Scale Sets (VMSS).

One item you use need to adjust to is the instance naming scheme that is used by VMSS.

I was looking at the two VM Scale Sets I have setup in my environment and was surprised to see the following.

The first VM Scale Set I setup has Instances with these IDs (0 and 1):

VMSS 01

and my second has Instances with IDs of 2 and 3:

VMSS 02

I thought this was a bit weird that two unrelated VM Scale Sets seem to be sharing a common Instance ID source – maybe something to do with the underlying provisioning engine continuing on Instance IDs across VMSS on a VNet, or something similar?

As it turns out, this is entirely coincidental.

When you provision a VMSS and you set the "overprovision" flag to "true" the provisioning engine will build more Instances than required (you can see this if you watch in the portal while the VMSS is being provisioned) and then delete the excess above what is actually required. This is by design and is described by Microsoft on their VMSS design considerations page.

Here's a snippet of what your ARM template will look like:

"properties": {
"overprovision": "true",
"singlePlacementGroup": "false",
"upgradePolicy": {
"mode": "Manual"
}
}

So, for my scenario, it just happens that for the first VMSS that the engine deleted the Instances above Instance 1, and for the second VMSS the engine deleted Instances 0 and 1!

Too Easy! 🙂

Tagged , ,

When to use Azure Load Balancer or Application Gateway

One thing Microsoft Azure is very good at is giving you choices – choices on how you host your workloads and how you let people connect to those workloads.

In this post I am going to take a quick look at two technologies that are related to high availability of solutions: Azure Load Balancer and Application Gateway.

Application Gateway is a bit of a dark horse for many people. Anyone coming to Azure has to get to grips with Azure Load Balancer (ALB) as a means to provide highly available solutions, but many don’t progress beyond this to look at Application Gateway (AG) – hopefully this post will change that!

The OSI model

A starting point to understand one of the key differences between the ALB and AG offerings is the OSI model which describes the logical layers required to implement computer networking. Wikipedia has a good introduction, which I don’t intend to reproduce here, but I’ll cover the important bits below.

TCP/IP and UDP are bundled together at Layer 4 in the OSI model. These underpin a range of higher level protocols such as HTTP which sit at the stop of the stack at Layer 7. You will often see load balancers defined as Layer 4 or Layer 7, and, as you can guess, the traffic they can manage varies as a result.

Load balancing HTTP using a Layer 4 capable device works, though using Cookie-based session affinity is out, while a pure Layer 7 solution has no chance to handle TCP or UDP traffic which sit below at Layer 4.

Clear as mud? Great!

You might ask what does it mean to us in this post?

I’m glad you asked 🙂 The first real difference between the Azure Load Balancer and Application Gateway is that an ALB works at traffic at Layer 4, while Application Gateway handles just Layer 7 traffic, and specifically, within that, HTTP (including HTTPS and WebSockets).

Anything else to know?

There are a few other differences it is worth calling out. I will summarise as follows:

  • Load Balancer is free (unless you wish to use multiple Virtual IPs). Application Gateway is billed per-hour, and has two tiers, depending on features you need (with/without WAF)
  • Application Gateway supports SSL termination, URL-based routing, multi-site routing, Cookie-based session affinity and Web Application Firewall (WAF) features. Azure Load Balancer provides basic load balancing based on 2 or 5 tuple matches.
  • Load Balancer only supports endpoints hosted in Azure. Application Gateway can support any routable IP address.

It’s also worth pointing out that when you provision an Application Gateway you also get a transparent Load Balancer along for the ride. The Load Balancer is responsible for balancing traffic between the Application Gateway instances to ensure it remains highly available 🙂

Using either with VM Scale Sets (VMSS)

The default setup for a VMSS includes a Load Balancer. If you want to use an Application Gateway instead you can either leave the Load Balancer in place and put the Application Gateway in front of it, or you can use an ARM template similar to the following sample to swap out the Load Balancer for an Application Gateway instance instead.

https://github.com/Azure/azure-quickstart-templates/tree/master/201-vmss-windows-app-gateway

The key item to ensure is that the Network Interface Card (NIC) is configured for each VM in the Scale Set to be a part of the Backend Address Pool of the Application Gateway.

So there we have a quick overview of the Azure Load Balancer and Application Gateway offerings and when to consider one over the other.

Either the Azure Load Balancer overview or the Application Gateway introduction provides a good breakdown of where you should consider using one or the other.

Happy (highly available) days! 🙂

Tagged , ,

Deploying to Azure VMs using VSTS Release Management

I am going to subtitle this post “the missing manual” because I spent quite a bit of time troubleshoothing how this should all work.

Microsoft provides a bunch of useful information on how to deploy from Visual Studio Team Services (VSTS) to different targets, including Azure Virtual Machines.

In an ideal world I wouldn’t be using VMs at all, but for my current particular use case I have to use VMs so the above (linked) approach worked.

The approach sounds good but I ran into a few sharp edges that I thought I would document here (and hopefully the source documentation will be updated to reflect this in due course).

Preparing deployment targets

Azure FQDNs

Note: Please see my update at the bottom of this post before reading this section. While you can use IP addresses (if you make them static) it’s worth configuring test certs with the FQDN.

I thought I’d do the right thing by configuring the Azure IP of my hosts to have a full FQDN rather than just an IP address.

As I found out this is not a good idea.

The main issue you will run into is the generated certs on target hosts only have the hostname in them (i.e. azauhost01) rather than the full public FQDN (i.e. azauhost01.australiaeast.cloudapp.azure.com).

When the Release Management infrastructure tries to connect to a host this cert mismatch causes a fatal error. I didn’t spend much time troubleshooting so decided to revert to use of IP addresses only.

When using dynamic IP addresses the first Release Management action “Azure Deployment:Select Resource Group action” is important as it allows for discovery of all VMs and their IPs (i.e. no hardcoding required). This apprach does mean, however, you need to consider how you group VMs into Resource Groups to allow any VM in the Resource Group to be used as the deployment target.

Select Resource Group

Local permissions

I was running my deployments to non-Domain joined Windows 2012 R2 server instances with local administrative accounts and had opened the necessary port in the NSG rules to allow WinRM into the servers from the VSTS infrastructure.

Everything looked good on deployment until PowerShell execution on the target hosts resulted in errors due to permissions. As it turns out the error message was actually useful in resolving this problem 🙂

In order to move beyond this error I had to prepare each target host by running these commands at an admin command prompt on the host:

winrm quickconfig

Enable-PSRemoting

We could drop these into a DSC module and run that way if we wanted to make this repeatable across new hosts.

There is a good troubleshooting overview for this from Microsoft.

Wait! Where did my PowerShell script go?

If you follow the instructions provided by Microsoft you need to add a deployment Powershell script (actually a DSC module) to your Web App (their example uses “ConfigureWebserver.ps1” for the filename).

There is one issue with this approach – the build step to package the Web App actually ends up bundling the PowerShell inside of a deployment zip which means once the files are copied to your target VM the PowerShell can’t be invoked.

The fix for this is to add an additional build step that copies the PowerShell to the drops folder on VSTS which means the PowerShell will be transferred to the target VM.

Your final build definition should look like the below

Build definition

and the Copy Files task should be configured like this (note below that /Deploy is the folder in my solution that contains the PowerShell I’ve included for deployment purposes):

Build Step

Once you have done this you will find that the script is now available in the VSTS drops folder and can be copied to the VMs which allows you to execute it via the Release Management action.

Wrapping up

Once I had these changes in place and had made some minor path / project name tweaks to match my project I could run the process end-to-end.

The one final item I’ll call out here is the default deployment location of the solution on the target VM ends up being the wwwroot of your inetpub folder with a subfolder named ProjectName_deploy. If you map this to an Application in IIS you should be good to go :).

Update – 8 December 2016

After I’d been running happily for a while my Release started failing. It turns out my target VMs were cycled and moved to different public IP addresses. As the WinRM HTTP certificate had the old IP address in the remote calls failed.

I found a great blog post on how to rectify this situation though: http://www.dotnetcurry.com/windows-azure/1289/configure-winrm-execute-powershell-remote-azure-with-arm

Happy days!

Tagged , ,

Migrating resources from AWS to Microsoft Azure

Kloud Blog

Kloud receives a lot of communications in relation to the work we do and the content we publish on our blog. My colleague Hugh Badini recently published a blog about Azure deployment models from which we received the following legitimate follow up question…

So, Murali, thanks for letting us know you’d like to know more about this… consider this blog a starting point :).

Firstly though…

this topic (inter-cloud migrations), as you might guess, isn’t easily captured in a single blog post, nor, realistically in a series, so what I’m going to do here is provide some basics to consider. I may not answer your specific scenario but hopefully provide some guidance on approach.

Every cloud has a silver lining

The good news is that if you’re already operating in a…

View original post 926 more words

Azure Automation Runbooks with Azure AD Service Principals and Custom RBAC Roles

If you’ve ever worked in any form of systems administrator role then you will be familiar with process automation, even only for simple tasks like automating backups. You will also be familiar with the pain of configuring and managing identities for these automated processes (expired password or disabled/deleted account ever caused you any pain?!)

While the cloud offers us many new ways of working and solving traditional problems it also maintains many concepts we’d be familiar from environments of old. As you might have guessed, the idea of specifying user context for automated processes has not gone away (yet).

In this post I am going to look at how in Azure Automation Runbooks we can leverage a combination of an Azure Active Directory Service Principal and an Azure RBAC Custom Role to configure an non-user identity with constrained execution scope.

The benefits of this approach are two-fold:

  1. No password expiry to deal with, or accidental account disablement/deletion. I still need to manage keys (in place of passwords), but these are centrally managed and are subject to an expiry policy I define.
  2. Reduced blast radius. Creating a custom role that restricts actions available means the account can’t be used for more actions that intended.

My specific (simple) scenario is stopping all running v2 VMs in a Subscription.

Create the Custom Role

The Azure Resource Manager (ARM) platform provides a flexible RBAC model within which we can build our own Roles based on a combination of existing Actions. In-built Roles bundle these Actions into logical groups, but there are times we may want something a little different.

The in-built “Virtual Machine Contributor” Role isn’t suitable for my purposes because it provides too much scope by letting assigned users create and delete Virtual Machines. In my case, I want a Role that allows an assigned user to start, stop, restart and monitor existing Virtual Machines only.

To this end I defined a custom Role as shown below which allows any assigned user to perform the functions I need them to.

Let’s add this Role by executing the following PowerShell (you’ll need to be logged into your Subscription with a user who has enough rights to create custom role definitions). You’ll need to grab the above definition, change the scope and save it as a file named ‘vm-power-manager-customerole.json’ for this to work.

New-AzureRmRoleDefinition -InputFile vm-power-manager-customrole.json

which will return a result similar to the below.

Name             : Virtual Machine Power Manager
Id               : 6270aabc-0698-4380-a9a7-7df889e9e67b
IsCustom         : True
Description      : Can monitor, stop, start and restart v2 ARM virtual machines.
Actions          : {Microsoft.Storage/*/read, Microsoft.Network/*/read, Microsoft.Compute/*/read
                   Microsoft.Compute/virtualMachines/start/action...}
NotActions       : {}
AssignableScopes : {/subscriptions/c25b1c8e-1111-4421-9090-1a12d7012dd3}

that means the Role shows up in the Portal and can be assigned to users 🙂

VM Power Manager Role

Now we have that, let’s setup our Service Principal.

Setup Service Principal

Microsoft provides a good guide on creating a Service Principal on the Azure documentation site already so I’m not going to reproduce that all here.

When you get to “Assign application to role” hop back here and we’ll continue on without needing to dive into the Azure Portal.

For the purpose of the rest of this post, these are the parameters I used to create my Service Principal.

Name: Azure VM Start Stop SP
Sign-on URL / App URI: http://myvmautomation
Client ID*: c6f7c745-1234-5678-0000-8d14611e75f4
Tenant ID*: c7a48abc-1990-4fef-e941-a1cd55422e41

* Client ID is returned once you save your Application. Tenant ID comes from your Azure AD tenant ID (see Microsoft setup instructions referenced above).

Important: you will also have to generate and grab a key value that you will need to use as it is the password for the Service Principal. Don’t forget to grab it when it’s displayed!

Assign the Service Principal the necessary Azure Roles

# Assign our custom Role
New-AzureRmRoleAssignment -ServicePrincipalName http://myvmautomation `
                          -RoleDefinitionName 'Virtual Machine Power Manager' `
                          -Scope '/subscriptions/c25b1c8e-xxxx-1111-abcd-1a12d7012123'

# Assign the built-in 'Reader' Role
New-AzureRmRoleAssignment -ServicePrincipalName http://myvmautomation `
                          -RoleDefinitionName 'Reader' `
                          -Scope '/subscriptions/c25b1c8e-xxxx-1111-abcd-1a12d7012123'

We now have all the baseline mechanics out of the way – next it’s onto using this information in our Runbooks.

Asset Setup

Azure Automation has the concept of an Asset that can be one of six items: Schedules, PowerShell Modules, Certificates, Connections, Variables and Credentials.

These are shared between all Runbooks in an Automation Account and are extremely useful in helping you deliver generic re-usable Runbooks.

For this post we are going to create a new Credential using the following process.

Our Automation Account is called ‘Core-Services’ and is hosted in a Resource Group ‘rg-test-01’

$username = "c6f7c745-1234-5678-0000-8d14611e75f4"
$password = ConvertTo-SecureString -String "YOUR_SERVICE_PRINCIPAL_KEY" -AsPlainText -Force

$newCreds = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $username,$password

New-AzureRmAutomationCredential -Name "VMPowerServicePrincipal" `
                                -Description 'Service Principal used to control power state of VMs' `
                                -Value $newCreds `
                                -ResourceGroupName 'rg-test-01' `
                                -AutomationAccountName 'Core-Services'

This creates a Credential we can now use in any Runbook.

The sample PowerShell Runbook below shows how we do this using the Login-AzureRmAccount Cmdlet using the -ServicePrincipal switch.

I also specify a Tenant identifier (this is the Azure AD Tenant identifier from when you setup the Service Principal) and the Subscription identifier so we set context in one call.

The Tenant and Subcsription identifiers are held as Automation Account Variables which we read in at the start of execution (the pattern below allows you to override which Variables you pass in should you want to use different ones).

So there we have it – a way to perform VM power state management in an Azure Automation Runbook that uses a non-user account for authentication along with custom RBAC roles for authorisation.

Enjoy!

Tagged , , ,

Easy Debugging of PowerShell DSC for Azure Virtual Machines

I’ve been doing a lot of PowerShell DSC on Azure VMs recently, so I thought I’d share my experience in debugging custom DSC Modules when working in Azure.

My blog entry is over on the Kloud blog so head on over and have a read.

Tagged , , ,

Connection Options When Building An Azure Hybrid Cloud Solution

Kloud Blog

If your business is migrating workloads to Azure the chances are at some point you will probably want to create a form of private interconnect with Azure. There is more than one way to achieve this, so in this post I’ll take a look at what options you have and the most appropriate scenarios for each.

We’ll work through the connection types from simplest (and quickest to provision) to more complex (where you’ll need IP networking expertise and hardware).

Hybrid Connection

This is your baseline interconnect option and is tied to the BizTalk Services offering within Azure. At time of writing the only Azure-based services that can leverage Hybrid Connections are Web Apps (formerly Websites) and Mobile Apps (formerly Mobile Services).

Hybrid Connections are a great way to quickly get access to on-premises resources without the complexity involved in firewall or VPN setups. If you look at the official documentation

View original post 1,363 more words

Tagged

Azure’s G Series VMs – Prime Compute Only One Click Away!

Kloud Blog

I’m going to start this blog post by making one thing clear. My intent in writing this post is light-hearted – I had some spare time on my hands over a lunch break and I wondered what I could do with it. The result was this blog post :).

Ever since Microsoft announced their G Series Virtual Machines for Azure I’ve been looking for a good reason to fire one up and kick the tyres. Today while I was skimming through my Twitter feed I came across a tweet showing the time it took to calculate the trillionth prime number on a 16 vCPU Linux instance running on GCP.

As any good propeller head knows, the first rule of having access to massive raw compute is to put it to use solving mathematical challenges. This may take the form of a pure maths challenge like finding the n-th digit of…

View original post 602 more words

Tagged ,

Secure Remote Management Studio access to SQL Server on Azure IaaS

If you have ever provisioned a SQL Server instance running on Azure IaaS and not used a VPN solution you will find that by default you are unable to connect to it from a local Management Studio instance.  By default all Virtual Machines are wrapped by a Cloud Service which behaves to a degree like ingress Security Groups do on AWS.  In this blog post I’ll show you how you can open up a connection and then connect securely to it using SQL Authentication.

Note: making this change effectively opens your SQL Server up to traffic from the Internet though it is on a non-standard TCP port.  If you don’t want this you should consider using an Azure Virtual Network and a VPN to protect connections to / from SQL server and a known location or device.  Alternatively you could setup a bastion or jump host that you first RDP to before connecting to SQL Server.

Updated: The release of the Azure SDK 2.0 introduces the concept of ACL on exposed endpoints and the 2.1 SDK exposes the setting of these values via PowerShell (see Set-AzureAclConfig). Awesome!

When you provision a new Virtual Machine by default it will provide two default TCP endpoints: Remote Desktop (RDP) and PowerShell.  As a first step we need to open access to port 1433 – we can do this using on the following two methods:

1. Via the Azure Management Portal:

  • Click on Virtual Machines in the left navigation and select your shiny new SQL VM.
  • Click on ENDPOINTS in the top navigation.  You should see a view similar to below:

Azure VM Endpoints

  • Now click the + ADD button at the bottom and select “Add Endpoint”.
  • On the Add Endpoint page complete:
    • Name: SQL TDS
    • Protocal: TCP
    • Public Port: Random number > 49152 (and not already in use on this Cloud Service or VM)
    • Private Port: 1433.
    • Click Tick to save new endpoint.

2. Via Powershell with the Azure Powershell Module.  Note that you will need to setup your Powershell environment to know about and connect to your Azure Subcription.  More information on this topic can be found on MSDN.

Get-AzureVM -ServiceName "yourcloudsevice" -Name "yourvmhostname" |
Add-AzureEndpoint -Name "SQL TDS" -Protocol tcp -LocalPort 1433 -PublicPort 57153 |
Update-AzureVM

Now that you completed the above you can connect to your SQL Server using Management Studio and encrypt the connection.  Open Management Studio and in the Connect to Server make the following changes:

1. Under “Server name” put the Cloud Service Public Virtual IP (VIP) address of your VM (find it on the Dashboard screen for the VM), a comma and then include the Public Port you mapped previously.  Your resulting input should look like “123.123.123.123,57153”.

SQL Connection Dialog

2. Click on the Options>> button and on the Connection Properties tab select “Encrypt connection”.

SQL Connection Dialog

3. Finally we need to tell Management Studio to trust the SSL certificate on the server. Click on the “Additional Connection Parameters” tab and enter “TrustServerCertificate=true”. If you don’t do this you will get an error and be unable to connect using encryption.

You should find that you can now connect to the VM.

I had a look to see if you could use Windows Firewall to restrict the traffic coming into your SQL Server by remote IP but at first glance it looks like it’s not possible due to the NAT occuring at the cloud service interface.  I haven’t had time to inspect the TCP traffic to see what’s coming into the host but I suspect you can probably create a firewall rule to protect your machine, though as I said up front – use a VPN and Virtual Network if you really want to be protected.

Updated: The release of the Azure SDK 2.0 introduces the concept of ACL on exposed endpoints and the 2.1 SDK exposes the setting of these values via PowerShell (see Set-AzureAclConfig). Awesome!

HTH.

Tagged , , , ,