Troubleshooting Private App Service Environments and App Deployments

If you've been following me for a while, you'll know of my appreciation for Azure App Services as a great way to host workloads using Platform-as-a-Service (PaaS) in Azure.

Most people deploy to App Service Plans that run on shared infrastructure, but there is an additional option to deploy your workloads on reserved compute and also into private networks, called App Service Environments (ASEs).

App Service Environments are a great way to host your demanding workloads and control their access via use of virtual network integration, but still have the benefits of PaaS.

In this post I will cover some of the issues you might come across when configuring and deploying apps to a private App Service Environment (ASE) and how to resolve them.

Why use an ASE?

If you look at the official documentation you'll see that App Service Environments help customers meet compliance, security and scale requirements and are best suited to scenarios covered by the following list from the documentation:

• Internal line-of-business applications.
• Applications that need more than 30 App Service plan instances.
• Single-tenant systems to satisfy internal compliance or security requirements.
• Network-isolated application hosting.
• Multi-tier applications.

The good thing with an ASE is you can deploy multiple App Service Plans onto the ASE and then host multiple applications in each Plan. Each Plan can host a mix of Function Apps, Logic Apps (Standard), Web Apps and Static Web Apps.

Consequently there is a bit more to deploying an ASE than to using App Service Plans which most public applications would use, particularly when you start looking at networking.

For the remainder of this post I am going to use a Function App as my deployment canary, but the same restrictions will apply to any of the application types I mentioned above. Some of the screenshots are of Logic App Standard deployments, but these run on top of Azure Functions.

Our ASE configuration

You can find a full quickstart sample that deploys a private ASE with appropriate DNS and networking configuration in the Azure Quickstart samlpes repostiory at Create an AppServicePlan and App in an ASEv3. I'd recommend going and have a read through that sample as a starting point if you are unfamiliar with this process.

Note: ASEs take a substantial amount of time to deploy - typically 2+ hours.

If you deploy the sample you'll end up with:

• 1 App Service Environment with an internal load balancer (ILB) attached to a virtual network subnet via delegation
• 1 Azure Private DNS zone for the ASEv3 instance that includes three records - '*', '@' and '*.scm'
• 1 'IsolatedV2' App Service Plan deployed to the ASE.
• 1 Web App deployed to the App Service Plan.

DNS configuration

First off we have the Inbound IP address of the Internal load balancer which will manage traffic into any Plan or Apps deployed into the ASE. You can see it in the middle of the image.

Then we have the private DNS zone for just this single App Service Environment. As you need to be able to use the DNS zone apex (basically the hostname of the ASE) you have to set DNS up this way so that the full DNS name of any deployed applications will resolve without you needing to add a DNS entry for every single one of them.

At this point you almost have a baseline setup , though there one required element missing which we'll look at next.

The magic storage account

App Service Apps (Functions, etc) use a Storage Account File share as a way to share files between Instances and persist files between Instance restarts. This is also heavily used by the Kudu (Advanced Tools) feature that is deployed automatically with each App.

This not well covered in official documentation, and gets no mention in setting up an ASE as it is not a direct requirement of the ASE.

You can read more about this requirement on the Kudu wiki. If you're not familiar with Kudu and you want to work heavily with App Services, you really need to learn about it by having a read of their wiki.

Kudu is the Swiss Army knife for deployments and debugging in App Services. Clearly the team agrees - that's why the icon for 'Advanced Tools' in the Azure Portal is a little blue Swiss Army knife 🙂.

Any application deployed to App Services has it's own Kudu instance which is served from the *.scm sub-domain for the app.

Why do I mention this storage account? Stay tuned!

Deployment Failures

Now the fun really starts! 🙃

You've deployed your ASE, an App Service Plan and a related a storage account. Now it's time to deploy the first application. For our sample, let's deploy a Function App.

Here's the bicep snippet to deploy a Function App. You can find the full bicep definition on a Gist I've created.

@description('Sample Function App')
resource sampleFunctionApp 'Microsoft.Web/sites@2021-01-15' = {
  name: functionAppName
  location: aseLocation
  kind: 'functionapp'
  properties: {
    siteConfig: {
      // Ensures that direct access from the Virtual Network is enabled
      publicNetworkAccess: 'Enabled'
      // Allows any IP address on your Virtual Network to access your Function App
      ipSecurityRestrictions: [
        {
          ipAddress: '*'
          action: 'Allow'
        }
      ]
      // Inherit the main site IP restrictions for SCM subsite
      scmIpSecurityRestrictionsUseMain: true
      appSettings: [
        {
          name: 'AzureWebJobsStorage'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccountName};EndpointSuffix=${environment().suffixes.storage};AccountKey=${sampleStorage.listKeys().keys[0].value}'
        }
        {
          name: 'WEBSITE_CONTENTAZUREFILECONNECTIONSTRING'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccountName};EndpointSuffix=${environment().suffixes.storage};AccountKey=${sampleStorage.listKeys().keys[0].value}'
        }
        {
          name: 'WEBSITE_CONTENTSHARE'
          value: toLower(functionAppName)
        }
        {
          name: 'FUNCTIONS_EXTENSION_VERSION'
          value: '~4'
        }
        {
          name: 'WEBSITE_NODE_DEFAULT_VERSION'
          value: '~18'
        }
        {
          name: 'FUNCTIONS_WORKER_RUNTIME'
          value: 'node'
        }
      ]
      ftpsState: 'FtpsOnly'
      minTlsVersion: '1.2'
    }
    serverFarmId: sampleAsePlan.id
    clientAffinityEnabled: true
    hostingEnvironmentProfile: {
      id: sampleAse.id
    }
  }
}

Let's go ahead and deploy this, assuming the storage account exists and the ASE and Plan match those from the quickstart sample above.

Failure Scenario 1: Azure Policy "No public endpoints" failure

You run your deployment and it fails when attempting to create the Function App resource. You inspect the reason for this you see the error:

Resource 'your-function-app-resource-name' was disallowed by Policy.

Reading the error message more you will also see:

Public network access should be disabled for PaaS services

Yes, you've guessed it, it's this property in your bicep:

publicNetworkAccess: 'Enabled'

Unfortunately, the meaning of this property changes depending on whether your ASE is public or private, but Azure Policy does not differentiate between these two configurations. You can find the Policy definition in the in-built policy definitions on GitHub.

Where does this setting show up in the Azure Portal? Under Networking for your deployed application, which is shown in the image below. The state in the image matches the above property setting of 'Enabled' which would be blocked for deployment by Azure Policy.

If you click on the Access restrictions label this is what you would see. In this configuration you have a functioning environment as your virtual network can access the deployed application's endpoints.

If you set the property to 'Disabled' to meet the Azure Policy your deployment will succeed, but then later steps will not. If you have deployed your bicep successfully and you open the restrictions screen this is what you would see. At this point your deployments will be broken. More on that below!

If you try to switch this setting in the Azure Portal by checking the box and clicking save, you will also be blocked by Aure Policy and receive the following error. Lucky you!

How to fix this?

After a lot of troubleshooting I opened an issue on the Azure Policy GitHub repository as the Policy in its current form breaks Function Apps and Logic Apps Standard deployed to private ASEs with direct network access required. There are different Policies for Functions and Logic Apps, and I suspect the same for Web Apps and Static Web Apps too.

How to do you get it to work in the meantime? You will need to add an Azure Policy exemption for any resource groups where you will deploy Function Apps, Web Apps, Logic Apps and Static Web Apps. This might be a different resource group to your ASE and App Service Plan.

So, on to the next one!

Failure Scenario 2: Bicep works, code deployment fails with 403 error

For this scenario, assume you have complied with Azure Policy around public access and have your networking with restrictions on and default settings (which is to allow no remote access).

Now, go ahead and run your build - the bicep will deploy you empty Function or Logic App, but when you attempt to deploy the actual code, potentially using zip deploy, you receive this message back:

Error: Failed to deploy web package to App Service. Ip Forbidden (CODE: 403)

The reason this occurs is, as you've probably guessed, the access restrictions (see last section) settings don't allow any connections which results in a HTTP 403 (Unauthorised) response, with the reason being... IP forbidden!

How do you fix this? You have to go back to the previous issue and allow direct access as you cannot specify IP address restrictions without enabling direct access. 🐔 meet 🥚!

Once you allow changes to be made to the Access Restrictions setting you can either allow any IP address, or limit those that can connect.

If you re-run your deployment you should find that it succeeds... unless you hit the next issue!

Failure Scenario 3: Bicep works, code deploymanet fails with 500 error

I've lived this dream, and I thought when I cracked the 403 issue that I was home and dry! Wishful thinking! 😂

So you've fixed your policy setting and have direct access allowed and then you re-run your build.

This deployment doesn't fail with a 403 IP address error, but instead gives you a 500 error (Internal Server Error).

At this point you think to troubleshoot what's going on using Kudu, so you head to your deployed application and select the Advanced Tools options to open Kudu.

After a few moments waiting for the Kudu main page to load you are granted by a site you may not have seen for while - a Yellow Screen of Death (YSOD)!

Even though you're amazed to have recevied a YSOD, the stacktrace did give you a hint as to what is causing the issue.

The network path was not found.

🤔

Return of the magic storage account

Hmmm. You're lucky because I've already given you a hint about why this may not be working. Yes, that's right, the magic storage account and its use for shared storage.

But you have a storage account, and the bicep deployment had already interacted with it, so why fail now?!

First you need to consider that Bicep utilises the Azure Resource Manager (ARM) API which does not live in your Virtual Network and is not subject to access restrictions.

The storage account, however, is attached to your virtual network via a group of private endpoints and is subject to any Network Security Group rules you put in place. This means any resource attempting to use the storage endpoints / APIs directly will be impacted if access is blocked.

So what is causing this 500 internal server error? Network Security Groups (NSGs)! Yay!

App Service Instances access Azure Files (which live on Azure Storage accounts) via SMB3 which uses TCP port 445 (no, not 443). If you have this port and protocol blocked on your virtual network, or within subnets that contain the ASE and/or the Azure Storage private endpoints, your result is this failure.

The fix is to ensure you have appropriate NSGs defined that allow SMB3 to flow between your ASE and Storage.

Deployment successful!

At this stage you should be able to resubmit your deployment and have it run end-to-end with no issues!

Hopefully you've learnt a bit about how App Service Environment, Azure Policy and Azure Storage can be combined, and how to troubleshoot issues that can arise.

If there's something you've seen that I haven't covered, but that you think would be useful please leave a comment below.

In the meantime - happy days! 😎