Zero-downtime deployments using VSTS and Azure

Using VSTS in combination with the power of Azure to do a deployment without any downtime.

by | May 2, 2017 | Azure IaaS, VSTS and TFS

Introduction

With continuous integration and continuous deployment is getting more popular; the amount of releases and deployments increases. In a classic IT environment those deployments need to be scheduled in a maintenance window. During the maintenance window the availability of the application is not guaranteed. This can be solved by using “zero downtime deployments.”

There are multiple “zero downtime deployment” scenario’s possible in Azure:

  • Via App Services – In app services you can define slots in which the application is installed. It is possible to create a production slot and a pre-production slot. When you have deployed your application to the pre-production slot, it is possible to swap the pre-production and production slots (so pre-production becomes production).
  • Via Azure IaaS – Not all applications are able to run in App Services. For those applications, you can use Azure IaaS (Azure Infrastructure as a Service). In this scenario, a load balancer gets in front of the production VMs. During the deployment, a new set of VMs will be deployed. The application gets installed on the newly created VMs. The next step is to add the VMs to the load balancer, and remove the old ones.

In this blogpost, I will focus on the Azure IaaS scenario. The code that I use to demonstrate the zero-downtime scenario is available via GitHub.

Not for all applications

Please note that a zero-downtime deployment is not suited for every application. The application must be able to run with two different versions at the same time. When the new VMs are connected to the load balancer; the old ones are still connected too. There are a couple of ways to make an application resilient to this:

  • Working with feature flags/toggles. With feature flags, it is possible to enable or disable a part of the application (feature). When the application gets installed, only the featuraes that are available and enabled on the old version of the application are enabled. Later, when the old VM’s have been disconnected from the load balancer, the new features can be enabled.
  • Make sure that the data source of the application is build that new and old versions of the applications can consume the data source. You can achieve this by having policies as: a table, field, schema etc. will never be deleted. Only new fields can be added.
  • Make sure that session data is placed on a data source that is available for the new and the old version of the application. For example; session data can be saved on a file share or in a database.

Infrastructure as a Code

When working with zero downtime deployments, not only the application gets deployed, but also the infrastructure and VM’s that are required to run the application. For Azure, we can define the infrastructure and VMs in ARM templates and PowerShell files. These files are “code.” ARM templates are declarative. This means that the template describes the expected situation of the resources. The templates and PowerShell script can be saved in a VSTS repository (for example: a GIT repository).

When building ARM templates; you can choose to have all resources in one template file or split them in separate template files. It is possible to build your ARM templates in Visual Studio; for each template a project is created. My advice is to create separate projects for each type of infrastructure components that you want to deploy. In my case I created projects for:

  • Virtual network – This project contains the ARM templates for the Virtual Network.
  • Storage – This project contains the ARM templates for the Storage
  • Availability set – This project contains the ARM templates for the Availability Set
  • VMs – This project contains the ARM templates for the VMs + NICs
  • Load balancer – This Project contains the ARM template for the Load Balancer
  • Scripts – This project contains the scripts that are required to do a zero-downtime deployment. Scripts are needed to add the new VM’s to a load balancer, and remove the old VMs

In the ARM templates that I create, I always make sure to output the name of the created resources. This can be very helpful when deploying multiple templates in a row. For example; when deploying a storage account and a VM (which uses that storage account to store its VHD); the VM deployment needs to have the storage account name as input parameter.

"outputs": {
    "created-storage-account-name": {
        "type": "string",
        "value": "[variables('NewStorageAccountNameUnique')]"
    }
}

VSTS Build Automation

Infrastructure code can be saved in a repository next to the application. By doing this; you can make the infrastructure code part of the application release. When an application code change requires an infrastructure change; you can easily apply that change in the infrastructure code. When the application gets released; the infrastructure change also gets released. A first step to get the infrastructure released with the application, is by making the ARM templates available as build output of the application.

The VSTS build task sequence is responsible for the compilation, testing and packaging of the application. This process needs to be used to package the ARM templates and save them as build output.

When using Visual Studio as repository; the “Copy files” task can be used to gather the ARM templates out of the projects, and copy them to a staging directory. This staging directory can then be added to the build output.

 

Release Management

The most of the magic happens in Visual Studio Team Services Release Management. During a release the application and infrastructure gets deployed for an environment. An environment can be seen as a stage in DTAP (development, test, acceptance, production). VSTS uses some kind of task sequences to have certain actions executed during the release.

The VSTS release task sequence is the key component in this zero-downtime scenario. It executes tasks which are responsible for a certain action during the build. It also can hold variables that are used during the release process. In case of this zero-downtime scenario; the task sequence is used to deploy Azure ARM templates, get output of Azure deployments (and save them to variables) and execute PowerShell scripts.

During a release; three types of tasks are used:

  • Azure Resource Group Deployment – This task is used to deploy an ARM templates
  • ARM Outputs – This task is used to gather output information from the ARM template that has been deployed earlier. The output is saved in VSTS variables that can be used in the next tasks.
  • Azure PowerShell – This task is used to run PowerShell scripts. For the zero-downtime deployment only two scripts run:
    • A script to attach the VMs to the load balancer
    • A script to remove the old VMs

 

VSTS Release Task Sequence Build-up

In this demonstration scenario, I only have one environment setup. In this environment, the first things that are happing are the deployments of the prerequisites needed to deploy a VM. In my case those prerequisites are a VNET, Storage Account and Availability Set. For the deployment of the resources I use the “Azure Resource Group Deployment” task. Directly after each deployment task, I use the “ARM Outputs” task. This task will gather the output information of the ARM deployment that just has been finished. This information (defined in the output section of the ARM template) is saved as variable in the VSTS release process.

Right after the deployment of the prerequisites, the virtual machines are deployed. For the deployment of the VMs the VSTS variables that are created via the “Output ARM” tasks are used. This makes sure that the VM gets deployed in the availability set, storage account and VNET that are created earlier. For the machine name, I use the release name. By doing this, the name of the VM’s is unique (for every release) and I can easily identify the VMs in Azure.

In the next task, the load balancer gets deployed. The deployment only deploys the load balancer; the VM’s are connected via a PowerShell task that gets executed after this task. In the last PowerShell task, old VM’s are removed.

Please note that the task sequences that are displayed in this blogpost only show the zero-downtime scenario. There is no application deployment happening here. A logical place to do application deployments is right after the VMs are deployed.

 

VSTS Release Variables

A lot of variables are used in this release scenario. In VSTS there are three different kind of variables:

  • System Variables – Those variables are defined by VSTS; we cannot edit them.
  • Release Variables – Variables that are available during the whole release process.
  • Environment Variables – These variables are only available for a certain environment.

The system variables are used to generate the name of the VMs that are rolled out during the release. In the demonstration for this blogpost; no release variables were used at all. I only used environment variables. I used them since the variables are only valid for a specific environment; in an other environment the variables would be different. Please don’t get confused with the environment variables that live in Windows. With environment variables I mean variables that are only available during the deployment of a certain environment.

 

Triggering a release

When a release gets triggered in VSTS, the task sequence will run. All templates and scripts will run.

Debugging a failed release

In some cases a release might not work out as expected. VSTS has a great log in which you can see the output of all tasks that have ran during the release process. The most errors/failures that I encounter are well described in the VSTS release log.

Code available via GitHub

It can be very hard to setup this scenario from scratch. To make it easy, I made the code and artifacts that are needed to test this scenario available via GitHub. You can find the sources here.

To import the build definition in your VSTS account, you need to have “Export/Import Build Definition” plugin. This plugin is available for free in the VSTS Marketplace

Share This