When a deployment fails, it is likely to leave an environment in an unhealthy state. A rollback strategy is required to get the environment back to a healthy state.
There are various options that can be considered as a rollback strategy.
- Option#1 : Undo the change by redeploying the previous release
The first option is to simply redeploy the previously successful release. This might work well for standalone applications.
Whenever an application depends on some external services or has a database involved, this approach does not work well. The dependent services might have upgraded and no longer compatible with the previous release. The database might have changed the schema making the previous release no longer healthy.
- Option#2 : Fix the issue, do another release
The second option is to simply do nothing. Something went wrong, troubleshoot it and fix it. Once we fix the issue, we can do another release.
This however means the environment would remain unhealthy for some time, as long as it takes for the fix to be ready and deployed.
- Option#3 : Understand what failed in the deployment and make a temporary change for the time being
Both the above options are both valid approaches but with some limitations. That brings us to a third option. While the fix is getting ready (option 2), make a minimum change to the environment to get it temporarily healthy.
We’re almost talking about option 2 being a major/minor release and option 3 being a hotfix like change to the application.
Let us look at how to implement option 3 using release management.
A deployment to an environment might fail when any of the tasks in the deployment workflow fails. In order to get the environment healthy again, we’ll look to execute a rollback script. The script itself would have the following requirement.
- Execute the script for the environment even when another task has failed.
With release management for Team Services, you can look to achieve this using a powershell task and mark it “Always Run”. This way the script shall always get executed, irrespective of the deployment being successful or failed.
Any of the tasks that are not marked “always run” shall be skipped in case a preceding task failed, but the always run task would be executed.
- Make selective changes to the environment based on the task that failed in the deployment.
Having the script executed always is only half the task. In order to avoid undoing anything that’s not required, we’ll need to know a bit more. First of all, we’ll need to detect if there was a failure at all. In case there was a failure, then what task failed.
Currently, there isn’t a way to know about these in the script. We’ll need to query the release from within the script for these. “Rollback powershell” task in the Release Management Utility Tasks extension makes this possible.
Implementing rollback script
Rollback powershell task helps you understand the history of the current deployment. An environment variable Release_Tasks is set before the rollback powershell script is run.
In the script, you’ll need to parse the value of Release_Tasks and include the rollback logic accordingly.
An example to access the task execution information is as follows.
$jsonobject = ConvertFrom-Json $env:Release_Tasks
Write-Verbose -Verbose “Error parsing Release_Tasks environment variable”
Write-Verbose -Verbose $Error
foreach ($task in $jsonobject | Get-Member -MemberType NoteProperty)
$taskproperty = $jsonobject.$($task.Name) | ConvertFrom-Json
Write-Verbose -Verbose “Task $($taskproperty.Name) with rank $($task.Name) has status $($taskproperty.Status)”
// Perform rollback action required in case $task.Name has status failed
Adding rollback task to deployment workflow
Once you install the Release Management Utility Tasks extension in your account, you’ll see a new task called “Rollback powershell” in the task catalog. For rollback, you’ll need to add this task to the workflow and mark it “always run”.
End to End Example
For an illustration here, we are going to take a simple example. The application deployment has two steps namely “deploy” i.e. copy static content from artifact to a folder and “configure” i.e. make config changes to point to the new location.
We’ll author rollback for cases when deploy fails i.e. copy the original files back.
The environment definition shall be the following. TargetFolderName is a configuration variable indicating the target folder for the application.
- Backup — We’ll perform a backup of the original files to use for rollback later
- BackupFolderName configuration variable is added to indicate the folder to be used for backup.
- PowerShell task is added with an inline script to copy files from target folder to backup folder.
- Deploy — Copy latest files from artifact to the target folder
- Powershell task is added with an inline script to deploy files from artifact to target folder
- Configure — Make configuration changes to the setup
- Powershell task is added and a configure script is called from the artifacts
- Rollback — Rollback the files from backup in case Deploy failed. Delete backup before exiting.
- Rollback powershell task is added to perform rollback of files if deploy task fails. Always run is enabled.
- The inline script is as follows.