This is the 2nd post in a series focused on leveraging DevTest labs to automate the creation of Custom Images in Azure. Please refer to the 1st post for an overview of the solution, Introduction: Get VMs ready in minutes by setting up an image factory in Azure DevTest Labs.
Today we will cover all the preparations needed to run the Image Factory from Visual Studio Team Services (VSTS) up through creating a virtual machine. (highlighted section in the diagram below)
Important note: Any orchestration engine will work! VSTS is not required, it’s just happens to be what we’re using for this blog series but the Image Factory is run using Azure PowerShell scripts, so it could be run manually, with Windows Task Scheduler, other CI/CD systems, etc.
Create the DevTest Lab for the Image Factory
The first step in setting up the Image Factory is to create the DevTest Lab in Azure. This is the ‘Image Factory’ lab where we create the virtual machines and save custom images. This lab is considered part of the overall Image Factory process. (the right side of the diagram above). Once you create a DevTesT Lab, make sure to save the name since you’ll need it later!
The Scripts and Templates
The next step in adopting the Image Factory for your team is to understand what’s available. The Image Factory scripts and templates are available publicly in the DevTest Labs GitHub Repo. Here is an outline of the pieces:
Image Factory <dir>
└ Configuration <dir> <-- The ‘inputs’ to the image factory
└ GoldenImages <dir> <-- JSON files that represent the definitions of custom images
└ Labs.json <file> <-- File where teams ‘sign up’ to receive specific custom images
└ Scripts <dir> <-- The ‘engine’ for the image factory
It’s a lot of PowerShell code and ARM templates to digest all at once, but we’ll be going through each of the key steps (create virtual machines, save images, distribute images and retention policy) in subsequent posts.
Create a VSTS Team Project
The reason we’re using VSTS in our sample is because we can store the source code, run the Azure PowerShell in one place, we can schedule recurring runs to keep images up to date, and there are good facilities for logging the results to diagnose any issues. Using Visual Studio Team Services isn’t a requirement however, you can use any harness/engine provided it can connect to Azure and can run Azure PowerShell!
To get started, just create a free account in VSTS! Visit https://www.visualstudio.com/ and click “Get started for free” right under “Visual Studio Team Services”. You’ll need to choose a unique account name and make sure to choose to manage code using Git. Once this is created, save the URL to your team project! It will look something like this: https://<accountname>.visualstudio.com/MyFirstProject
If you have an existing VSTS account or project you would like to use instead, no problem! Just skip this step on creating the VSTS account.
Check in the Image Factory to the Git Repo
All the PowerShell, templates and configuration for the Image Factory are located in the public DevTest Labs GitHub repo here: https://github.com/Azure/azure-devtestlab/tree/master/Scripts/ImageFactory .
The fastest way to get the code into your new team project is to “Import a repository”. This will pull in the whole DevTest Labs repository (so you’ll get extra docs, samples, etc.). To do this, visit the VSTS Project you created above (URL looks like https://<accountname>.visualstudio.com/MyFirstProject ) , click “Import a Repository”, enter the clone URL for the DevTest Labs Repo: https://github.com/Azure/azure-devtestlab , and click “Import”.
Now we have all the files checked in! If you decide to only check in exactly what’s needed (the Image Factory files), please follow the steps here to clone the Git repo and push only the files needed located in the “scripts/ImageFactory” directory.
Create a Build and Connect to Azure
At this point we have the source files stored in a Git repo in TFS and now we need to setup our pipeline to run the Azure PowerShell. We have lots of options here, we only need to execute Azure PowerShell! We are using a build definition for simplicity in this blog series, but this works with VSTS Build, VSTS Release (single or multiple environments), other execution engines like Windows Task Scheduler or any other harness that can execute Azure PowerShell.
NOTE: One important point to keep in mind that some of the PowerShell files take a long time to run when there are a lot (10+) custom images to create. Free Hosted VSTS Build/Release agents have a timeout of 30 min, so we can’t use the free hosted agent once you start building many images. This timeout challenge applies to whatever harness you decide to use, it’s good to verify up front that you can extend the typical timeouts for long running Azure PowerShell scripts. In the case of VSTS, you can either use Paid Hosted Agents or use your own build agent.
To start, click on the “Setup Build” button on the homepage of your VSTS Project:
Next, we select an “Empty” build definition, click “Apply” to create your build! At this stage you can choose “Hosted” for the build agent. I also edited the title of my build to be “Build and Deliver Images to DevTest Labs”. Don’t forget to save!
Configure the Build Variables
To simplify the command line parameters, we encapsulate the key values that drive the image factory to a set of build variables. Click on the “Variables” tab and you’ll see a list of several default variables. Here’s the list of variables to enter in to VSTS:
|/Scripts/ImageFactory/Configuration||This is the full path in the repository to the “Configuration” folder. If you imported the whole repo above, the value to the left is correct. Otherwise update to point to the Configuration location.|
|MyImageFactory||The name of the Azure DevTest Lab used as the factory to produce images. If you don’t have one,
! Make sure the Lab is in the same subscription that the Service Endpoint has access to.
|1||The number of images you want to ‘save’ of each type. Good default is 1.|
|*******||The built in admin account password for the Virtual Machines. This is a transient account so just make sure it’s secure. Click the little lock on the right to ensure it’s a secure string.|
|ImageFactoryUser||The built in admin account username for the Virtual Machines. This is a transient account.|
|30||The timeout we should wait for regular Azure operations.|
|39df6a21-006d-4800-a958-************||The Subscription Id where the Azure DevTest Lab exists and that the Service Endpoint has access to.|
|Standard_A3||The size of the Virtual Machine to use for the “Create” step. The VMs created are transient, so it just needs to be of a size where there’s enough
Subscription Cores Quota
a size that’s enabled in the DevTest Lab
Connect to Azure
The next step is to setup something called a “Service Principal”. This is an identity in Azure Active Directory that enables the VSTS Build Agent to operate in Azure on the user’s behalf. To do this, we’ll get started adding our first Azure PowerShell Build Step.
Click “Add Task” and search for “Azure PowerShell”. Once you find it, click “Add” to add the task to the build. When you do this, you’ll see the task appear on the left side as added.
The fastest way to setup a Service Principal is to let VSTS do it for us. To do this, click on the task we just added, and for “Azure Connection Type” choose “Azure Resource Manager”. Once you’ve done this, click on the “Manage” link to setup the service principal. NOTE: if you already have a service principal setup, you can skip to the next step! This blog post has all the information you’ll need to get this working: https://blogs.msdn.microsoft.com/visualstudioalm/2015/10/04/automating-azure-resource-group-deployment-using-a-service-principal-in-visual-studio-online-buildrelease-management/ . When you click the “Manage” link you’ll land in the right place in VSTS (2nd screenshot in the blog post) to setup the connection to Azure. Just make sure to choose “Azure Resource Manager Service Endpoint” when setting this up.
Complete the “Create Virtual Machines” Build Task
Finally, we will complete the build task we started above. If you click on the build task, you’ll see all the details on the right pane that should be filled in. First, let’s name the build task “Create Virtual Machines”. Choose the Service Principal created above by choosing “Azure Resource Manager” and then the Service Endpoint. For “Script Path’, click the “…” on the right and navigate to “MakeGoldenImageVMs.ps1” script. Script Parameters should look like this:
-ConfigurationLocation $(System.DefaultWorkingDirectory)$(ConfigurationLocation) -DevTestLabName $(DevTestLabName) -vmSize $(VMSize) -machineUserName $(MachineUserName) -machinePassword (ConvertTo-SecureString -string '$(MachinePassword)' -AsPlainText -Force) -StandardTimeoutMinutes $(StandardTimeoutMinutes)
Verify it works - Queue the Build!
Let’s verify that we have everything setup correctly by queuing up a new build. While the build is running, switch to the Azure Portal and click on “All Virtual Machines” in your Image Factory DevTest Lab to confirm that everything is working correctly. You should see three virtual machines get created in the lab!
The first step in setting up the Image Factory based on Azure DevTest Labs is complete! In the next post in the series, we’ll get those Virtual Machines generalized and saved to Custom Images, then distributed to all your other Azure DevTest Labs, retention policies on images and finally cleanup.
Please let us know how we can continue to improve DevTest Labs by sharing your ideas and suggestions at the DevTest Labs feedback forum.
If you run into any problems with the features or have any questions, we are always ready to help you at our MSDN forum.
|Peter Hauge, Principle Software Engineer
Peter is part of the Visual Studio and .Net engineering team focused Visual Studio and Azure customers. He has been at Microsoft for 15 years, working on developer technologies throughout that time.