Tips and tricks on Azure DevTest Labs artifacts development and debugging

Acknowledgement: This post is contributed by Roger Best, Senior Software Engineer in Customer Success Team. He’s been working on a solution to help our customers to adopt Azure DevTest Labs.

I’m recently working with some customers getting their DevTest labs up and running. During that, I spent some time developing and debugging artifacts, and figured out several useful things that accelerate my artifact development experience. In this article, I’d like to share these tips and tricks, and hopefully they will be helpful for you to develop your artifacts.

A little bit about Artifacts

In case you are not very familiar with DevTest Labs artifacts yet, here’s a little bit useful information you’d better know before you start creating your own custom artifacts.

Artifacts is a very important and useful concept in DevTest Labs. They are used to deploy and configure your application after a VM is provisioned, which can be either of the following:

  • Tools that you want to install on the VM - such as agents, Fiddler, and Visual Studio.
  • Actions that you want to run on the VM - such as cloning a repo.
  • Applications that you want to test.

An artifact contains a definition file (Artifactfile.json) and optionally scripts to be executed. DevTest Labs loads artifacts from GitHub or VSTS Git repository. By default, the public repository is connected to a lab at creation. You can also add your own custom artifact repository using a personal access token.

At very high-level, DevTest Labs artifacts is a thin wrapper to execute Azure VM custom script extension or any other Azure VM extensions, and enables an end-user friendly experience on top of it. The artifacts that execute the custom script extension run your PowerShell or bash scripts defines “runCommand” in Artifactfile.json as below:

"runCommand": {

"commandToExecute": "powershell.exe -ExecutionPolicy bypass -File startChocolatey.ps1 -PackageList 7zip.install"


Instead, artifacts that execute any other Azure VM extensions defines “runAzureVMExtension” property (Check out this sample that uses Chef Client extension).

Developing custom artifacts follows the steps below:

  • Setting up the repository(s)
  • Scripting
  • Debugging deployment

You can add your own custom artifact repository using a personal access token. In the following sections, we will walk through the tips and tricks I learned from my experience for each of the steps.

Set up the repository(s)

Let’s start with setting up the repository. Currently the DevTest Labs supports GitHub and VSTS Git repositories as the source to load artifacts. It’s recommended to set up the primary lab to be connected to the “master” branch of the repo, then create a second lab as the development lab connected to the “dev” branch from master. As DevTest Labs supports multiple custom artifact repos connected to a lab, you can even setup another branch from “dev” and have it connected to the development lab. Setting up the environment, in this fashion, has a couple of benefits:

  • Easier to confirm functionality without extra work before making the appropriate pull requests into master,
  • Development without adversely affecting the production lab, and
  • The capability to manage access from the lab.

The Azure Portal DevTest Labs artifact blade helps by displaying which repo the artifact is in, so that existing artifacts being modified can be easily identified and debugged.

Script Debugging

There are several things worth keeping in mind for this part:

Create custom baseline image

So now let’s focus on the scripts and files that are being used to define the artifact. Given the iterative nature of debugging, it’s recommended to build a DevTest Labs VM with the artifacts or tools for development, then creating a custom image of that machine. It speeds things up when you need to get back to a known state quickly.

Keep scripts focused

Another tip to set yourself up for future success is to make the scripts more granular within an artifact. DevTest Labs today doesn’t support sharing scripts between artifacts. If you have multiple artifacts and part of the logic in the artifact scripts can be reused, It’s recommended to break up scripts into smaller reusable components, and use the same script names for those reusable components between artifacts. This helps you to easily find all the artifacts that contain a script that needs to be updated. When DevTest Labs allows you to share scripts across artifacts in the future, it also makes it easy to move the script to a common location.

Detailed custom logging

Custom logging within the scripts is key, these scripts will be run using Azure VM Custom Script Extensions (CSE). If possible, it’s recommended to store all logs to one common location like “C:\ArtifactLogs\...”. Otherwise, it will be very hard to look at different locations to find the files. Cleanup is easier when all the logs are at the same location, especially when your artifacts deal with installations that require reboots. Normally the recommendation would be to use the Temp environment location to store logs, but once CSE has finished that location no longer exists, making it difficult to review failures.

Handling machine reboots

Azure VM Custom Script Extension doesn’t handle rebooting. You must setup up scripts to be executed before and after the reboot. If possible, force the program that will be installed by the artifact to not restart, and then in the script, force a machine restart as the last step of the artifact. Managing the reboot in this way sets up a more consistent state for the next artifact and easier debugging when dealing with multiple artifacts being executed.

Where are artifact files go?

Finally, artifact deployment. there are different stages for the deployment of artifacts. The first is to get the DevTest Labs to load the artifact successfully. The ArtifactFile.json is key, as DevTest Labs will ignore the artifact if there’s any issue in this file. At this point, you may run into a couple of questions like below, and here are the answers:

Where can I find deployed in a VM?

Once the artifact is available, we will need to confirm that the correct script was downloaded and executed. The Azure VM Custom Script Extensions have logs in Windows VMs at “<RootDrive>:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtensions\1.8”. Specifically, the CustomScriptHandler.log has detailed information on what occurred. For Linux machines the log is located at “/var/log/azure/Microsoft.OSTCExtensions/CustomScriptForLinux/1.0/extension.log”.

For Windows systems, the artifacts are stored locally at “<RootDrive>:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Downloads”, so you can review the scripts being executed.

Why don’t my latest artifact updates take effect in the VM?

Sometimes after you update artifact files and reapply it to the VM, you may find the new changes are not applied to the VM. This is caused by a small delay between commit of the code to the repo and when the DevTest labs gets the changes. When an artifact is selected to apply to a VM, DevTest Labs uploads the artifact files from the source control to Labs’ storage account. DevTest Labs sets up multiple storage accounts at lab creation for different purposes. The artifact files are uploaded to the storage account with the “artifacts” container.

If it doesn’t look like the changes you made executed, check the downloaded file. To force DevTest Labs to retrieve the scripts from the repo, remove the scripts from the Azure blob storage where artifact files are uploaded by using either the Azure Portal or Visual Studio Cloud Explorer.

One more thing

Before this article is published, I heard that Azure DevTest Labs team is working on improving the artifacts debugging experience, including exposing all the logs to Azure portal. That will make it even easier to create your own custom artifacts.

Additional resources

Below is additional information about custom artifacts in the DevTest Labs:

If you have any feedback for DevTest Labs, please submit your idea or vote others at its feedback forum.

If you have any questions, please check out the MSDN forum.

Hope this helps.

Roger BestRoger Best, Senior Software Engineer

Roger is part of the Visual Studio and .NET engineering team focused on Visual Studio and Azure customers. He has been at Microsoft for 18 years, focusing on developer technologies for the past decade or so. In his spare time, he watches too many movies, and tries to survive triathlons.

Comments (0)

Skip to main content