Running tests in parallel using VSTS Release Management

  • Article

Release Management now supports running the tasks inside a phase in parallel on multiple agents.  In this blog post, I will walk you through how this can be leveraged to run tests in parallel.

Scenario:  I have a dotnet core project. I want to test it against Windows, Linux, iOS. Additionally, I want these tests to run in parallel.

Step 1:  Create a new variable TargetOS variable and initialize it with Windows_NT ,linux ,darwin.

targetosvariable

Step 2. Add two dotnet core tasks for restore and test.

dotnetcoretask

Step 3: Select Multi-Configuration from Type of parallelism.

Step 4: Specify Multipliers as TargetOS and set Maximum number of agents as equal to more than the OS in TargetOS variable.

If you have lots of runtimes and want to run those in parallel, but you want to limit the concurrency so that you do not starve another build/release for an agent. Then you should set Maximum number of agents to the value equal to the desired number of agent it is allowed to take simultaneously.

Step 5:  In demand specify Agent.OS -equals $(TargetOS)

Depending on the number of OS specified in TargetOS variable,  test runs will be created. Here three test run will be created each having the corresponding value of variable TargetOS variable. For example, the test run for Windows_NT will have TargetOs as Windows_NT.

We use this variable in demand to target appropriate agent. For Windows demand will evaluate to  Agent.OS -equals Windows_NT. This test will run on the agent with the capability Agent.OS as Windows_NT.

Also, make sure that the agent capabilities match the demands after the variable is replaced.

Step 6: Queue the release

parallelexecutionlist

This will create three test run and each will target agent with respective OS. The first test run is for iOS, second for Windows and third for Linux. We can check this in the name of Multiplier Node.

If your tests take very long to run and you want to cut the time, there is an upcoming version of VS Test task which can slice the tests in a single test run, each of which can run parallelly.  To use this new task, you can use Multi-Agent parallel execution. If you chose to run release with Parallel execution as Multi-Agent, the test is simply duplicated to run on multiple agents and each agent pull the appropirate slice from the server.