Parallel Test Execution

An early post on Parallel Test Execution drew attention to its subtle semantics. Three considerations directly contributed to that (1) Reach (2) Composability (3) Non-disruptive roll out.

The Visual Studio Test Platform is open and extensible, with tests written using various test frameworks and run using a variety of adapters. To reduce on-boarding friction, the feature ought to work on existing test code. It especially needs to work on existing MSTest framework based test code – there is a huge corpus of such tests already written, and it would be unrealistic to expect users to go in and update their test code just to take advantage of the feature. The feature must acknowledge that adapters for frameworks like NUnit, and xUnit.net already enable parallel test execution. And importantly it must not break existing test runs. Accordingly, the feature took shape as follows:

  • Parallel Test Execution is available to all frameworks.
  • It is available from within the IDE, the command line (CLI), and in VSTS.
  • From within the IDE it is available from all launch points (Test Explorer, CodeLens, various “Run” commands, etc.).
  • The feature composes with test adapters/frameworks that already support parallel execution.
  • The feature has a low-friction on-boarding experience – one that requires no changes to existing test code.
  • The feature meets the test code where it is.
  • The feature is OFF by default – users have to explicitly opt-in.

The Solution at Visual Studio Update 1

Parallel test execution leverages the available cores on the machine, and is realized by launching test execution on each available core as a distinct process, and handing it a container worth of tests (assembly, DLL, or relevant artifact containing the tests to execute) to execute. The unit of isolation is a process. The unit of scheduling is a test container. Within each container, the tests will be executed as per the semantics of the test framework. If there are many such containers, then as processes finish executing the tests within a container, they are handed the next available container.

In effect a coarse-grained level of parallelism is supported. The test platform intentionally leaves fine-grained control over parallelism to the framework of choice – i.e. it composes over it. Both levels parallelism can co-exist.

The feature is OFF by default.

The .runsettings file is the artifact used to configure how the tests get run, and is common in the IDE, the CLI, and the VSTS workflows. The feature can be turned ON by authoring a .runsettings file with an entry for MaxCpuCount, and associating that with the test run.

processparallel
The value for MaxCpuCount has the following semantics:

  • ‘n’ (where 1 <= n <= number of cores): upto ‘n’ processes will be launched.
  • ‘n’ of any other value: The number of processes launched will be as many as the available cores on the machine.

Typically, a value of 0 indicates that up to all of the available free cores may be used.

Guidance for leveraging Parallel Test Execution

As mentioned earlier, not all test code already written might be done so in a manner that is parallel-safe. For pure unit tests, it should just work. For other kinds of tests, you will need to experiment a little to see if they are assuming exclusive use of global resources, and refactor/rearrange them appropriately. In general, the following iterative approach may be used to leverage the feature:

Partition tests in terms of a taxonomy as follows:
(1) “Pure Unit Tests” (can run in parallel)
(2) Functional Tests that can run in parallel with some modifications (e.g.: 2 functional tests trying to create/delete the same folder can be ‘fixed’ to remove the assumption that they have exclusive use to the folder).
(3) Functional Tests that cannot be modified to run in parallel (e.g.: 2 Coded UI tests doing mouse actions on the desktop, 2 functional tests writing to the Bluetooth/IR port, etc.)

Gradually evolve the partitioning as follows:
a) Run the tests in parallel, and see what tests fail and classify them into (2) or (3) above.
b) For tests in (2), fix them so that they can run in parallel.
c) For tests in (3), move them out into a separate test run (where parallel is OFF by default).

Limitations

Parallel Test Execution is not supported in the following cases:
(1) If the test run is configured using a .testsettings file.
(2) It is not supported for test code targeting the Phone, Store, UWP application platforms.

How the feature evolved at Visual Studio Update 2

We received feedback that enabling parallel test execution using .runsettings was not very discoverable. Also, it is likely that some tests might fail when run in parallel, and our guidance on how to leverage the feature required the user to be able to separate out such tests in an iterative manner – an exercise that would require switching between parallel and non-parallel execution multiple times during one or more sessions. Therefore, we evolved the feature along the following lines:

  • Make the feature more easily discoverable.
  • Make it easy to turn the feature ON/OFF.

Starting with Visual Studio 2015 Update 2, we added the following alternative ways to enable Parallel Test Execution in the IDE, CLI and VSTS:

IDE
Parallel Test Execution is surfaced as a button on the Test Explorer.

parallelInTE
This is an ON/OFF setting.
If ON, the value of MaxCpuCount is set to 0.
The value is persisted with the solution.
The value is merged with any .runsettings file (if one is associated) just before a run begins.
The only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.

VSTS
In VSTS, “Execute Tests in Parallel” is surfaced as a checkbox in the VSTest task.

parallelInCI2
If ON, the value of MaxCpuCount is set to 0.
The value is merged with any .runsettings file (if one is associated) just before a run begins.
The only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.

CLI
vstest.console.exe supports a /Parallel command line switch.
If the switch is thrown, the value of MaxCpuCount is set to 0.
The value is merged with any .runsettings file (if one is associated) just before a run begins.
The only way to tweak the value of MaxCpuCount is by explicitly adding a .runsettings file.

Merging with .runsettings

But what if you already had a .runsettings file with MaxCpuCount? Well, the following tables summarize how the .runsettings will be merged:

IDE
TEmerge

CLI
CLImerge

VSTS
VSTSmerge

Summary

Parallel Test Execution is one of the key features we have shipped as part of the “Efficient Execution” theme, and we hope that this post has shed some light on the considerations that shaped it.

We are eager to know more about your experience with using this feature. Your inputs have informed the evolution of this and other features in the Test Plattform, so please keep the feedback coming. – post them as comments on this post, or using Visual Studio’s “Report a Problem”/”Provide a Suggestion” feature, or using connect, or twitter.

Looking forward to hearing from you.

6