VSTest task dons a new avatar – testing with unified agents and phases

Visual Studio Test (VSTest) and the Run Functional Test (RFT) tasks are used widely for continuous testing with Team Build and Release Management. As we thought about how test execution in the pipeline should evolve the guiding principles were to ensure that test execution in the pipeline is fast and reliable for all types of tests, be it unit tests (native MSTest as well as 3rd party) or functional tests – both UI and non-UI. Build and Release agents are already unified, so we were wondering if we could have Test Agent integrated as well  and get a ‘single automation agent to rule’ 🙂

So, what advantages does the unified agents work bring?

You can leverage the agent pool for all purposes – build, release and test.

  • Reusable agents means that you no longer need dedicated machines (as required by RFT task). Admins can setup a reusable pool and managing machines becomes easier.
  • You can use the unified agent for single machine as well multi-machine distributed execution.
    • VSTest task has been running on the agent, so single machine was always covered.
    • Distributed execution provided by RFT on remote machines using the Test Agent now comes to the unified agent, so you no longer need the ‘Deploy Test Agent’ like step.
    • ‘Deploy Test Agent’ was based on WinRM and WinRM had it’s own set of limitations, making the whole thing a steep learning curve. So yay, another complexity gone.
    • Since all execution is now local to the automation agent and phases downloads the artifacts automatically to the machines, you also don’t need ‘copy files’ tasks that were required to copy test assemblies and their dependencies when running tests remotely using RFT.

Let us get to the how-to now. The capabilities discussed here apply to the Visual Studio Test v2 (preview) task, so make sure you switch the task versions.

v2task

 

The first one is getting to know phases. When you create a release definition, you can add different types of phases – agent phase, deployment group phase or a server phase. You can add multiple phases of different types, in the order you need them to build your pipeline. Lets take a closer look at the agent phase settings.agentphasesettings‘Run on multiple agents in parallel’ provides 3 options:

  1. None – this means that a single agent from the specified queue will be allocated to this phase. This is the default and all tasks in the phase will run on that agent. When VSTest task runs, it runs exactly how it runs today – no change. You get a single agent test execution. So if I wanted to deploy an Azure web-app and run a small number of quick tests on it (for which a single agent suffices), along with some pre and post test setup/cleanup activities, I can model my environment as follows:
  2. Multi-agent – this mode means that multiple agents will get allocated to the phase. You can specify the number of agents to be allocated from the pool and the set of tasks in that phase will be run across all agents.

multiagent

VSTest task in this mode is special. It recognizes that it’s a multi-agent phase and runs tests in a distributed manner across the allocated agents. Since other tasks are run across all agents, any pre and post test steps I may want to do, also run equally on all the agents – so all the agents are prepped and cleaned up in a consistent manner. Test execution also does not require all agents to be available at the same time. If some agents are busy with another release or build, the phase can still start with the available number of agents that match the demand and test execution starts. As additional agents become available, they can pick up any remaining tests that have not been run yet. Here’s a screenshot of logs from my multi-agent test run, where some tests have failed.

multiagenttestrun

Artifacts are automatically downloaded when the phase starts, so my test assemblies and other files are already on the agent and I don’t need a copy files task.

So now if I want to publish an Azure web-app and run a large set of tests with fast test execution, I will model my Release environment as 2 phases – 1 being the deploy phase (runs on a single agent – I don’t want multiple agents to deploy the same app concurrently :-)) and a test phase that uses multi-agent mode to do test distribution. This also means that I can use different agent queues for the 2 phases, allowing me to manage agents for different purposes separately if I so choose.

distributedtests

3. Multi-config – this mode is driven by ‘multipliers’ pretty much the same way as a multi-config build is. Define the multipliers as variables and based on the values for these variables, the various configurations are run. In the case of Build, typically you would use BuildPlatform and BuildConfiguration as multipliers. Let us see how to apply this to testing. Same example as before – I want to deploy a web-app to Azure and run cross-browser tests on IE and Firefox. So I will model my environment as 2 phases – 1 deploy phase and the second is test phase. multiconfigThe test phase is setup as multi-config using ‘Browser’ variable, that has the values IE and Firefox. My phase will now run using these configs (2 in this case), 1 agent gets assigned a config and the appropriate config values are available for tasks to use.

multipliers

In my case, I want to use the Browser value to instantiate the right browser in my tests, so I will pass them as Test Run Parameters and access the value using TestContext in the test code. I will also use this value to title my test runs appropriately so that if a test fails in a particular config, I can easily figure that based on which run it came from.

multiconfigtasksettings

Here’s what the execution would look like:

multiconfiglogs

That’s all for today’s post. Give the VSTest v2 task a spin and let us know your feedback. Leave comments below or reach me at pbora AT microsoft DOT com.

FAQ

  1. How do I do this with Build?
    • The phases capability is currently available only in Release Management. It will become available in Build in a few sprints.
  2. Does VSTest v1 task behave the same way as the v2 task?
    • No, the v1 task does not do distribution. In the single agent (default, no parallelism) setting, the task will run as the way it has been running today. In the multi-config and multi-agent scenarios, it will get replicated on the agents, like all other tasks.
  3. What is needed to run UI tests?
    • To run UI tests, be sure to run agent in interactive mode. Agents set to run as service cannot run UI tests. Interactive agents in their current form will go down if the machine reboots for any reason. Enhancing the agent to survive reboots in interactive mode is being worked on. Also disable screensaver and unlock the machine so that UI actions in the test don’t get blocked. Automatic configuration of agents to do this is in the works.
  4. Can I run UI tests on the hosted agents?
    • No, running UI tests on the hosted agents is not possible currently.
  5. What does the ‘Test mix contain UI tests’ checkbox do?
    • Currently, it’s there only to serve as a reminder to run agents interactively if you are running UI tests. 🙂 If you are using an agent pool with a mix of interactive and ‘running as service’ agents, you may also want to add an ‘Interactive’ capability to your agents and use that demand in your test phase to pick the set of agents that can run UI tests.
  6. In multi-config mode, do I get distribution of tests as well?
    • No, multi-config mode assigns only one agent per config.
  7. How do I map the config in multi-config to my Test Configurations in TCM?
    • Currently this is not possible.
  8. How else can I use multi-config?
    • This mode can be used whenever you need multiple agents to do parallel jobs. For some other examples, refer the docs.
  9. Has the Run Functional Tests task changed also?
    • No, the Run Functional Tests (RFT) task has not changed. If you are using RFT task, you DO need the ‘Deploy Test Agent’ step. Please note that since tasks get replicated in the multi-agent and multi-config mode, using Run Functional Tests task in that mode will lead to undesirable effects.
  10. Do I need to install Visual Studio on all the machines to use the VSTest v2 task?
    • Currently, yes. We are looking at alternate means to run tests without needing Visual Studio on the agent, so that you can create a separate pool of agents for testing purposes.
  11. I am using my own test runner (not VSTest) in the pipeline. What happens to it?
    • In the multi-agent and multi-config mode, the task will get replicated on each of the agents. You can leverage the multi-config mode to partition your tests on different configs using the config variable (e.g., if you have a config variable called Platform that takes values of x86 and x64, you can run the two sets of tests on 2 agents in parallel by referring to your test assemblies using ‘**\$(Platform)\*test*.dll’
  12. How does the VSTest v2 task run on Deployment Groups?
    • Yes, the VSTest v2 task can be used to run on Deployment groups as well. If you have scenarios that necessitates running tests on machines in the deployment group where the app is deployed, you can use the VSTest v2 task. If multiple machines are selected (say, using tags) in the ‘Run on Deployment Group’ phase, the tests will get replicated on each of the machines.
11