Executing Ordered Tests on remote machines using Run Functional Tests task

  • Article

The Run Functional Test task helps you execute your tests on one or more remote machines after you have setup the test agent on them, using the Deploy Test Agent task. This functionality can help you run your functional/UI tests on machines outside of your build agent.

 

During our first release of this new workflow in Team Foundation Server 2015 and Visual Studio Team Services we consciously left out some functionality that wasn’t complete for the release. One of these was the support for ordered tests. While we are working hard to get this scenario lit up for the next release we have heard enough requests to provide a workaround.

 

So what is the workaround?

You can copy the ordered test adapter for vstest to the right directory on your test agent and the scenario will light up.

If you are using the BDT build template then add a remote file copy task before your "Run Functional Tests" task.

copyorderedtestadapterdll

In the copy files task set the source as <VSIntallDir>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.dll and destination as <TestAgentInstallDir>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.dll
The <TestAgentInstallDir> would be in the %systemdrive%\Program Files (x86)\Microsoft Visual Studio 14.0 directory

 

runfunctionaltestorderedtest

In the Run Function Tests task you can now specify **\*orderedtest.dll as the test filter.

One of the issues that we noticed with this approach is that ordered test failures are not registered as failures in the test run. To overcome this

  1. Enable oauth access to the agent in the build definition -> options
  2. Place the ps1 script in a location accessible to the build agent ( https://github.com/Microsoft/customer-scripts/blob/master/test/build/orderedtest/throwif-trxfromtestrun-ofbuild-hasfailedoutcome.ps1 )
  3. Add a powershell task after the run task and provide the path to the ps1 in this task
  4. The ps1 will error out if the ordered test has failed

The script checks the latest trx file associated with the test run associated with the current build and throws an error if the trx specifies the test run has a failed outcome. Feel free to contribute back if you find any issues in the script.

 

Some points to note

  1. Given that an ordered test is single unit of execution it wont be distributed across machines, if you specify multiple machines in the machines input.
  2. The reporting shows one entry per ordered test. You can drill down further by downloading the .trx file which is attached to the test run
  3. Ordered tests within a test suite are not supported