Remote Test Execution using Team Foundation Server 2015 RC2 and beyond

  • Article

 

The Agents for Visual Studio suite containing the test controller and agents allowed you to execute your tests on a set of remote machines by providing the right configuration in your .testsettings file. Based on the feedback we have received we are providing a new and simpler way for you to do your remote automated testing starting Team Foundation Server (TFS) 2015 RC2.

In this blog post we will quickly setup a test rig for remote test execution using TFS 2015 RC2

 

Pre-requisites

  1. TFS 2015 RC2 installation with a Build vNext agent configured on a machine with internet access.
  2. A (virtual) machine to be used for testing with following requirements

 

Step-By-Step Guide

Head to your TFS homepage and navigate to the team project you want to work with. The below slides will guide you through this entire scenario.

NOTE: The "Run Tests using Test Agent" task is renamed to "Run Functional Tests"

 

 

So what has changed

There have been a lot of changes to get the stability and resiliency that you demanded in test execution. To summarize the top few improvements

  1. The test controller has been removed as the intermediate hop and the agents now directly communicate with the TFS server using HTTP.
  2. You can now run tests using 3rd party test frameworks like nunit/xunit/cpp using the relevant unit test adapter, similar to Visual Studio.
  3. Automated installation and configuration of your test agent on your test machines as a part of the Build workflow using our tasks in Build vNext.
  4. Improved reporting for automated runs in Test Management.

 

What are our limitations

  1. We do distribution on a per test container basis ie if you have 2 machines and 2 test dlls we will execute all tests from testdll1 on machine1 and the rest from testdll2 on machine2.
  2. We don’t have support for workgroup/isolated/azure machines as yet. As long as you have http connectivity from the agent to TFS/Visual Studio Team Services and your build machine you will be good to go.
  3. We do not support Azure VMs as machines in the machine group used for testing
  4. Yes the scenario can be used in TFS/Visual Studio Team Services as well. However we do not support hosted build pools when using on-prem machines. For this case you would have to create a new build pool and register a new build agent with it.

 More up-to-date details @ https://aka.ms/remotevstest

FAQ

Why do we have a test agent and a build agent? Why can’t there just be a common agent?

The build agent does provide you with unit testing facilities but only after you install Visual Studio on your build machine. The test agent has no such prerequisite. This is the sole reason why we have two agents instead of one. We are definitely looking at ways in which we no longer need to make such a distinction.

Will I be able to run my java tests on my remote Linux/OSX machines using the test agent?

Today the cross platform functionality is available in Build vNext. We are looking for the right data points before we build the remote testing scenario for other platforms.

 

We hope that this post will allow you to get started with the new remote test execution workflow on your on-prem TFS installation. Try it out and let us know what you think devops_tools@microsoft.com.

 

Additional links

1. A good Hands On Lab for CI/CDP - https://blogs.msdn.com/b/charles_sterling/archive/2015/06/22/ci-and-cd-hands-on-lab.aspx

2. A demo video on setting up CI/CDP by one of the MVPs - https://blogs.msdn.com/b/visualstudioalm/archive/2015/07/17/video-configuring-continuous-integration-and-continuous-testing-with-visual-studio-2015.aspx