Remote Test Execution using Team Foundation Server 2015 RC2 and beyond


 

The Agents for Visual Studio suite containing the test controller and agents allowed you to execute your tests on a set of remote machines by providing the right configuration in your .testsettings file. Based on the feedback we have received we are providing a new and simpler way for you to do your remote automated testing starting Team Foundation Server (TFS) 2015 RC2.

In this blog post we will quickly setup a test rig for remote test execution using TFS 2015 RC2

 

Pre-requisites

  1. TFS 2015 RC2 installation with a Build vNext agent configured on a machine with internet access.
  2. A (virtual) machine to be used for testing with following requirements

 

Step-By-Step Guide

Head to your TFS homepage and navigate to the team project you want to work with. The below slides will guide you through this entire scenario.

NOTE: The “Run Tests using Test Agent” task is renamed to “Run Functional Tests”

 

 

So what has changed

There have been a lot of changes to get the stability and resiliency that you demanded in test execution. To summarize the top few improvements

  1. The test controller has been removed as the intermediate hop and the agents now directly communicate with the TFS server using HTTP.
  2. You can now run tests using 3rd party test frameworks like nunit/xunit/cpp using the relevant unit test adapter, similar to Visual Studio.
  3. Automated installation and configuration of your test agent on your test machines as a part of the Build workflow using our tasks in Build vNext.
  4. Improved reporting for automated runs in Test Management.

 

What are our limitations

  1. We do distribution on a per test container basis ie if you have 2 machines and 2 test dlls we will execute all tests from testdll1 on machine1 and the rest from testdll2 on machine2.
  2. We don’t have support for workgroup/isolated/azure machines as yet. As long as you have http connectivity from the agent to TFS/Visual Studio Team Services and your build machine you will be good to go.
  3. We do not support Azure VMs as machines in the machine group used for testing
  4. Yes the scenario can be used in TFS/Visual Studio Team Services as well. However we do not support hosted build pools when using on-prem machines. For this case you would have to create a new build pool and register a new build agent with it.

 More up-to-date details @ https://aka.ms/remotevstest

FAQ

Why do we have a test agent and a build agent? Why can’t there just be a common agent?

The build agent does provide you with unit testing facilities but only after you install Visual Studio on your build machine. The test agent has no such prerequisite. This is the sole reason why we have two agents instead of one. We are definitely looking at ways in which we no longer need to make such a distinction.

Will I be able to run my java tests on my remote Linux/OSX machines using the test agent?

Today the cross platform functionality is available in Build vNext. We are looking for the right data points before we build the remote testing scenario for other platforms.

 

We hope that this post will allow you to get started with the new remote test execution workflow on your on-prem TFS installation. Try it out and let us know what you think devops_tools@microsoft.com.

 

Comments (3)

  1. Max Rubinstein says:

    Hi Allen,

    We have upgraded our CodedUI projects to refer to v14.0 versions of Microsoft test assemblies.

    Our plan was to upgrade our test VMs from VS 2013 agents to VS 2015 agents, however it now looks like we would have to wait till TFS 2015 release, and then redesign our builds to properly integrate CodedUI test deployment and execution.

    So, with upgraded assembly references,  we are experiencing symptoms mentioned in the following link:

    blogs.msdn.com/…/test-agents-support-for-visual-studio-2015.aspx

    We have tried the app.config binding redirect workaround mentioned in the link, and it did not seem to work for us.

    My question is: what is the recommended way to get CodedUI tests using v14.0 assemblies to work on VS 2013 agents? Should we still follow the workaround from the link above, even though it was written for VS 2015 CTP? Or is there a different option available?

    Thanks

    -Max

  2. Why don't you reply to your comments?

  3. Hi Max,

    Yes. For VS2013 agents you would have to add in the assembly redirections as mentioned in the blog post. In short the idea is that TFS2015 will continue to support VS2013U4+ agents to enable customers to bring their assets forward.

    If the redirections do not work for you then do reach out to me with more details on my email address allendm-msft at hotmail dot com and I will loop in the folks for you.

    If you are not particular about using the 2013 agents then you can try out the new experience I have outlined in this blog post, which is available in TFS 2015 RC2/RTM.

    HTH

Skip to main content