Test Controller, Test Agent versions and their usage in different scenarios
November 23, 2016
There have been several questions on the different scenarios related to running automated tests in various workflows and which versions can / cannot be used.
Let’s use the following 2 names for categorizing how agents get used:
- Test Controller/Test Agent (TC/TA) scenarios – this refers to on-premises load testing, running automated tests in the XAML build workflows, triggering an on-demand execution of automated tests using MTM and creating and managing lab environments using Visual Studio Lab Management.
- Team Build / Release Management scenarios – this refers to running functional tests in the Team Build / Release Management workflows, using the ‘Visual Studio Test Agent Deployment’ and ‘Run Functional Tests’ tasks from the task catalog.
Agents for Visual Studio 2013 Update 5
- Can be used for TC/TA scenarios
- Cannot be used for Team Build/Release Management scenarios
- Can run tests that were built using Visual Studio 2013 and Visual Studio 2015.
Agents for Visual Studio 2015 and Agents for Visual Studio 2015 Update 3
- Cannot be used for TC/TA scenarios
- Can be used for Team Build / Release Management scenarios
- Can run tests that were built using Visual Studio 2015 and Visual Studio 2013.
- Can be used for TC/TA scenarios
- Can be used for Team Build / Release Management scenarios.
- Can run tests that were built using Visual Studio 2017.
Offline installers can be created using the steps similar to the Visual Studio offline installer, described here.
So, for example, to create an offline installer for Test Agent:
- First download the setup bootstrapper (vs_testagent.exe)
- To create the offline installation, run “vs_testagent.exe –layout c:\VSTestAgent2017Offline” from the directory where vs_testagent.exe got downloaded to.
For options to customize the offline installer, refer to the doc.
TFS 2017
TFS 2017 RTM – TFS 2017 RTM shipped with ‘Visual Studio Test Agent Deployment’ (v1) and ‘Run Functional Tests’ tasks that can be used in the Team Build / Release Management scenarios. These tasks use the Agents for Visual Studio 2015 Update 3 package. As such, the tasks in their current form can be used for running tests built using Visual Studio 2015 and Visual Studio 2013. They cannot run tests built using Visual Studio 2017.
TFS 2017 Update 1 – TFS 2017 Update 1 shipped with ‘Visual Studio Test Agent Deployment’ (v2) and ‘Run Functional Tests’ tasks that can be used in the Team Build / Release Management scenarios. The test agent deployment task has the ability to specify which Agent version (2015 or 2017) should be deployed. Based on the selection, the appropriate agent will get deployed and can run tests built using Visual Studio versions as described in the previous sections.
Hi,
Just wondering if/when will the TestController/TestAgent support MSTest V2 and potentially other frameworks like NUnit/XUnit etc?
I assume at some point they will not only run MSTest and will eventually support vstest.console.exe to allow this?
Cheers,
Sam
@Sam Thwaites:
Third party frameworks such as NUnit, xUnit, etc are already supported with the Test Agent in the task based build and release management system. This has been supported since TFS 2015.
So the ‘Deploy Test Agent’ + ‘Run Functional Tests’ tasks (in addition to the VSTest task) do support running these tests.
There is no plan to support this in the XAML build based workflows – so Lab / MTM will not support xUnit / nUnit and other test frameworks.
Hi.
How do I create an offline installer for ‘Visual Studio Test Agent Deployment’ (v2), like the old exe ‘vstf_testagent.exe’ for TFS 2015?
/Magnus
@MagnusTim
An offline installer can be created similar to creating an offline installer for Visual Studio, as described here
https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio
The setup file for test agent is vs_testagent.exe. Once you download it, running
“vs_testagent.exe –layout C:TA2017Setup” should create the offline installer in the C:TA2017Setup folder.
I will add this to the blog and get the Test Agent specific example in the docs as well.
In TFS2017 U1, Deploy Test Agent Step (V2), using Test Agent Version “Visual Studio 2017”, it is possible to define the location of the Test agent, as it was with the previous version of this Step.
Previously, if you provided a value for this field, pointing to an offline installer for the agent (vstf_testagent.exe), the build step would push this installer to each agent at the start of the Test Run and install it silently (assuming “update test agent” is ticked or the agent did not previously exist). With the new Agent (vs_testagent.exe), what happens in the following situations:
1) If a location for vs_testagent.exe is provided, when it is pushed to each Test Agent before a Test Run (for running Functional Tests), one assumes that the installation happens silently, and each agent must go online and download the required install files as this is a thin web installer?
2) If an offline installation folder is created using “vs_testagent.exe –layout c:VSTestAgent2017Offline”, and this is placed in the folder that is provided in the “Test Agent Location” field in the build step, will the build step push this all to the Test Agents, meaning that downloading install files per agent is unnecessary? For example, if i navigate to an individual Test Agent, will i see the VSTestAgent2017Offline folder along with the installer?
Many thanks!
Steve
@Cytron201 – The option to specify a location was designed for the non-internet connected scenarios. So, no, you cannot specify the bootstrapper in that one.
If you create an offline installer and place it on a share, yes, it will be used by all machines specified in the Deploy Test Agent task.
I’m following TC/TA scenario using “Agents for 2017”. The “Configure Test Agent” dialog provides an option to “Register with Test Controller”. How do I install test controller. My current test rig is setup with 2013 test controller and 2015 test agents. I’m accustomed to running load tests and parallel UI automation tests from visual studio against test rigs setup with test controllers and agents. Can I have the same setup in my on-premise test rig using TC/TA 2017.
There is a spelling error in command describing how to create offline agent installer. Parameter needs double dash prefix:
“vs_testagent.exe –layout c:VSTestAgent2017Offline”
But this blog site interpreters double dash and renders it as a “long dash” instead… Very confusing.