Running Automated Test on agent machine using vNext Build

We have been getting lot of customer requests for asking the steps to execute automated test on agent machine using vNext build.

This blogs will help you configure and execute an automates test on agent machine using vNext build.

1. Create a new Empty vNext build definition and add the following steps. The remainder of the article explain the configuration for each step.

2. Visual Studio Build: Configure the solution or project which need to be build. Configure the project which has the Test Methods. This task will compile the configured solution or project in the build agent folder.

3. Copy and Publish artifacts: Configure the task to copy the build artifact to build drop location. Artifact Type provide you option to copy the build artifact to a network share or store in the server. Select the value as File Share to configure the UNC path for the build drop. Artifact Type as Server will store in the TFS database. This configuration will consume space in database to store the build artifact.

4. Visual Studio Test Agent Deployment: This build task will deploy the vNext test agent on the configured machine. Mention the machine group name under Machine section. The test agent will get deployed to all the machines configured in the machine group. The user account should have local administrator privileges on these machines to install and configure the test agent. Also need to enable the WinRM Firewall settings on the agent machines.

By default, this task will download the latest test agent from the internet. However, you can specify the test agent installation media path, in case you have the test agent installed available. This should be a UNC path or relative path on the build agent machine. And the build service account should have privilege to access this folder contents.

To configure the test machines under the Machine groups, navigate to Test => Machines tab from TFS web access. Add a new machine group and configure the FQDN or IP address of the test agents machine. Provide the user account which has local administrator privileges on the test agent machine.

Under the Agent Configuration, provide the user credentials for the agent service account. The password field in this section is a plain text. However, you can create a build variable and store the user password as encrypted. This variable can be used in the Agent Deploy task.

To enable the WinRM Firewall settings on the test agent machine, open a Command Prompt in elevated mode and perform the command winrm quickconfig.

5. Run Functional Test: This is final task which executes the test cases on the test agent machine. Mention the Machine configured name and Build drop location path from where the test agent can pick the test binaries.

By Selecting the Test Assembly option in Test Selection, you can execute all the test cases in a test dll. However, the Test Filter Criteria will filter the criteria based on the parameter provided (Name=TestMethod1 or TestCategory=TestCategory1 or Owner=OwnerName).

In case you want to execute all the test case configured under a Test Plan or Test Suite, change the Test Selection option values to Test Plan. This would list the Test Plan and Test Suite configured for the specific Team Project and you can select the values.

Hope this helps!

Content: Sinjith Haridasan Reeja
Review: Chandra Sekhar Viswanadha

• Anonymous
  January 27, 2017
  The comment has been removed
  • Anonymous
    March 03, 2017
    I was able to resolve my first previous issue with the help of a MSDN thread. In both the build and file copy tasks from my test project's build, I selected the Clean option (advanced option on copy task). This picked up the adjustments I made to my test's playback settings and I was able to get the correct settings to make my tests run correctly. Seemed pretty obvious in retrospect, but wasn't obvious during my troubleshooting because I wasn't adding/modifying test steps each time, only playback settings. I hope this follow up comment helps someone with a similar troubleshooting "problem".