Executing Automated tests in Build vNext using Test Plan, Test Suites
March 31, 2016
With Sprint 97 deployment on VSTS, users who have invested in planned testing using Test Plan, Test Suites can trigger automated runs in Build vNext. This functionality has been added to the Run Functional Tests task (formerly called Run Tests using Test Agent) as shown below. This functionality is also shipping in Team Foundation Server 2015 Update 2. If you are new to testing in Build vNext, please see https://msdn.microsoft.com/en-us/library/vs/alm/test/continuous-testing/continuous-testing.
Existing ‘Lab Build and Test’ workflow users will find the settings in the task familiar to the Test section of the workflow. Once you choose Test Selection to be Test Plan, choose the Test Plan from which you want to run your tests. You can then select one or more test suites from a Tree view control followed by a Test Configuration.
In the Lab Build and Test workflow, you would choose a Test settings file that was stored in MTM. There were issues with this approach, including the fact that you could not use the same settings file to run tests in your IDE and your BDT workflow. With Build vNext, we are enabling you to specify runsettings or testsettings file from the source code repo or from disk.
That’s it. You can queue your Build Definition and your automated tests from the suites selected will be run. Test results can we viewed in the same way as results from Assembly based execution – in the Tests tab of the Build Summary page.
To see the results per Test Suite, just select “Test Suite” in the “Group by” drop down
Clicking on a Test Suite name will navigate you to the Test Hub where you can see the Test Plan, Test Suite and Test Cases and the history of test case results





How do we add configurations for iOS and Android to execute our test plans against our mobile device configurations in Test Cloud?
How does this task handle very large test plans – the ‘old’ functionality in the xaml builds would distribute tests evenly across all of the machines in an environment – is there any ‘parallelization’ here? Our Regression test plan has about 5000 tests in it and they can run from a few minutes to nearly an hour each, we simply can’t run these sequentially on a single test machine.
Hi Brian, Is this issue resolved for you. How you are handling this? Thanks in Advance.. Eswar
We haven’t resolved the issue yet. According to user voice they have additional features around running tests in parallel that are available now for Team service but won’t be made available for on-premise until tfs vnext. We are sticking with the old tried and true xaml builds for the cases where we need to be able to test in parallel. Its just frustrating that this means we’ve got to sustain two different sets of tools and builds for each of our products.
https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/12411765-allow-the-visual-studio-test-using-test-agent-ta
We are in need of this functionality to help migrate existing XAML build to the new build system. We checked Update 2 and VSTS but still do not see the functionality in the task described in this blog. When should we expect this enhanced task features to be available.
Never mind my earlier comment. I found that I was not using the drop down to change from test assembly to test plan. DOH!
One final note, it appears this feature in in the latest release of VSTS but did not make it into TFS Update2 for on-premise. Any update on when it will be available for TFS on-premise?
Atin,
After following these directions exactly, my tests produce the following error:
Could not load file or assembly ‘Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified’
Why does the deployed test agent look for 2013 .dlls? Why is it looking in the GAC? How can I workaround this error in order to execute my BDT workflow?
Ryan, please take a look at https://blogs.msdn.microsoft.com/sathar/2016/08/01/running-coded-ui-tests-built-using-visual-studio-2013-with-team-foundation-server-2015/. This should help you get unblocked.
Thanks!
I’m having trouble getting the data driven iteration on a unit test to work in this context when I connect back to the test case to get the test case parameters to iterate on. The data source decorator looks like this:
[DataSource(“Microsoft.VisualStudio.TestTools.DataSource.TestCase”, “***;***”, “308570”, DataAccessMethod.Sequential), TestMethod] where the ***’s represents the url of the collection and the project. When I run the test it indicates the following error:
The unit test adapter failed to connect to the data source or to read the data. For more information on troubleshooting this error, see “Troubleshooting Data-Driven Unit Tests” (http://go.microsoft.com/fwlink/?LinkId=62412) in the MSDN Library. Error details: Unable to find the requested .Net Framework Data Provider. It may not be installed.
This used to work fine with xaml lab deploys and now that you’ve added support for associating a unit test method with test plans and suites – I thought you’d also be enabling this data driven capability again as well. Is this currently supported – and if so – what am I doing wrong.
Thanks!
Unfortunately, Data driven tests are currently only supported if the data is coming from xml, csv or sql end points. Run Functional Tests task cannot be used for data coming from VSTS. Adding this support is on our backlog. Thanks!
Atin, I’m interested in following the “TestCase” DataSource work in TFS 2015 from VSTS. Is your backlog public? If so, can you provide a link to the workitem? Do you have any estimate on when that work will get started and completed? Thanks.
Is there a way to use the Run Functional Tests to run an orderedtest. I’ve tried targeting the .orderedtest file but with no luck. It seems the orderedtest file in my solution has hardcoded paths to /bin/debug/mytestassembly.dll and i cannot figure out how to get that to change. The build output from my build task does not match that structure, so the test runner never finds any tests associated.
I’m certain that there is some kind of transform that would need to happen on the orderedtest file so that the storage path is targeting the correct location, but im at a loss as to what that would be.
Any help is appreciated,
Mike
I am having similar issues with not being able to run a generictest. Is this not implimented yet, or am I missing something.
Please take a look at https://blogs.msdn.microsoft.com/allendm/2016/08/09/executing-ordered-tests-on-remote-machines-using-run-functional-tests-task/ to execute ordered tests. Thanks!
I have TFS2015 update 3 installed. When I try to use the Run Functional Tests step, I am to select Test Plan as the Test Selection and I see my test plans in the drop down for Test Plan. However, the Test Suite drop down is completed blank. I have Test Suites and used them extensively under TFS2013 for codedUI testing. Any insight on why the Test Suite drop down is empty? This issue is preventing us from changing our codedUI Automation over to TFS2015 using the new web based build definitions.
I experienced completely the same issue. Did you found solution for this issue?
Is this resolved? If not, can you try updating your agent (Update Agent check box in the Deploy task). If it still does not work, please mail me at atinb at microsoft dot com.
hi,
I am also facing this issue, we recently updated from TFS 2010 to latest TFS 2015 and when I select a Test Plan, the Suites below are not shown in the drop down list…
In my caase, I think this has nothing to do with the update agent check box…
I also had this issue, i had to recreate a new test suite and re add the tests there. It seems that extremely old test plans have an issue loading them into that dropdown
Hello. we think that we have configured everything appropriately. But we get the below error when executing our build. Has anyone else ran into this trying to utilize Test plan/ Test Suite capability in new vnext build process as outline above? We are now running TFS 2015 Update 3 as well.
Also is a runsettings/testsettings file required?
ystem.InvalidOperationException: No automated test cases were found matching the specified test plan, test suites, test configuration. Ensure that right values were specified and try again.
at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.RunTests.QueueAndWaitForRunToComplete()
at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.RunTests.Run()
at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.TaskRunner.Run(Dictionary`2 arguments, VssConnection connection, TaskType taskType)
at Microsoft.TeamFoundation.DistributedTask.Task.DistributedTestAutomation.RundistributedTestsCmdLet.ProcessRecord()
Kevin, I am guessing your Test Plan has automated tests. Please see if the Test Assembly field in the Run Functional Test task has anything filled out, and if so, please clear the field and try. If nothing works, please mail me at atinb at microsoft dot com.
I would like to know how I can bind my existing unit test or smoke test code to the test plan and then run it in build
Hi,
I have several unit test and smoke test code which are located in my test project, when I want to link the manual test case to my test project method, it only shows one performance test and not the one I want to link, do you have some suggestions ?