How to run automated tests on different machines in parallel?

For this post, I am assuming that the tests are stored in test case management store of Tfs and you are queuing the test run from the tcm clients (Either MTM, TCM.exe or lab workflow). In case your tests are only in VS and you are running them via VS or MSTest, then you can control the distribution using the bucketing knobs (mainly bucket-size property, see appendix below for more details) available in VS.

To distribute the tests for a tcm run, you have following options.


Option 1 (Recommended one)

- Create multiple environments with each environment consisting of single machine (read single agent).

- Divide all your test cases onto different sets and queue one set on one environment and other set on other environment and so on.

- With this each set of test will run in parallel and on different machine. 


Option 2

This option will work if you have less than 100 (default bucket size) tests in your test run.

- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role.  (In your test settings, you should specify this role as the execution role.)


- Change the MinimumTestsPerAgent property (see appendix below for more details) to 1 in the test controller configuration file (QTControllerConfig.exe.config). Make sure that you restart the test controller to bring this change in effect.

- Now queue the test run consisting of your <100 tests and you will notice that all the agents are busy in running the tests.


Option 3

This option will work if you have more than 100 tests in your test run.

- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role.  In your test settings, you should specify this role as the execution role.)

- Let us assume that you have 600 tests and have 3 machines to run them.

- Now in the test settings, change the bucketSize from default 100 to 200 (number of tests/number of machines i.e. 600/3). Please note that there is no gesture in MTM with which you can do this and will have to use this utilitywith following syntax.

UpdateTestSettings /collection:http:abc:8080/tfs/DefaultCollection /teamProject:myProject /settingsname:mysettings /bucketsize:200

- Now queue the test run and you will notice that all the agents are busy in running the tests.


Appendix: -

Bucket Size

Bucket size denotes the maximum size of each bucket that is assigned to an agent. Once the agents finishes the execution of a bucket of tests, it is given another one if more exist. The default value of the bucket size is 100. In VS it can be controlled by the Execution/Buckets/size attribute in the test settings as shown below.

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="d023da83-9635-4625-bcf9-fc8b0e92a9e1" xmlns="">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
    <Buckets size="100"/>
    <TestTypeSpecific />
    <AgentRule name="Execution Agents">



This property is used for second level of bucketing and comes into picture when the number of tests that need to be distributed creates a partial bucket. For example if the number of tests are 20 (which is less than default bucket size 100) or 120 (in which you have 1 full bucket and other partial bucket with 20 tests), then for all partial buckets, this second level of bucketing is used.

Default value of this property is 10 and is controlled by “MinimumTestsPerAgent” app setting in controller’s configuration file (QTController.exe.config). This property is not present in the configuration file and you will have to add it.

  <add key="MinimumTestsPerAgent" value="1"/> 
  <add key="LogSizeLimitInMegs" value="20"/>

Comments (71)
  1. Brandon says:

    Will this work for Coded UI Tests? For example, I have an environment with 10 machines. I have one Coded UI Test that I want to run on each machine in parallel.

  2. Yes, this will work for coded UI tests as well.

    So in your example, if you have 10 machines in an environment and 10 coded UI tests, then you can run 1 test on each machine in parallel.

  3. Brandon says:


    That's not my example. I have 1 (single) CodedUI test that I want to run on each machine in parallel. I don't have 10 seperate tests.I want to run the same single test on each machine in parallel.

  4. Brandon, Thanks for the clarification.

    This post only talks about distributing the tests across machines (like 5 tests to 2 machines) which can run in parallel and does not talk about running same test on n different machines.

    However for your scenario, you will have to either create 10 seperate test runs containing the single test each targetting different machine or in a single test run, add 10 different tfs test cases corresponding to single coded UI test method.

  5. can you please tell how to change the role to execution role

    Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role.  (In your test settings, you should specify this role as the execution role.)

    I am seeing only server, web client , web server , database server and desktop client options only

  6. Suman,

    You can set the execution role in your test settings (Open the test settings in MTM -> Roles -> Select the role to use to run your automated tests).


    Aseem Bansal

  7. Thanks Aseem it has woked.

    Now the test cases are getting distributed to the agent randomly .How can we control that let' say execute this test suite1 on this agent and execute test suite2 on agent 2  parallely and execute Test suite 3 once suite and suite 2 gets completed only.

  8. Suman,

    To achieve that, you will have to create environment with single agent and then queue32 separate runs. Run suite1 on environment1 (which contains agent1) and run suite2 on environment2 (which contains agent2). Queue suite3 on environment1 or 2 which will get picked up once that agent is done with its current suite.


    Aseem Bansal

  9. I have set up a physical environment with 4 agents and one controller.I have 9 test cases for execution.

    When i start the Run from the Labmanger.

    each agent is executing one test cases. So at a time 4 test cases are getting executed and rest of the test cases are queued .But once  4 test cases are getting executed completly.The second set of test cases are sittilng in queued status for long time (5 to 7 mins) and the agent is showing as still running in progress.After waiting for 5 to 7 mins the agent is executing the second set of test cases.

    Can you please help here how can i make the agent continously execute the test cases so that  the run will complete fast.

    Below are myappseting in config file of controller


        <add key="LogSizeLimitInMegs" value="20"/>

        <add key="AgentConnectionTimeoutInSeconds" value="120"/>

        <add key="AgentSyncTimeoutInSeconds" value="300"/>

    <add key="MinimumTestsPerAgent" value="1"/>  

       <add key="MaxConcurrentTcmRuns" value="4"/>

        <add key="ControllerServicePort" value="6901"/>

        <add key="ControllerUsersGroup" value="TeamTestControllerUsers"/>

        <add key="ControllerAdminsGroup" value="TeamTestControllerAdmins"/>

        <add key="CreateTraceListener" value="yes"/>


  10. Hi Aseem,

    Thanks for nice post.

    I did configuration as per Option 1. Now, when I try to run any test which is suppose to run in other machine, it is throwing exception:

    Unit Test Adapter threw exception:

    Could not load file or assembly 'eXimius.AutomatedTesting.WealthPlan, Version=, Culture=neutral, PublicKeyToken=6676d32b6740bbf3' or one of its dependencies. The system cannot find the file specified..

    Other machine i have installed only Test Agent and register with controller. So, Is it require to do some more settings?

    Waiting for  your reply…it is very urgent for me


    • Umesh
  11. Umesh,

    Is the test working fine locally using mstest.exe?

    Also can you please check whether the mentioned assembly is deployed on the test agent machine or not?Typically the directory in which the tests are deployed on the agent machine is something like %LocalAppData of the user under which test agent is running% VSEQTQTAgent<Guid>Deployment.


    Aseem Bansal

  12. Aseem,

    Sorry! for late reply. Yes, Locally it is working fine. Actually, locally all dlls are registerd in GAC but in other machine we don't have VS2010 itself. only Test Agent.

    I verified that location which had suggested. Dlls were not deployed there so, manually I have added all required dlls to same location "C:Usersumesh.singhAppDataLocalVSEQTQTAgentf447527b-5b42-4bd8-aa6e-e351a380a04fTFBEMEV0268". Later I will add in Deployment setting of Lab center.

    I still not able to see it running inside other machine. Showing same error.  


    • Umesh
  13. Thanks Aseem. After adding the deployment attribute in TestSetting of Lab Center, It started working fine.

  14. Hi Aseem,

    One strange issue I am facing. When I run one test(belong to other machine) from one MTM, It is executing 1st step of test case "opening application browser with URL". After this it is not doing anything. I guess, it might not recognizing UI elements. I am not sure.

    It is not logging any exception. So, unable to recognize the issue.


    • Umesh
  15. Good to know that the test worked fine for you.

    Regarding your new question, Can you please share whether you are running a "manual test" or something else?

    Also it will be great if you can post your question on the test manager form as someone more knowledgeable can reply to your question much sooner. Here is the link to the forum.…/threads


    Aseem bansal

  16. I running automation test. I have one TestProject which are having so many CodedUI test cases. These test cases associated with Test Plans in MTM.

    I am going to place same question to test manager forum.

    Thanks for quick response.

    • Umesh
  17. maria says:

    Hi Aseem,

    have you ever encountered the problem to get no code coverage results once the tests are executed in parallel?



  18. Hi Maria,

    As mentioned in the below msdn link, code coverage is not supported in remote runs as well as MTM based test runs. This means that without parallelism also, it must not be working for you.…/dd504821.aspx

    >You can only use code coverage when you run an automated test, and only from the computer that is running the test. Remote collection is not supported. You cannot use code coverage when you run tests from a test plan using Microsoft Test Manager.


    Aseem Bansal

  19. datasystest says:

    Great info.. Thanks for the piece of code as well..The process seems to be very simple but only after executing it we will come to know about the difficulties..thanks a lot for the info related to testing on machines.. If people interested to know more details about various testing process on machines kindly visit

  20. Srini says:

    I have set up a environment with 4 agents and one controller.I have 12 test cases for execution.

    When i start the Run from the Labmanger.

    each agent is limited to execute only 4 test each. so for example the 4 tests taken by agent1 is large test case, it takes 2 hr to complete but the 4 tests taken by agent4 is small it complete by 30 mintues and it goes to idle.

    how can we utilize all the free agent (in this case agetnt4)?

    i mean since agent 4 is done with test, can it borrow test from the bucket.?

    is there any settings in configuration file we can do that?

  21. I have set up a environment with 4 agents and one controller.I have 12 test cases for execution.

    When i start the Run from the Labmanger.

    each agent is limited to execute only 4 test each. so for example the 4 tests taken by agent1 is large test case, it takes 2 hr to complete but the 4 tests taken by agent4 is small it complete by 30 mintues and it goes to idle.

    how can we utilize all the free agent (in this case agetnt4)?

    i mean since agent 4 is done with test, can it borrow test from the bucket.?

    is there any settings in configuration file we can do that?

  22. jophy says:

    Is possible to run same test on all test agents . Because we like to run the same test in different OS ( XP ,Win 7 and Win 8)  for compatibility check of code ..

    is it possible .

  23. Jpohy, You should queue separate test runs, one for each OS.

  24. Kal says:

    Aseem, the image in your blog above for Option 2 is not visible. Can you correct the image or repost, so we can see what you mean by execution role?

    Thanks, Kal

  25. Kal, Does it work now?

  26. Boon says:

    Hi Aseem,

    I have the same requirement as what Brandon posted in the 1st post.

    "I have 1 (single) CodedUI test that I want to run on each machine in parallel. I don't have 10 seperate tests.I want to run the same single test on each machine in parallel."

    Any idea how could that be achieved in parallel without having to create 10 test cases targeting a single test method? I have like 100+ test methods that would like to be re-run in 4 platforms and creating test cases solely for that purpose doesn't seems feasible. 🙁

  27. Boon,

    You can create n test runs containing the single Coded UI test each targetting different machine. Right?


    Aseem Bansal

  28. Kanchan says:

    Hi Aseem,

    Thanks for such a nice post and souce code.

    As said above I have added bucket size=1.


       <Buckets size="1"/>


    After thsi when i restarted my machie, I observed that VSTS Controller service started throwing error and could not start. Any help on this is really appriciable.



  29. Hi Kanchan,

    In which file have you added this bucket size?


    Aseem Bnasal

  30. Kanchan says:

    Hi Aseem,

    I have VSTS2012 installed on my machine and I am using the latest utility uploaded at…/update-testsettings-utility.aspx .

    I have added below line of code in QTController.exe.config file.

    <TestSettings name="CISystemDPTestSetting" id="d023da83-9635-4625-bcf9-fc8b0e92a9e1" xmlns="…/2010">

     &lt;Description&gt;These are default test settings for a local test run.&lt;/Description&gt;    
        &lt;Timeouts testTimeout=&quot;0&quot; /&gt;
        &lt;Buckets size=&quot;2&quot;/&gt;
        &lt;TestTypeSpecific /&gt;
        &lt;AgentRule name=&quot;Execution Agents&quot;&gt;


    While restarting qtcontoller services it throws error message

    "Windows could not start the Visual Studio Test Controller service on Local Computer. Error 1053: The service did not respond to the start or control request in a timely fashion."



  31. Sandeep Kulkarni says:

    Can we run a same test case on different VMs simultaneously.

  32. Sandeep, If you have to run the same test case simultaneously on different VMs, then you will have to different test runs, each targeting a VM.


    Aseem Bansal

  33. How to run parallel says:

    I have set up a environment with 4 Work Stations (Vs-2010 Ultimate) Client for load Execution. Each workstation has granted for 3 users. I have 8-agents and two controllers on remote machine. My Load scenario configured has remote.

    User A and User B has access the Workstation (1)  login with their credentials.

    A and B has trying to execute two different load scenarios on Workstation (1).

    User A Scenario has shows in process, user b has showed in progress. It has not executed until User A execution complete. How could i run this parallel.

  34. Sandeep Kulkarni says:

    Hi Aseem,

    We are using build definition to execute a set of cases automatically on every new build.currently we have 1 test env with 1 test machine.

    To reduce the test time, I want to distribute the cases across 2 to 3 environements ( 1 machine per env).I cannot use the other options other than Option 1 as I will have to include the install cases on all the environments.

    When I edit the build definition & go to Process option,it asks for eveironment & associated test plan & test settings for that plan.

    Let me know how to add multiple environements over here & associate different set of cases to each environement.

    I am assuming that the first option that you mentioned in the blog is on the same lines.

  35. Diganta Roy says:

    With the option '1' Aseem mentioned, you still need to trigger of 'n' number of MTM test runs, or 'n' number of builds(of different definition) each using a different environment and running different tests. There is no way to use more than one environment in a single BDT or MTM test run.

    Having said that, you can still accomplish parallelism by using option '3', as you can create a single environment and prepare multiple machines where tests can run (you anyway have to do this for multiple environments as well).

  36. Sandeep Kulkarni says:

    Thanks Diganta for a quick reply

    Let me explain my test scenarios in bit more detail.We have  500 cases approximately to be executed.These cases are grouped into different folders under the Test Suite.Among these cases are the Install & Configuration cases.The remaining cases are dependant on the above set of cases.

    So if we need to distribute the entire suite in 2 to 3 machines,the Install & Config cases will have to be run on all the three machines.I dont have a clue on how to achieve this part.

    Once this is done, I can then set the bucket size to 100 or 200 & run the cases simultaneously on the 3 setups.

  37. Sandeep Kulkarni says:

    This is a blocking issue for us.Any help would be highly appreciable.

  38. Prashant says:

    Hi All,

    I am setting up configuration for executong codedui test in parallel,

    I have set up environment and added 2 agent machines, now as i am new in VSTS and MTM can any one suggest me from where to queue test in MTM and hwo to start execution.

    Quick response should be highly appreciated

  39. Prashant,

    You should do the following: –

    1. Import the test case to your test plan if they are not imported already.
    2. Select the above tests in the testing center (MTM -> Test) and do a run with options specifying the environment.

    You can find more details about this here.…/dd286580(v=vs.110).aspx


    Aseem Bansal

  40. One of my colleague (Ravi) gave me a very good link about the flow.…/dd380741.aspx

    Please try this out and do let me know in case you have any further questions.


    Aseem Bansal

  41. Prashant says:

    Thanks a ton for quick response,

    Aseem, I have setv environment and already we have manual test cases in out test plan, can you suggest me how to import automation scripts in test plan.

    again, wuich response will be highly appreciated 🙂

  42. The msdn link which I have mentioned above contains the steps to import the automated tests in to test plan as well. Copy pasting the steps from the above link.


    Aseem Bansal

    1.Open a Developer Command Prompt, and change directory to the output director of your Visual Studio solution.

    cd MySolutionMyProjectbinDebug

    2.To import all the test methods from the solution:

    tcm testcase /collection: CollectionUrl /teamproject:MyProject /import /storage:MyAssembly.dll  /category:"MyIntegrationTestCategory"

    The category parameter is optional but recommended. You only want to create test cases from integration or system tests, which you can mark by using the [TestCategory ("category")] attribute.

    3.In the Test hub in Team Web Access or in Microsoft Test Manager, use Add Existing to add the test cases to a test suite.

  43. Prashant says:

    Is it necessary to configure build ID  under build heading for automation run.?

  44. Prashant says:

    In addition to above comments,  Automation run is throwing error "The build directory of the test run either does not exist or access permission is required." is it necessary to enter build  as in the settings build is not required.

  45. To execute the test cases, you need to tell the location of your test binaries and build is one way of telling it. If you have a local build and you want to pass it as well, then you will have to use the command-line to do it. See the following blog post on how to use the command-line and pass the build #.…/how-to-execute-tests-locally-using-tcm-exe-command-line.aspx


    Aseem Bansal

  46. Prashant says:

    Thanks Aseem,

    I have done basic configuration and setup agent controller setup..

    I am getting  error "cannot create automated test run. A build associated with the active test plan is required to submit an automated test run".

    Can you please suggest what is this error about|?

    would appreciate quick response as the issue is delaying my work,.

  47. Prashant says:

    Thanks  for sharing this info, however we have multiple build ID and build is not locally placed. also I tried selecting specific build in filter for build dropdown there is no Build available under available build dropdown to assign this build to a plan…

    Is there a configuration issue that no build is available (as more likely it looks) or any configuration I am missing.

    What I did-

    –> selected build id in test plan settings.

    –> Saved the configuration.

    –> Trying assigning the selected build to same plan, getting no build available in dropdown..

    Please suggest, may be this can be a weird/silly issue for you 🙂 but it has now become a showstopper for us 🙂 ..please suggest.

  48. Prashant says:

    Is it necessary to select build ID etc , I need to know as we are using/invoking URL(updated) in our scripts… Please let me know if we can skip this step and how?  

  49. Prashant,

    You need to have a build which contains tests. Another user had faced similar issue, can you please check the instructions mentioned here works for you or not?…/running-automated-tests-in-microsoft-test-manager


    Aseem Bansal

  50. Jatin says:

    I am getting below error while executing parallel execution:-

    Unit Test Adapter threw exception: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information..

    Please advise

  51. Jatin says:

    I have got the below log:-

    <Description>The description for Event ID '0' in Source 'VSTTExecution' cannot be found.  The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them.  The following information is part of the event:'(QTAgent32_40.exe, PID 5464, Thread 14) Unit Test Executor threw an exception. Most likely the test run was aborted by the user or the framework:System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

      at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)

      at System.Reflection.RuntimeModule.GetTypes()

      at System.Reflection.Assembly.GetTypes()

      at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.GetAssemblyInfo(Assembly assembly)

      at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.GetTypeInfo(Type type, Boolean checkAlreadyExaminedType)

      at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.GetTypeInfo(Type type)

      at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.ResolveMethods()

      at Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestExecuter.Initialize(UnitTestResult result, UnitTestRunner runner, ConsoleOutputRedirector redirector)'</Description>

    How to solve this issue

  52. Prashant says:

    Hi Aseem,

    Request you to help us on above error shared by Jatin, the same exception/error occured while we started execution.

    Apprecate your help.



  53. pankaj says:

    Hi Aseem,

    We are facing some issue while executing VSTS scripts in parallel.

    While running all scripts in automation manual run got started and autoamtion did succed in that case if we are running 1 automation script at a time from MTM then it is running.

    How can we runn all scripts using automation in one go

    without starting manual run.

    Please suggest.

  54. Jatin/Prashant, Are the tests running on your machine using mstest.exe? It looks like one of required assembly is missing in your build.

    Pankaj, Do you want to convert your manual test cases to coded UI tests?

    Jatin/Prashant/Pankaj, Can you please post your question on the forum for faster replies by the community?…/home


    Aseem Bansal

  55. Prashant says:

    It worked Aseem, Thanks a lot for your support….

    What all benefits do you think it provides as compare to executing on single machine.

  56. Prashant, The underlying engine is same in both the cases, so if it is possible for you to use your single machine you can continue doing that.  

    But there are cases when you cannot use your machine because of resource constraints or time taken, then you would want framework to use the lab machine(s) and distribute the tests across machines so that they can complete as fast as possible.

  57. Jatin says:

    Hey Aseem. Is it possible to assign multiple builds for different environment.

    e.g. I have created 2 environment in MTM. One is Dev and other is Test environment. Now, i want the different build for Dev and different build for test. Is So that when i run the tests, i can select the build.

  58. Yes, You can do that. There is an action called "Run with options" in MTM with which you can specify the build # for the new test run.…/dd553270(v=vs.110).aspx


    Aseem Bansal

  59. Jatin says:

    Hi Aseem, thanks for the clarification. I have created two environments. But the problem I am facing is that one environment works and if i run other environment, then "Waiting for Controller" is displayed under Analyze Test runs link. Only one environment is working. If i want to run both the environments then is there any settings in config file. Appreciate your inputs.

  60. Syed says:

    I have 120 tests with the BucketSize = 100 and MinimumTestsPerAgent = 1 and out of the 120 tests the last 20 are data driven with each of them running say 30 iterations each. Effectively if I count each iteration as a test, I have 20 * 30 = 600 + 100 = 700 tests. How would the distribution work in this scenario?

  61. Syed says:

    Found the answer myself. The bucket size has no regard to number of iterations a test can have (when data driven). It treats a test with one iteration the same a test with 100 iterations.

    I suppose Microsoft could do more here..

    1) Provide easy accessibility to settings like MinimumTestsPerAgent and BucketSize

    2) Implement more settings that will enable user to configure the test runs to make the most efficient use of the resources available.

  62. Thanks Syed. These are good suggestions. We have added these asks to our backlog for consideration.

  63. Syed says:

    I have hit another blocker now!  

    My tests take about 6 hours if run on one machine so I distributed them across 3 VMs and managing a overnight run with 2 hours appox. As it turns out  some tests that update the database of AUT as they run are ending up in race conditions and the validation of one test in one VM is done on update done from another test in another VM !!

    Do we have any solution to solve this problem already? Could not find any after a good deal of search.

    If not, It looks like we need some flag on test suites which can be used to choose to ignore or pay respect to settings like MinimumTestsPerAgent and BucketSize.  So if a Build  runs two suites A and B with the flag in question set on and off respectively; all tests of test suite A will be distributed and run in parallel and all tests of test suite B will be run sequentially and dispatched to one VM …(or if a sequential run can be ensured still being a distributed run, that would be an out of the world solution !).

  64. HI Syed

    We unfortunately do not have fine grained control on how to restrict/avoid distribution of tests within a run.

  65. Prachi Bora (MSFT) says:

    Hi Syed,

    Have you explored ordered tests? Ordered tests will ensure that tests run in the specified order.

  66. Ruth says:

    Thank you so much for the information! Finally figured it out, thanx to you!

  67. Ruth says:

    Some tests run twice(cannot tell if on different machines or on the same machine). How come?

  68. Mayank says:


    I have 30 Test scripts and 6 Machines. So lets assumes it will run 5 on each machine. I have written code to send result email in my Test Cleanup. Now every agent is sending emails so I am getting 5 emails. But i want one single email.

    Could you please tell me how can i implement it?

  69. sudha says:


    I have two test scripts associated to 2 test cases in MTM. Also i have configured lab manager with one environment having 2 machines .

    I want to run both the scripts simultaneously on two machines . can you please let me know what other setting do i need to do so that i can run 1 TS(test Script) on 1 m/c and other TS on other m/c simultaneously ?

  70. Prachi says:


    How to send a consolidated single email in parallel execution. As i got two emails for one of each agent.

    How to get one email.

  71. Hi Prachi

    Have you tried to invoke your send email from the cleanup script, which you can specify in your testsettings?

Comments are closed.

Skip to main content