Running tests in mstest without installing the VS IDE


Yes, you CAN run your unit tests or any other VS test type on a machine without having Visual Studio IDE installed on that machine. You need the Visual Studio Team Agent to be installed on your test machine to run your tests. This is a 15 min lightweight install and is pretty easy to configure.

Those of you familiar with remote testing from our previous versions, will already know that you can trigger a test run on the client to run tests on a remote machine with the test agent installed. Just use the test settings to point to a test agent where your tests can run remotely. With the VS Team Agent installed on machine A and the client (VS Ultimate or Pro or Test Elements) installed on machine B, you can run tests remotely on A by triggering the run from client B.

Another alternate scenario where you would want to run tests without having the Visual Studio IDE installed is during builds. If you use Team Build to do builds and run tests as part of your build process, you can setup the build agent on a machine with no VS IDE installed. You can now go ahead and run all your tests without any additional installs on the build machine in VS 2010.

What if you want to run mstest from the command line though? You can choose to run mstest.exe from either the test agent or the build agent to run tests, but note that you can run only the unit test and ordered test types.

Comments (15)

  1. Rob says:

    Can the Team Agent collect and publish code coverage data or do you still need a VS install for this?

  2. anutthara says:

    Good q. When you run tests remotely  kicking off a run from a client on a team agent, the team agent can collect and publish code coverage data if you are using CC as a collector to run your automated tests.

    However,if you want to run mstest on the agent macine,you cannot collect CC data – only run unit and ordered tests.

  3. VishwasLondhe says:

    Can we execute load tests created in VSTS from CMD line or a script?

    The reason behind the question is we need to execute load tests everytime we take a build(weekly).

  4. anutthara says:

    Sure – just use mstest to run your load test and pass the load test file as an argument. If you want to run as part of team build, you can just configure that in the build definition

  5. Craig says:

    Question: Do the Team Agents require TFS 2010 as well, or could this work with TFS 2008 running VS 2010 Test Projects?

    [Anu] Team agents dont really need TFS to run mstest – it can work standalone. You cannot get TFS 2008 to connect to a 2010 controller or a 2008 controller to connect to a 2010 agent.

  6. autotest says:

    We are using VS2010 Premium to run automated tests (lists of generic tests) for the purpose of collecting test impact data.  When running these tests in the VS IDE a test impact xml file is created for each test in the list and the signature data file (testimpactdata.sdf) is updated.  When running the same lists of tests using mstest, test impact xml file is created for each test but the signature data file is not updated.  Why is this?

    [Anu] As a result, are you not seeing impacted tests?

  7. autotest says:

    In this trial, only subsets of our automated tests were run (the entire test set takes 6 hours).  Running one subset (list) of tests in the VS IDE generates test impact xml files for that set and the signature data file is updated.  Running another different subset using mstest generates test impact files for that set but the signature data file is not updated.  Making changes to methods that are known to be called by tests in the first subset of tests (run using the VS IDE) and rebuilding shows the impacted tests.  Making changes to methods that are known to be called by tests in the second subset of tests (run using mstest) and rebuilding shows no tests are known to call these methods and thus no impacted tests.  It is my understanding that the signature data file contains the actual test impact data and that every time tests are run, the test impact data is updated.

  8. autotest says:

    In this trial, only subsets of our automated tests were run (the entire test set takes 6 hours).  Running one subset (list) of tests in the VS IDE generates test impact xml files for that set and the signature data file is updated.  Running another different subset using mstest generates test impact files for that set but the signature data file is not updated.  After running the tests changes were made to methods that are known to be called by tests in the first subset of tests (run using the VS IDE).  After making the changes and rebuilding the solution, the test impact view shows the tests that are known to call those methods and which ones are impacted.  Changes were also made to methods that are known to be called by tests in the second subset of tests (run using mstest).  After making those changes and rebuilding, the test impact view shows no tests are known to call these methods and thus no impacted tests.  It is my understanding that the signature data file contains the actual test impact data and that every time tests are run, the test impact data is updated.  

    Because our tests take so long to run we prefer to run them at night.  Our builds and tests are run from scripts.  The test script executes mstest and specifies which lists to run.

  9. Raj says:

    Can I run the mstest without test controller for command line in standalone mode i.e. there no server having the test controller ?

    [Anu] Sure Raj – you just have to run the tests locally though

  10. xtran says:

    My test is running fine in IDE, but fails in the command line (mstest). Do you have any ideas?

    Thanks,

    Xtran

    [Anu] XTran – I'll need your test result and logs to debug. Can you post this on the MSDN forums if you haven't already

  11. IrinaM says:

    We have problem to run web.webtest  in the LabCenter. LabCenter environment  has installed: test agent , lab agent (test controller defined on other host); VS2010 is not installed.

    If I run "mstest.exe  /testconteiner:C:qaproduction.webtest"  the next error appeared:

    File extension specified ‘.webtest’ is not a valid test extension

    Please help.

    [Anu] Irina – can you please try running this via mstest on your local client first? You might have a typo in your switches

  12. Chen Xuanyi says:

    Hi!

    I have created a Coded UI Test and it runs fine on my desktop with VS2010 in IDE and MSTest Command Line, but the test failed (unable to hand window handlers ) to run on a MSTest command Line with TestAgent  installed on another PC

    [Anu] You need to run the remote agent as a desktop service http://msdn.microsoft.com/en-us/library/ee291332.aspx

    Thanks,

    Xuanyi

  13. Rakhi says:

    We have installed and configured the Test Controller and Test Agent with Team foundation server default collection. Configured Test Agent is displayed on Manage test controller screen and we have also changed Local.Testsettings file for Remote execution with the configured agent attribute.

    We are not understanding how to trigger Coded UI Test on Remote m/c using Test controller and test agent without using Test manager. Please let us know if you have any solution.

    [Anu] Rakhi – Can you please write to me via my send msdn request so I can help debug your problem?

  14. Justin says:

    The link to the Visual Studio Test Agent doesn't seem to point to the correct place.  Any chance you could update it?

    [Anu] Thanks for reporting this, Justin. The link is now fixed.

  15. The thread in the link below talks about a way to run tests from mstest without installing Visual Studio AND without installing Test Agent:

    social.msdn.microsoft.com/…/4c80703b-db52-495c-b28d-31a52bc36894

    The thread describes how to extract mstest.exe (and all the DLLs and registry entries associated with it), copy them to another machine, and run tests.

    Can you comment on whether that solution is:

    1. valid

    2. recommended

    3. supported by Microsoft