Use test categories rather than test lists (vsmdi files)


Years ago, I posted an msbuild task to run tests without test lists for VSTS 2005.  That functionality made it into VSTS 2008.  Then Pierre Greborio added support for test categories.  That test category capability is now available in VS 2010.  Using test categories is now the preferred way of running groups of tests, and you no longer need to deal with tests lists (.vsmdi files), which are tedious to maintain and very difficult to merge.

Here’s the documentation on MSDN to show you how to make use of test categories in the 2010 release.

Defining Test Categories to Group Your Tests

If you have created automated tests using Microsoft Visual Studio 2010, you can manage these tests by categorizing them with test categories. When you run these automated tests using Visual Studio, Team Foundation Build, or mstest.exe, you can use these test categories to easily select which tests you run. You add test categories as attributes to your test methods.

Test categories provide more flexibility than the test lists functionality from earlier versions of Microsoft Visual Studio. You can use logical operators with test categories to run tests from multiple categories together or to limit the tests that you run to tests that belong to multiple categories. Also test categories are easy to add as you create your test methods and you do not have to maintain test lists after you have created your test methods. By using test categories, you do not have to check in and check out the <solution name>.vsmdi file that maintains the test lists.

more…

Technorati Tags: ,
Comments (8)

  1. Brian T says:

    Hi Buck,

    I am trying to use the mstest.exe /Category functionality from the 'Upgrade template' build in VSTS 2010.  

    This does not allow me to set the category property as it would if I were using the 'default template', do you know of a way to pass the /Category parameter to mstest.exe from a tfsbuild.proj ?

    Appreciate your thoughts on this.

  2. brianteare says:

    Just because I was not signed in when I made the 1st comment on this posting, commenting now to associate with my account..

  3. patcarna says:

    Brian,

    It looks like you should be able to do this — unfortunately it involves making a small edit to the Microsoft.TeamFoundation.Build.targets file on the build machine. The TestToolsTask supports a Category property which may be set to the expression you would normally provide to the mstest.exe command line, but we did not expose it as a property for you to set/override in the TfsBuild.proj file. In order to get this functionality you should create a property and pass it to the 'Category' property of the TestToolsTask. You may then set it in your TfsBuild.proj file to run tests based on the category.

    Please let me know if you need further assistance.

    Patrick

  4. Greg says:

    I want to create keyboard shortcuts to run certain categories of tests, like unit tests and not the integration tests. I have keyboard shortcuts for running existing commands, such as "all tests in solution", but I wanted to make use of the categories and make it as easy as possible to run tests.

    Thanks,

    Greg

  5. Rob says:

    Anybody experienced a serious performance hit when using test categories on 64 bit platforms, compared to the old fashioned vsmdi?

    We finally got round to switching to test categories, initially using a category for each list. Running exactly the same set of tests (albeit potentially in a different order) is taking approx 5 times as long. The tests aren't failing.

    Doesn't make any sense. The slow-down is not apparent on 32 bit machines.

    We are using multiple test containers (dlls) and have an expression of the form "cat1|cat2|cat3|cat4"

  6. Abhishek [MSFT] says:

    @Rob, What you are reporting is not a known issue. I will be interested in getting more details to debug this. Will it be possible to provide us a repro.

    Thanks,

    Abhishek Agrawal [MSFT]

    Program Manager, Visual Studio

    abhiag A T msft

  7. Very Long command line says:

    The new way of running MSTest using categories also requires use of testcontainer arguments.

    We have maybe 25 test containers and the path to each of them is up to 50 chars long. This combination leads to a command line of at lease 1200 chars. There has got to be a better way!

  8. buckh says:

    That would certainly be painful.  Is there not a way to use a wildard pattern to pick them up based on a naming convention?  I wouldn't expect you to have to literally list all 25 by full path.

    Buck