Update to "How to run tests in a build without test metadata files and test lists (.vsmdi files)": Test Categories


 Pierre Greborio, a developer over in MSTV, has contributed a great new feature to the power tool task that allows you to run tests without .vsmdi files: test categories.  Those of you who have used NUnit are probably familiar with the Category attribute.  Test categories allow you to execute specific groups of unit tests.  To tag a unit test method as belonging to a category, just add the Category attribute.  Unlike the test container feature, the test category feature will not be in Orcas (if you’d like to see it in a future release, be sure to let these folks know).

The new version of the TestToolsTask, targets file, and documentation are in an updated zip file attached to the original post, which has also been updated (download here).

While the details are discussed in the TestToolsTask.doc file, here’s how to use it.

Using the new Category attribute

IMPORTANT NOTE: The Category attribute feature will not be included in Visual Studio Codename Orcas.

The Category attribute must be specified for every test method that should be included in that test category. Note that there is no way to specify that all tests not in a test category should be run.

In order to use categories you need to:

1. Add a reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll in your project.

2. Add the namespace in the code:
using Microsoft.TeamFoundation.PowerTools.Tasks;

3. Add the Category attribute to the test method:
    [TestMethod]
    [Category(“Integration”)]
    public void MyTestMethod()
    {
        // TODO: Add test code here
    }

Finally, you will either need to specify the TestCategories in TfsBuild.proj if you are using Team Build or modify your project to call the task directly.

Pierre both wrote and tested the code (making heavy use of Reflector in the process).  Thanks, Pierre!

The other feature that’s new in this release is the support for test names.  This just exposes the equivalent to the mstest.exe /test command line switch.  The name that’s specified is implicitly a wildcard match.  Specifying “Blue” as the test name, for example, will execute every test method that has Blue in the name, including DarkBlue and BlueLight.  The TestName property will be available on the Orcas TestToolsTask (beta 2 and beyond).  The details are described in the documentation file.

Comments (10)

  1. Buck posted a new cool update of the Power Tools for Visual Studio 2005 (Team Test and Team System) and

  2. Our good friend Buck Hodges (TFS TeamBuild) has worked with some internal enthusiasts to produce a great

  3. Glenn says:

    Hi Buck,

    Where is the best place to log a bug about v1.3 of the TestToolsTask?  It throws a System.NullReferenceException when used with a test list metadata file?

    Thanks

    Glenn.

  4. Buck Hodges says:

    We’ll look into it.  Thanks for reporting it.

    Buck

  5. The Teams WIT Tools Blog on Understanding the TFS Cube. Buck Hodges on Update to "How to run tests in…

  6. Sander Bruinsma says:

    Hi,

    Our project uses the TestToolsTask to execute all unit tests compiled into different assemblies (about 10 files in total).

    Is it possible to see for wich assembly the different test results are, both in the test results page, and on disk?

    Now the build output window in Vs2005 shows our test results correctly (failed/succeeded etc), but not the assembly name which was used to generate these results (i am talking about the "Test Run" link(s) under "Test results:").

    Also, te subfolders inside the "TestResults" folder on disk do not contain the unittest assembly name.

    Does anyone know if this is possible?

    Thanks,

    Sander Bruinsma, NL.

  7. Kevin Burton says:

    I downloaded the "latest" power toys and I see a document on the whole package but nothing on <TestToolsTask> and there is nothing on Category. This .msi is from 2.26.2007 is there a later version?

    Kevin

  8. Buck Hodges says:

    Kevin, the new Category feature hasn’t made it into the official power tools distribution.  You’ll need to download the zip file attached to http://blogs.msdn.com/buckh/archive/2006/11/04/how-to-run-tests-without-test-metadata-files-and-test-lists-vsmdi-files.aspx, which has been updated.

    Buck

  9. Buck Hodges says:

    Sander, you should be able to see that information in the buildlog.txt file.  However, showing it in the build report would be useful.  I’ll look into whether we can fix that for Orcas.

    Buck

  10. こんにちは! フォーラム オペレーターの服部清次です。 昨日に続きまして、 MSDN フォーラムの Team Foundation Server 関連 FAQ コンテンツ で紹介しました英語スレッドの日本語訳をもう