How to run tests in a build without test metadata files and test lists (.vsmdi files)

[UPDATE 6/16/2010]  The VSTS 2008 release added support for test containers (/testcontainer) in the product, and the 2010 release added support for test categories.  This post now only applies to TFS 2005.

Since the beginning, running tests in Team Build (or MSBuild in general) has meant having to use .vsmdi files to specify the tests to run.  Tons of people have complained about it, as it’s a burden to create and edit the files, as either VSTS for Testers or the full suite is required in the 2005 release, and merging changes to the file is painful when multiple developers are updating the file.  While mstest.exe has a /testcontainer option for simply specifying a DLL or load/web test file, the test tools task used with MSBuild does not expose an equivalent property.

Attached to this post is a zip archive containing the files needed to run tests without metadata.  There are three files.

  1. TestToolsTask.doc contains the instructions for installing the new task DLL and modifying the TfsBuild.proj files to use it.
  2. Microsoft.TeamFoundation.Build.targets is a replacement for the v1 file by the same name that resides on the build machine.
  3. Microsoft.TeamFoundation.PowerToy.QualityTools.dll contains the new TestToolsTask class that extends the TestToolsTask class that shipped in v1 and exposes a TestContainers property that is is the equivalent of mstest.exe’s /testcontainer option.

After you read the instructions (please, read the instructions), you’ll be able to run all of your unit tests by simply specifying the DLLs or even specifying a file name pattern in TfsBuild.proj.  Here are a couple of examples.  The TestToolsTask.doc file explains how they work, including what %2a means.

<TestContainerInOutput Include=”HelloWorldTest.dll” />
<TestContainerInOutput Include=”%2a%2a\%2aTest.dll” />
<TestContainer Include=”$(SolutionRoot)\TestProject\WebTest1.webtest” />

This new task will be included in future releases of the Team Foundation Power Toys (soon to be called Power Tools).  The TestToolsTask in the shipping product will support test containers and Team Build will support using it in the next release of the product.

I started with code and documentation that someone else wrote and finished it off.  Thanks to Clark Sell for digging up an old email thread about the original work and Tom Marsh, development lead in VSTS Test, for pointing me to the code and doc.  Thanks to Aaron Hallberg and Patrick Carnahan, Team Build developers, for their help.  Thanks to Brian Harry for letting me post it.

Please post your feedback in the comments to this post.  We’d like to know if this hits the mark, or if there’s something else we need to support.

[UPDATE 4/26/2007] New features: Test categories and test names

Pierre Greborio, a developer over in MSTV, has contributed a great new feature: test categories.  Those of you who have used NUnit are probably familiar with the Category attribute.  Test categories allow you to execute specific groups of unit tests.  Unlike the test container feature, the test category feature will not be in Orcas.

While the details are discussed in the TestToolsTask.doc included in the zip file attached to this blog post, here’s the quick version.

  1. Add the Category attribute to your unit test method.
  2. Specify the category to run in your tfsbuild.proj file.

The other feature that’s new in this release is the support for test names.  This is equivalent to the mstest.exe /test command line switch.  The name that’s specified is implicitly a wildcard match.  Specifying “Blue” as the test name, for example, will execute every test method that has Blue in the name, including DarkBlue and BlueLight.

Pierre did his testing on Vista.  Because the dll is not signed, he ran into some trust issues.  If you hit a similar problem, he recommends this forum post for how to get it to work.

[UPDATE 11/9/2006] Bug fix

I’ve updated the attachment with a new zip archive file.  Unfortunately, the task I posted originally didn’t result in builds being marked as Failed when the tests had errors.  I missed that in my testing.  The reason for the problem is that the v1 Team Build logger looks for the task name, TestToolsTask, and the power toy task was originally called TestToolsTaskEx.  With this new release, the task has the same name as the original v1 task, so that builds will be marked as failed when the tests fail.

If you downloaded the original release, you simply need to copy the Microsoft.TeamFoundation.Build.targets and Microsoft.TeamFoundation.PowerToys.Tasks.QualityTools.dll files from the zip to get the fix (see the Word doc for the paths).

Thanks to Thomas for pointing out this bug!

tags: , , , , , ,

Comments (46)

  1. 在執行 Team Build 時,如果要配套 Team Testing, 有個缺點,你必須要先採用 Test Manager 來歸類整理暨有的 Test Case. 這是個美意卻也造成一點缺點,因為如果你安裝的版本

  2. Clark Sell says:

    That blasted VSMDI file. Well Buck, Tom and crew did it. They have finally given us the ability to run

  3. Tim Dallmann says:

    Just wanted to pass on a big THANK YOU for putting this together.  While we do have team suite in house, I find it much simpler to just include all unit tests within a set of dll’s and have them automagically added to the build.  I’ve only just added it to a few of our team builds, but so far it’s working flawlessly!  

  4. This has always been an issue for my customers! Link to Buck Hodges : How to run tests without test metadata

  5. anutthara says:

    Awesome stuff! Must come in really handy…so, the "all tests" test list need is really obliviated now, ain’t it?

  6. One of the big complaints we’ve gotten from customers trying to build a Continuous Integration or similar

  7. Another project that sounds promising. Like the last, I’ve heard a lot of noise about this, as well as

  8. I have received this question a couple of times recently. When setting up a Team Build you have the option

  9. bob53050 says:

    YES, this has been a thorn in our side for running tests in our nightly build.

    Thanks for your efforts :)

    Bob Hanson

  10. David Boss says:

    This is just what we needed.  I’ve been updating those damn test lists on a daily basis just to keep up with our CI.

    How many ways can I say thank you!

  11. Thomas says:

    First of all thank you for the task. I have one question:

    Using the task i am observing a slightly different behavior in Team Builds build status. Bevor using TestToolsTaskEx (using test lists) Team build sets the build status on a build with some failing tests to "FAILED". With TestToolsTaskEx Team build sets the build status of a build with some tests failung to "Successfully completed".

    Is this intended or did i miss something in configuration?

  12. buckh says:

    Thanks for the great feedback, everyone.

    Thomas, you are correct.  Sorry, I missed that.  The difference seems to be a build step that’s inserted by the original that’s not there when using test containers.  I’ll look into it and report back.


  13. buckh says:

    Thomas, that was a bug in the original release.  I’ve updated the post and the attachment with the fix.

    Thanks for reporting the problem!


  14. Buck Hodges says:

    I’ve updated the TestToolsTask that supports using test containers, in addition to the test metadata

  15. Ralf says:

    Just one Question: What will happen to the Coverage when your Task is used. afik the assemblies for the coverage are namen within the vsmdi file?

  16. Buck has an update on his blog that explains how to run unit tests on Team Build without using vsmdi

  17. Joe says:

    I know I must be missing something..

    But anywyas,  I love your add-on, it will be so helpful. But I seem to have lost code coverage when I use it. When I ran the  TFS server  the old way I had code coverage (same project), so I’m lead to believe that I have set something up wrong. But before I go down the “What did I mess up path “, I want to make sure that your add-on supports code coverage. So is code coverage suppose to work?

  18. Rob Caron says:

    If you run tests as an MSBuild task in Team Foundation Build, you will probably be interested in a post

  19. buckh says:

    Joe, Dominic Hopton states the following in the forums.

    "You need to specify a .testrunconfig file (which has coverage turned on) to get code coverage with team build."


  20. Jay says:

    Buck,  Pardon me but I am new to all of this.  Just where am I supposed to specify the .testrunconfig file to use?  Is it part of the <TestContainerInOutput> tag?


  21. buckh says:

    Jay, you would need to set the RunConfigFile property to specify the .testrunconfig file (see for a list of all of the TestToolsTask properties).

    The TestContainer and TestContainerInOutput tags are used to specify tests without having to use the MetaDataFile and TestLists properties (the VSTS for Testers SKU is required to generate test metadata files and test lists in VS2005).


  22. Joe says:


    I had Code Coverage  before, so I believe the Config file is correct. But I believe I have figured out the error. The project that does not work, is one of many projects in one Team Project. The problem I believe is that that the system looks for the  localtestrun.testrunconfig at the solution root, which is actually a folder above the project folder. This might be something I have done ( to mess this up that is). But I was able to overcome this be specifying the Config location in the  vsmdi and then reading it in the So for every vsmdi that does not work for CC I just point it to the correct file. If this isn’t necessary or I did something totally wrong, please let me know. And once again great add-on.





    <ConfigLocation>E:TeamBuildsMainProjBuild Foo AppSourcestestFoolocaltestrun.testrunconfig</ConfigLocation>

    Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets

       <!– TestContainer tests for non-desktop builds. –>


             Condition=" ‘$(IsDesktopBuild)’!=’true’ and ‘%(TestContainer.Identity)’ != ” "










             ContinueOnError="true" />

       <!– TestContainer tests for desktop builds. –>


             Condition=" ‘$(IsDesktopBuild)’==’true’ and ‘%(TestContainer.Identity)’ != ” "





             ContinueOnError="true" />

  23. buckh says:

    Okay, so the .vsmdi file allows the config file location to be specified.  Now it makes sense that the behavior would change when you switched away from the .vsmdi file.

    So, did you actually modify the .targets file to use the ConfigLocation property, or did you somehow import the setting from the .vsmdi file?  Normally, that’s the RunConfigFile property (has the same name as the TaskProperty).


  24. Joe says:

    I created a new property in the vsmdi called ConfigLocation  and then had the target read that location. I’m not sure where the system was getting the  org config location, but once I switched to your add-on it was not looking in the correct spot. So I just over-ridded the location and had it look at the vsmdi’s property that I created.

    Change in vmsdi (I added this)

    <ConfigLocation>E:TeamBuildsMainProjBuild Foo AppSourcestestFoolocaltestrun.testrunconfig</ConfigLocation>

    Change I did to the target:


    Hope that makes sense.



  25. buckh says:

    Thanks for the clarification.  If the new property were called RunConfigFile rather than ConfigLocation, it would work without changing the .targets file.  Alternatively, you could put that setting in the TfsBuild.proj file.


  26. Joe says:

    Woops I said vmsdi. I meant the project file. But anyways, your suggestion works great. Thanks for all your help.


  27. Dylan Smith says:

    How can we specify the location of the testrunconfig file in the TfsBuild.proj file?  Can you provide an example?



  28. buckh says:

    Dylan, you would need to set the RunConfigFile property in your TfsBuild.proj for the build type (via the standard msbuild <PropertyGroup> tag).  Team Build will pass that to the TestToolsTask.  You can find the documentation on the TestToolsTask’s RunConfigFile property at


  29. Paul says:

    thank you. I haven’t tried it yet, but this was a LARGE annoyance for us since most people here just have Team Editon for developers

  30. Buck Hodges says:

    Brian Harry posted a TFS roadmap . I’d like to expand on the portion that describes the features specific

  31. Do you want to be able to run tests in Team Build without having to use a .vsmdi file. Well now you can.

  32. volt says:

    this ‘hack’ is a great thing!

    To those who can’t make it running properly – make sure you have in your TFSBuild.proj, tag <RunTest>true</RunTest>

    On my installation it was false by default :)

  33. In my new project, we&#39;re using Team Foundation Server (TFS) as our source code repository and probably

  34. Buck Hodges says:

    We are happy to announce the release of version 1.2 of Team Foundation Power Tools (formerly known as

  35. Buck Hodges says:

    Pierre Greborio , a developer over in MSTV, has contributed a great new feature to the power tool task

  36. If you have done any serious unit testing with Visual Studio Team System and tried to include these tests

  37. Buck Hodges says:

    Aaron pointed out this post by Ben Day that talks about using the RunConfigFile property with a build

  38. Aaron pointed out this post by Ben Day that talks about using the RunConfigFile property with a build

  39. Long ago i learned you really don’t get to know your peers/peeps/coworkers until you spend an evening

  40. GrabBag says:

    This post was originally published here . In part 1 and 2 of this series, I gave an overview of Team

  41. こんにちは! フォーラム オペレーターの服部清次です。 昨日に続きまして、 MSDN フォーラムの Team Foundation Server 関連 FAQ コンテンツ で紹介しました英語スレッドの日本語訳をもう

  42. ScTest says:

    What part of this is still valid with TFS/Visual Studio 2008?


  43. buckh says:

    ScTest, this functionality is built into the 2008 release.  In a tfsbuild.proj file generated by 2008, you will find a section with comments about how to use the feature.  You can also turn it on during the process of creating a new build definition in 2008.