Updated version of new TestToolsTask


I’ve updated the TestToolsTask that supports using test containers, in addition to the test metadata files supported by v1.  Unfortunately, the task I posted originally didn’t result in builds being marked as failed when the tests had errors.  I missed that fact in my testing.  The reason for the problem is that the v1 Team Build logger looks for the task name, TestToolsTask, and the power toy task was originally called TestToolsTaskEx.  With this new release, the task has the same name as the original v1 task, so that builds will be marked as failed when the tests fail.

If you downloaded the original release, you simply need to copy the Microsoft.TeamFoundation.Build.targets and Microsoft.TeamFoundation.PowerToys.Tasks.QualityTools.dll files from the zip to get the fix (see the Word doc for the paths).

Thanks to Thomas for pointing out this bug!

The updated version is attached to the original post.  I apologize for the inconvenience.

tags: , , , , , ,

Comments (12)

  1. bob53050 says:

    Hello Buck,

    Sorry, but I had problems with the updated changes:

    The following doesn’t occur with the original custom build task:

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaselaborvalidationtest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Data’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaselaborvalidationtest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Windows.Forms’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaselaborvalidationtest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Drawing’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaselaborvalidationtest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Xml’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaselaborvalidationtest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseatmelcommstest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Windows.Forms’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseeismainapptest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Drawing’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseeismainapptest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseeismainapptest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Xml’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseeismainapptest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Data’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseeismainapptest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleasedatabasehandlertest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Data’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleasedatabasehandlertest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Windows.Forms’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleasedatabasehandlertest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Drawing’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleasedatabasehandlertest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Xml’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleasedatabasehandlertest.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Windows.Forms’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseapisoftware.windowsce.test.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Drawing’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseapisoftware.windowsce.test.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseapisoftware.windowsce.test.dll’ was not found.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : Warning: Test Run deployment issue: The assembly or module ‘System.Xml’ directly or indirectly referenced by test assembly ‘c:joshuanightlybuildjoshuanightly buildbinariesreleaseapisoftware.windowsce.test.dll’ was not found.

    I switched back to the original one and these errors are resolved.

    Thanks for your help :)

    Bob Hanson

  2. buckh says:

    I’m not sure what’s going on here.  You posted those warnings earlier in the forums, but you said that you found a solution (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=906826&SiteID=1).

    There’s a similar issue described on Connect: http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=121440.  The bug was ultimately closed, so I assume that Winnie’s comment about CopyLocal=true was the root of the problem since it wasn’t reactivated.

    Buck

  3. MMann says:

    Hello Buck,

    The library is of great help.  There is however one problem that I am having and that is I use the

    <TestContainerInOutput Include="<testdll>.dll" /> in our TFSBuild.proj and everything works fine until the tests attempt to run as the test dll cannot be found.  I believe the reason is that the path to the dll is showing up with a \ at the end of the folder path in the build log.  

    for example:  C:<folder to binaries>\<test dll>.dll

    Any ideas?

    -Matthew Mann

  4. buckh says:

    Matthew, when I’ve seen that occur, the task has handled it.

    One thing you can do to test the theory is use the TestContainer tag instead and hard code the path to the dll to see if it works when it doesn’t have the double slash.

    Buck

  5. bob53050 says:

    Hello Buck,

    As you mentioned, I did have the resolution to the problem which is the first version of your custom build. The second version is what is causing the errors so when I reverted back to the first version, I do not have the errors.

    Please let me know if you have a suggestion for the second version.

    Thanks for your help :)

    Bob Hanson

  6. buckh says:

    I’m stumped at the moment.  I can’t think of why there would be a difference here, since both the old and new ultimately invoke the same test tools code that loads and runs tests.

    Buck

  7. Jason says:

    I’m getting unusual exceptions with this task… any ideas? Here’s my line from the build project on the build server:

    <TestContainerInOutput Include="TestPermissions.dll" />

    And here’s the errors:

    Target RunTestWithConfiguration:

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018: The "TestToolsTask" task failed unexpectedly.

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018: System.ArgumentNullException: Value cannot be null.

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018: Parameter name: path1

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018:    at System.IO.Path.Combine(String path1, String path2)

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018:    at Microsoft.VisualStudio.TestTools.TaskHelpers.ReflectAssemblyHelper.LoadAssembly()

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4018:    at Microsoft.TeamFoundation.PowerToys.Tasks.TestToolsTask.set_TestContainers(String[] value)

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(617,11): error MSB4026: The "TestContainers=%(TestContainer.Identity)" parameter for the "TestToolsTask" task is invalid.

           C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets(609,5): error MSB4063: The "TestToolsTask" task could not be initialized with its input parameters.

       Done building target "RunTestWithConfiguration" in project "TFSBuild.proj" — FAILED.

       Done building project "TFSBuild.proj" — FAILED.

  8. Adam Singer on Get your users for nothin’ and your sync for free. Buck Hodges on Updated version of…

  9. buckh says:

    Jason, what’s happened is that code is trying to use the "InstallDir" value at the registry key "hklmsoftwaremicrosoftvisualstudio8.0," which is apparently not set on your build machine for some reason.  The code should clearly check for null and produce a better error message (I’ll fix that), but it appears that something isn’t installed properly on your build machine.

    Buck

  10. Glenn says:

    Hi,

    I’m seeing these warnings in the build log and wondered if anyone can help?

    Target RunTestWithConfiguration:

    TestToolsTask SearchPathRoot="C:BuildsTestProjectTest BuildBuildType..Sources..BinariesRelease" PathToResultsFilesRoot="C:BuildsTestProjectTest BuildBuildType..Sources..TestResults" PublishServer="http://vstf-server:8080/&quot; TeamProject="TestProject" PublishBuild="Test Build_20061121.10" BuildFlavor="Release" Platform="Any CPU"

    Loading C:BuildsTestProjectTest BuildBuildType..Sources..BinariesRelease\Photography.Metadata.Tests.dll…

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : TestTypes{37e36796-fb51-4610-8d5c-e00ceaa68b9f} could not be loaded because the TIP could not be instantiated for the following reason(s): Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.LoadTestPackage, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : TestTypes{4e7599fa-5ecb-43e9-a887-cd63cf72d207} could not be loaded because the TIP could not be instantiated for the following reason(s): Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.LoadTestPackage, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : TestTypes{7a24f8ff-85da-d664-ef44-83ff4743dae6} could not be loaded because the TIP could not be instantiated for the following reason(s): Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.Tips.ManualTest.Tip, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : TestTypes{982b8c01-1a8a-48f5-b98a-67ee64bc8687} could not be loaded because the TIP could not be instantiated for the following reason(s): Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.Tips.GenericTest.Tip, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

    C:Program FilesMSBuildMicrosoftVisualStudiov8.0TeamBuildMicrosoft.TeamFoundation.Build.targets : warning : TestTypes{fa4c85ff-1236-4463-b954-3ba2f0eabd95} could not be loaded because the TIP could not be instantiated for the following reason(s): Could not load file or assembly ‘Microsoft.VisualStudio.QualityTools.LoadTestPackage, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

    Starting execution…

    I also have code coverage enabled for the unit tests  that are run during the build but when I look at the code coverage results all I see is "Code coverage is not enabled for this test run".  Any ideas why this could be happening, I assume that the TestToolsTask can produce the code coverage results?

  11. buckh says:

    Glenn, in order to run tests on the build machine, you must have VSTS for Testers or VSTS Suite installed on the build machine (unit tests will work with VSTS for Developers installed, but I don’t think load tests will).  The necessary dlls and other files are not installed on your build machine, based on the messages.

    You don’t have to buy a separate license for the build machine if it’s going to be used by folks licensed for one of the previously mentioned flavors of VSTS.  Please scroll to the section about the build machine (it’s short and readable) in the crosoft Visual Studio 2005 Team System Licensing White Paper at http://www.microsoft.com/downloads/details.aspx?familyid=1FA86E00-F0A3-4290-9DA9-6E0378A3A3C5&displaylang=en and read the details (I’m no expert on licensing).

    Buck

  12. Glenn says:

    Thanks for the information Buck, thought that might be the case.

    Glenn.