How to running multiple tests in team build?


 


In beta3 you have better control on the way tests are executed. Some common scenarios are:-



  • To run tests of different categories:-



    • You need to modify tfsbuild.proj file to introduce additional test category. For example if your metadata item is defined as


<MetaDataFile Include = “$(SolutionRoot)\HelloWorld\HelloWorld.vsmdi”>


                   <TestList> BVT1 </TestList>


     </MetaDataFile>


             and you want to execute another category of test (i.e. BVT2). You can easily add the additional test category.



<MetaDataFile Include=”$(SolutionRoot)\HelloWorld\HelloWorld.vsmdi”>


                   <TestList> BVT1; BVT2 </TestList>


     </MetaDataFile>





    • Do note that build will fail if one of the following things hold true

      • Test category BVT2 is not defined in HelloWorld.vsmdi
      • BVT2 contain tests that evaluates to inconclusive


  • Using multiple vsmdi files to running tests:-


    • This is not working on beta3 bits. There is a bug in file Microsoft.TeamFoundation.Build.targets. You need to make the following change at two locations in above mentioned targets file.  

From:-


<TestToolsTask


            Condition = ” ‘$(IsDesktopBuild)’ ! = ‘true’  “



To:-


<TestToolsTask


Condition =” ‘$(IsDesktopBuild)’ ! = ‘true’ and ‘ %(MetaDataFile.Identity)’ ! = ”



              Do note that this is fixed in post beta3 sources.




    • If want to run additional tests defined in different vsmdi file using team build. You need to define new item for the corresponding vsmdi file. For example you want to run test of the category “BVT2” defined in HelloWorld2.vsmdi. You need to do the following steps:-

      • Modify tfsbuild.proj file and add new MetaDataFile item for HelloWorld2.vsmdi file 


      <ItemGroup>


 


<MetaDataFile Include = “$(SolutionRoot)\HelloWorld\HelloWorld.vsmdi”>


                   <TestList> BVT1 </TestList>


</MetaDataFile>


 


<MetaDataFile Include = “$(SolutionRoot)\HelloWorld\HelloWorld2.vsmdi”>


                   <TestList> BVT2 </TestList>


</MetaDataFile>


 


      </ItemGroup>



  • Running tests selectively for particular “platform|configuration” combination:- 


    • Make sure you have the above mentioned fix of “Microsoft.teamfoundation.build.targets” file.
    • Modify tfsbuild.proj file. For example if you want to run the test for only “Debug|Any CPU” configuration, you need to modify the corresponding MetaDataFile item and add the condition (evaluate this item value only when “platform|configuration” is “Any CPU|Debug”)


<MetaDataFile


            Condition = ” ’$(Platform)’ = = ’Any CPU’ and ‘$(Flavor)’ = = ’Debug’ ”


Include = “$(SolutionRoot)\HelloWorld\HelloWorld.vsmdi”>


                       <TestList>BVT1</TestList>


</MetaDataFile>


 


Comments (3)

  1. Pablo says:

    Hi Manish. We have several TBTypes which compile our projects calling MSBuild directly (without using the SolutionsToBuild tag).

    Everythings works fine except for the execution of our test cases. We are using the following tag but they are being ignored by the teambuild:

       <MetaDataFile Include="$(SolutionRoot)Sourceproj1proj1.vsmdi">

         <TestList>Team Build Test Cases</TestList>

       </MetaDataFile>

    Is it because we do not use SolutionToBuild?

    Everything is checked in and also have :

       <RunTest>true</RunTest>

    Thanks.

  2. Shrinivas Kundarapu says:

    Hi

    Above sample Condition property work

    Please use below code

    <MetaDataFile Condition=" ‘$(Configuration)|$(Platform)’ == ‘Release|Any CPU’ "

    Include = "$(SolutionRoot)HelloWorldHelloWorld.vsmdi">

                          <TestList>BVT1</TestList>

    </MetaDataFile>

  3. Pietro says:

    Hi,

    I set up the following in my .proj file:

    <MetaDataFile Condition="’$(Configuration)|$(Platform)’ == ‘Release|Any CPU’"

                     Include="$(BuildProjectFolderPath)/NewConnectionPrj.vsmdi">            

         <TestList>BVT1;BVT2;</TestList>

       </MetaDataFile>

    i have also <RunTest>true</RunTest>

    but i receive the following error from msbuild:

    C:Program FilesMSBuildMicrosoftVisualStudioTeamBuildMicrosoft.TeamFoundation.Build.targets(1264,5): error MSB4062: The "TestToolsTask" task could not be loaded from the assembly PrivateAssembliesMicrosoft.VisualStudio.QualityTools.MSBuildTasks.dll. Could not load file or assembly ‘file:///C:PrivateAssembliesMicrosoft.VisualStudio.QualityTools.MSBuildTasks.dll’ or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, and that the assembly and all its dependencies are available.

    Do you have a clue?