Recursion, and ItemGroups inside Targets

  • Article

Greetings MSBuilders!

An internal team (Office Live, I think) asked us how to make their build start up faster. Their traversal project at the root of their tree started like this:

<Project xmlns="https://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Databases\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\External\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Library\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Tools\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Utils\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\WebControls\**\*.csproj"/>

    <DotNetProjects Include="$(BuildPath)\**\*.csproj" Exclude="@(ExcludedDirectory)"/>
    <NativeProjects Include="$(BuildPath)\**\*.CPPproj" Exclude="@(ExcludedDirectory)" />
    <WebProjects Include="$(BuildPath)\**\*.webproj" Exclude="@(ExcludedDirectory)" />
    <WixProjects Include="$(BuildPath)\**\*.wixproj" Exclude="@(ExcludedDirectory)" />
    <LocProjects Include="$(BuildPath)\**\*.locproj" Exclude="@(ExcludedDirectory)" />
  </ItemGroup>

...

The way they had this set up, was their developers to run a shared batch file at any point in the tree. It would set an environment variable BuildPath to the current directory, and then build this shared traversal project. The traversal project would find every project below the current directory, and build it. This allowed them to avoid putting traversal projects at every point in the tree and remember to add new projects to them. But the wildcard evaluation was slow.

They changed it to this

<Project InitialTargets="FindProjects" xmlns="https://schemas.microsoft.com/developer/msbuild/2003">

  <ItemGroup>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Databases\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\External\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Library\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Tools\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\Utils\**\*.csproj"/>
    <ExcludedDirectory Include="$(BuildPath)\**\Common\WebControls\**\*.csproj"/>
    <AllProjects Include="$(BuildPath)\**\*.*proj" Exclude="@(ExcludedDirectory)" />
  </ItemGroup>

  <Target Name="FindProjects">
      <ItemGroup>      
       <DotNetProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.csproj'" />
       <NativeProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.cppproj'" />
       <WebProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.webproj'" />
       <WixProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.wixproj'" />
       <LocProjects Include="@(AllProjects)" Condition="'%(AllProjects.Extension)'=='.locproj'" />
      </ItemGroup>
   </Target>
  
   ...

With the old approach, this took 12.6s to list all projects in the tree to the console. With the new approach target it takes 3.1s to list all projects in the tree. Of course, most of the time, the build starts lower in the tree, so the time taken is smaller. I wonder if they could have reduced the ExcludedDirectory line to just one line to make it faster still?

    <ExcludedDirectory Include="$(BuildPath)\**\Common\**\*.csproj"/>

This example illustrates a new feature you'll find in Orcas Beta 1 -- ItemGroup and PropertyGroup are now allowed inside Targets. (In MSBuild v2.0 you would have to use CreateItem and CreateProperty instead, which were ugly and inconsistent.) I moved the ItemGroup inside the Target here, so that I could use batching to exclude specific extensions from each list.

I'll be blogging in detail about this new syntax in a future post.

Have fun!
[ Author: Dan, MSBuild developer ]

Update: They couldn't have a single ExcludedDirectory line as I suggested above, because one directory in that tree (Common\Platform) needed to be included. So they now do this:

    <AllProjectFiles Include="$(BuildPathActual)\**\*.*proj" Exclude="$(BuildPathActual)\**\Common\**\*.csproj" />
    <!--We need to re-include the platform directory, so that it gets built-->
    <AllProjectFiles Include="$(BuildPathActual)\**\Common\Platform\**\*.csproj"/>

Which takes it down from 11 to 3 traversals.