MSBuild Batching - Generating a Cross-Product

  • Article

I've been playing around with the batching functionality in MSBuild quite a bit lately, and thought I'd share my various little discoveries with the world...  If you aren't sure what batching is in MSBuild, check out this MSDN page.  Alternatively, here's my five second explanation - batching allows you to simulate a foreach loop over the Items in an ItemGroup.  For example:

 <?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="PrintFoo" xmlns="https://schemas.microsoft.com/developer/msbuild/2003" >

 <ItemGroup>
       <Foo Include="foo1">
          <FooMetadata>1</FooMetadata>
        </Foo>
        <Foo Include="foo2">
          <FooMetadata>2</FooMetadata>
        </Foo>
    </ItemGroup>

  <Target Name="PrintFoo">

      <Message Importance="high" Text="FooMetadata=%(Foo.FooMetadata)" />

   </Target>

</Project>

The Message task in the above MSBuild file is effectively saying something like:

 foreach (Item foo in ItemGroup)
{
    <Message Importance="high" Text="FooMetadata=" + foo.FooMetadata />
}

To try it out, just cut and paste the code into a text file and run MSBuild.exe on it - you should get the following output: 

Project "c:\foo.txt" (default targets):

Target PrintFoo:
FooMetadata=1
FooMetadata=2

This isn't quite right, since really the batching occurs over collections of items with identical values for the referenced metadata property (see the above reference MSDN page for details on this).  But - to a first approximation this is what is going on.

Things get interesting when you want to do a loop within a loop...  Consider the following:

 <?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="PrintFooAndBar" xmlns="https://schemas.microsoft.com/developer/msbuild/2003" >

   <ItemGroup>
       <Foo Include="foo1">
          <FooMetadata>1</FooMetadata>
        </Foo>
        <Foo Include="foo2">
          <FooMetadata>2</FooMetadata>
        </Foo>
    </ItemGroup>

  <ItemGroup>
       <Bar Include="bar1">
          <BarMetadata>a</BarMetadata>
        </Bar>
        <Bar Include="bar2">
          <BarMetadata>b</BarMetadata>
        </Bar>
    </ItemGroup>

  <Target Name="PrintFooAndBar">

        <Message Importance="high" Text="FooMetadata=%(Foo.FooMetadata), BarMetadata=%(Bar.BarMetadata)" />

   </Target>

</Project>

The first time I tried this, I fully expected to get output like:

Project "c:\foobar.txt" (default targets):

Target PrintFooAndBar:
FooMetadata=1, BarMetadata=a
FooMetadata=1, BarMetadata=b
FooMetadata=2, BarMetadata=a
FooMetadata=2, BarMetadata=b

Instead, though, I got the following:

Project "c:\foobar.txt" (default targets):

Target PrintFooAndBar:
FooMetadata=, BarMetadata=a
FooMetadata=, BarMetadata=b
FooMetadata=1, BarMetadata=
FooMetadata=2, BarMetadata=

When MSBuild encounters two collections to batch over, that is, it apparently batches one and then the other - not both at the same time.  Back in our foreach syntax, it does something like:

 foreach (Item foo in ItemGroup)
{
    <Message Importance="high" Text="FooMetadata=" + foo.FooMetadata + ", BarMetadata=" />
}

foreach (Item bar in ItemGroup)
{
    <Message Importance="high" Text="FooMetadata=, BarMetadata=" + bar.BarMetadata />
}

How, then, can we get MSBuild to do something more like:

 foreach (Item foo in ItemGroup)
{
   foreach (Item bar in ItemGroup)
 {
       <Message Importance="high" Text="FooMetadata=" + foo.FooMetadata + ", BarMetadata=" + bar.BarMetadata />
  }
}

I have come up with two solutions - perhaps you have others?  If so, please post them in the comments!

(1) Use an MSBuild task to call back into our project file.

In this solution, we use the $(MSBuildProjectFile) property (a reserved property in MSBuild) to call back into our project file.

 <?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="PrintFooAndBar" xmlns="https://schemas.microsoft.com/developer/msbuild/2003" >

    <ItemGroup>
       <Foo Include="foo1">
          <FooMetadata>1</FooMetadata>
        </Foo>
        <Foo Include="foo2">
          <FooMetadata>2</FooMetadata>
        </Foo>
    </ItemGroup>

  <ItemGroup>
       <Bar Include="bar1">
          <BarMetadata>a</BarMetadata>
        </Bar>
        <Bar Include="bar2">
          <BarMetadata>b</BarMetadata>
        </Bar>
    </ItemGroup>

  <Target Name="PrintFooAndBar">

        <MSBuild Projects="$(MSBuildProjectFile)" Targets="InternalPrintFooAndBar" Properties="Foo=%(Foo.FooMetadata)" />

 </Target>

 <Target Name="InternalPrintFooAndBar">

        <Message Importance="high" Text="FooMetadata=$(Foo), BarMetadata=%(Bar.BarMetadata)" />

   </Target>

</Project>

There are various downsides here.  The Message task does get executed for each combination of FooMetadata and BarMetadata, but in two separate invocations of the MSBuild task.  Our output looks like this:

Target PrintFooAndBar:
__________________________________________________
Project "c:\foobar.txt" is building "c:\foobar.txt" (InternalPrintFooAndBar target(s)):

    Target InternalPrintFooAndBar:
FooMetadata=1, BarMetadata=a
FooMetadata=1, BarMetadata=b
__________________________________________________
Project "c:\foobar.txt" is building "c:\foobar.txt" (InternalPrintFooAndBar target(s)):

    Target InternalPrintFooAndBar:
FooMetadata=2, BarMetadata=a
FooMetadata=2, BarMetadata=b

Additionally, the collection aspect of batching that I alluded to earlier is lost in this approach.  That is, in the InternalPrintFooAndBar target, the Foo item group has been reduced to a single property value, with all of its ItemGroupness thrown away. 

(2) Write a custom task to generate the cross product.

In this solution, a custom task is used to explicitly generate a new ItemGroup with the cross product of two input ItemGroups.  (More specifically, the cross product of two sets of Metadata values - these could potentially come from the same ItemGroup)

 <?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="PrintFooAndBar" xmlns="https://schemas.microsoft.com/developer/msbuild/2003" >

  <UsingTask TaskName="CustomTasks.ItemGroupCrossProduct" AssemblyFile="E:\My Documents\Visual Studio 2005\Projects\BlogProjects\CustomTasks\bin\Debug\CustomTasks.dll" />

  <ItemGroup>
       <Foo Include="foo1">
          <FooMetadata>1</FooMetadata>
        </Foo>
        <Foo Include="foo2">
          <FooMetadata>2</FooMetadata>
        </Foo>
    </ItemGroup>

  <ItemGroup>
       <Bar Include="bar1">
          <BarMetadata>a</BarMetadata>
        </Bar>
        <Bar Include="bar2">
          <BarMetadata>b</BarMetadata>
        </Bar>
    </ItemGroup>

  <Target Name="PrintFooAndBar">

        <ItemGroupCrossProduct ItemGroup1="@(Foo)" Metadata1="FooMetadata" ItemGroup2="@(Bar)" Metadata2="BarMetadata">

           <Output TaskParameter="CombinedItemGroup" ItemName="CombinedItems" />

     </ItemGroupCrossProduct>

      <Message Importance="high" Text="FooMetadata=%(CombinedItems.FooMetadata), BarMetadata=%(CombinedItems.BarMetadata)" />

   </Target>

</Project>

This post is getting too long, and I have other work to do, so - the code and an explanation of the Task wil have to wait until another day...  (I have to do something to keep you coming back)