Building projects in parallel



Greetings MSBuilders!


 


Orcas MSBuild introduces a new feature allowing build authors to build projects in parallel. To enable this feature we have introduced a new parameter to the MSBuild task called “BuildInParallel” and a new command line parameter called “maxcpucount” or “m”. Note that if you do not set the “maxcpucount”, even though you set “BuildInParallel” parameter to true, your projects will be built serially.


 


Each MSBuild process (including the main or parent process started from command line) contains a component called the node. Projects are processed and built by the node component. Thus using “maxcpucount” of 2 will result in 2 MSBuild processes. If your projects have dependencies on other projects that are being built in parallel, then you will need to set appropriate project to project references so that dependent projects get built first. For instance in the below example (Example 1) if project d has a dependency on project c, building them in parallel could result in any of these projects being built first or both building concurrently. If project d got built first then your build would fail. In order to prevent this you will need to set up a project to project reference from project d to project c. Thus, if project d is built first, it will first follow the project to project reference and build project c, then it would build project d. Setting a project to project reference include invoking the MSBuild task to build the dependent project. Example 2 briefly demonstrates this.


 


The order at which the projects are built is dependent on the scheduling algorithm of MSBuild. You cannot assume any particular order.


 


Example 1:


a.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003” ToolsVersion=”3.5”>


     <Target Name=”default”>


          <MSBuild Projects=”b.proj;c.proj;d.proj” BuildInParallel=”true”/>


     </Target>


</Project>


 


b.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003 ToolsVersion=”3.5”>


     <Target Name=”default”>


          <Message Text=”Building project b”/>


     </Target>


</Project>


 


c.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003 ToolsVersion=”3.5”>


    <PropertyGroup>


       <OutputType>DLL</OutputType>


       <AssemblyName>c</AssemblyName>


    </PropertyGroup>


 


     <Target Name=”default”>


          <Message Text=”Building project c”/>


     </Target>


</Project>


 


d.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003 ToolsVersion=”3.5”>


    <PropertyGroup>


       <OutputType>EXE</OutputType>


       <AssemblyName>d</AssemblyName>


    </PropertyGroup>


 


    <ItemGroup>


         <Reference Include=“c”/>


    </ItemGroup>


 


     <Target Name=”default”>


          <Message Text=”Building project d”/>


     </Target>


</Project>


 


Command line: msbuild a.proj  /m:2


 


In order to prevent the build from failing in case project c and project d build simultaneously we need to set up the following project to project reference


 


Example 2


c.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003 ToolsVersion=”3.5”>


    <PropertyGroup>


       <OutputType>DLL</OutputType>


       <AssemblyName>c</AssemblyName>


    </PropertyGroup>


 


     <Target Name=”default”>


          <Message Text=”Building project c”/>


     </Target>


</Project>


 


d.proj


<Project xmlns=”http://schemas.microsoft.com/developer/msbuild/2003 ToolsVersion=”3.5”>


    <PropertyGroup>


       <OutputType>EXE</OutputType>


       <AssemblyName>d</AssemblyName>


    </PropertyGroup>


 


    <ItemGroup>


         <ProjectReference Include=”c.proj”/>


    </ItemGroup>


 


     <Target Name=”default” DependsOnTargets=”BuildProjectReferences”>


          <Message Text=”Building project d”/>


     </Target>


 


     <Target Name=” BuildProjectReferences “>


          <MSBuild Projects=”@(ProjectReference)” />


     </Target>


</Project>


 


[Author: Jay, MSBuild Software Engineer]

Comments (20)

  1. Rui Rosa Mendes says:

    I will build solutions (their projects) in paralell too?

    Thanks,

    Rui

  2. DDG says:

    Is this parameter available already? If not then when it will be available?

  3. Derek says:

    DDG, you need to be using the msbuild executable shipped in the 3.5 Framework (currently beta 1). I set it up on the project I’m currently working on — it knocked about 3 seconds off of a full build (12 seconds vs. 9 seconds on a low end T2060-based laptop). Not bad — as we add more modules, I expect this could notch up a bit more.

  4. MSBuild is introducing a new feature in Orcas where projects can be built in parallel – see their team

  5. Mike Fourie says:

    Rui Rosa Mendes,

    Yes, projects within your solution can also be built in parallel when building the solution from the command line. Please note that it will not build in parallel when building from the IDE. You may need to specify /BuildInParallel=true from the command line.

    Thanks,

    Jay (jaysh@microsoft.com)

  6. Mike Fourie says:

    DDG,

    This parameter will be available in the next version of .Net framework (Version 3.5). You can also get this version by installing Visual Studio (code name Orcas) beta versions.

    Thanks,

    Jay Shrestha (jaysh@microsoft.com)

  7. Greetings MSBuilders! Passing properties to projects specifically when you want to pass different sets

  8. Greetings MSBuilders! In Visual Studio 2005 you could only target the 2.0 Framework tools. MSBuild targets

  9. Buck Hodges says:

    While some of the build agent properties are available in the VS GUI, buried in the tfsbuildservice.exe.config

  10. While some of the build agent properties are available in the VS GUI, buried in the tfsbuildservice.exe

  11. Earlier this week, I spent some quality time porting a moderately complicated build over to MSBUILD 3.5,

  12. Rüdiger says:

    Is there any reason why Microsoft is not able to do the compilation of source to obj files in parallel, like GNU make does since years?

    I’d consider this the first and easiest step to improve build performance, especially with the rise of multi-core processors.

  13. GreenReaper says:

    You mean like with <a href="http://msdn.microsoft.com/en-us/library/bb385193.aspx">the /MP compile option</a>?

    It even worked in 2005, although it was undocumented and unsupported, and quite possibly buggy.

  14. Grom says:

    Try to use codeblocks. It supports parallel build.

  15. With Visual Studio Team Build 2008 there is no way to run two builds from within the same Team Project

  16. TeamBuild can build one build definition from a single Team Project at a time. can run multiple build

  17. Alessio says:

    I’m thinking of >1 MLOC C++ solutions that take hours to build. Many companies end up buying from http://www.xoreax.com

    I wonder how difficult it would be to plu-in a more generic grid engine like http://www.digipede.net/

  18. Mirek says:

    Can you explain why provided sample is much faster without /m switch? (the same happens for a solution I’m trying to build)

  19. Leonid Shalupov says:

    Because of the (possibly) bug connect.microsoft.com/…/msbuild-exe-appears-to-hang-when-maxcpucount-is-used-and-a-large-number-of-project-references-exist

    Unfortunately, building of my large solution is much faster with single-process msbuild

  20. Jay Dixit says:

    I am facing a issue related to parallel build. As I have almost 900 projects, I am building them parallel but getting exception of file access. eg. "Unable to copy x.dll into … as it being used by another process."