How to: Fail a Build When Tests Fail


Got an email from Martin Woodward this morning asking:

What would be your preferred way to fully fail the build on test failure rather than partially succeed the build?

A bit of background – Team Build 2008 marks builds as Succeeded if no errors are encountered during the build process, Partially Succeeded if compilation succeeds but an error is encountered at some other stage of the build process (e.g. a unit test fails), and Failed otherwise.  Depending on where you are in your development process, this logic may or may not make sense to you.  For example, if you are two weeks away from shipping your product, a unit test failure might be just as important to you as a compilation failure. 

The design of Team Build 2008 doesn’t really allow for setting the overall status of the build directly – the setting of the build status is what indicates to TFS that the build is “complete”, which triggers the build queue logic to start up the next build, etc.  As such, you really shouldn’t ever set this property of the build yourself!

You can get around this by setting the properties Team Build uses to determine the overall status of the build, however – CompilationStatus, TestStatus, and an internal logger property that keeps track of whether any errors have occurred.  Note the above algorithm for determining status one more time – Succeeded if no errors are encountered, Partially Succeeded if CompilationStatus is Succeeded but an error occurs at some other stage, and Failed otherwise.  So – if you want the overall status of the build to be Failed, you’ll need to set CompilationStatus to Failed.

So – here is my recommended approach:

  <Target Name="AfterTest">

    <!-- Refresh the build properties. -->
    <GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
                        BuildUri="$(BuildUri)"
                        Condition=" '$(IsDesktopBuild)' != 'true' ">
      <Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
    </GetBuildProperties>

    <!-- Set CompilationStatus to Failed if TestSuccess is false. -->
    <SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
                        BuildUri="$(BuildUri)"
                        CompilationStatus="Failed"
                        Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">

  </Target>

The idea here is to check whether or not tests have succeeded (see my previous post here on this topic) and then update the CompilationStatus (and thus the overall Status) of the build accordingly.  If you want the build to halt immediately in addition to marking the status as Failed, you can use an Error task, and you’ll probably also want to move the logic to the AfterTestConfiguration target (to avoid running tests for each configuration before halting the build). 


Comments (7)

  1. The SRL Team Blog on Start working with eclipse and VSTS (basic needs). Tiago Pascoal on TFS: Fetching…

  2. BassettHound says:

    I am getting an error with this.  I have tried Status, BuildStatus and CompilationStatus, all yield the same result which is as (or similar) to below.

    I would like to get the build to fail, not "partiallly succeed".

    What might I be doing wrong?

    c:dBuildTypeTFSBuild.proj(335,25): error MSB4030: "Failed" is an invalid value for the "CompilationStatus" parameter of the "SetBuildProperties" task. The "CompilationStatus" parameter is of type "Microsoft.TeamFoundation.Build.Client.BuildPhaseStatus".

    Task "SetBuildProperties"

    c:dBuildTypeTFSBuild.proj(335,25): error MSB4030: "Microsoft.TeamFoundation.Build.Client.BuildStatus.Failed" is an invalid value for the "Status" parameter of the "SetBuildProperties" task. The "Status" parameter is of type "Microsoft.TeamFoundation.Build.Client.BuildStatus".

    Done executing task "SetBuildProperties" — FAILED.

    Done building target "AfterTest" in project "TFSBuild.proj" — FAILED.

    Done Building Project "c:dBuildTypeTFSBuild.proj" (EndToEndIteration target(s)) — FAILED.

    Task "SetBuildProperties"

    c:dBuildTypeTFSBuild.proj(335,25): error MSB4030: "Failed" is an invalid value for the "Status" parameter of the "SetBuildProperties" task. The "Status" parameter is of type "Microsoft.TeamFoundation.Build.Client.BuildStatus".

    Done executing task "SetBuildProperties" — FAILED.

    Done building target "AfterTest" in project "TFSBuild.proj" — FAILED.

    Done Building Project "c:dBuildTypeTFSBuild.proj" (EndToEndIteration target(s)) — FAILED.

  3. Ah yes – there was a bug in earlier versions of the SetBuildProperties task where these properties were of their real enum types (BuildPhaseStatus/BuildStatus) rather than strings.  I had assumed – wrongly – that MSBuild could manage the conversion for me.  I believe the issue should be fixed in Orcas Beta 2 – let me know if I am incorrect here.  If so, it is definitely fixed in the RTM version, which should be released here shortly…  Sorry for the confusion!

  4. BassettHound says:

    Aaron,

    We’re getting some funky behavior with this using MaxProcesses=2.

    With MaxProcesses=2, we’re getting TestSuccess=False using the code below.  All unit test have passed.

    Thoughts?

    tim

     <Target Name="AfterTest">

       <GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)">

         <Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />

       </GetBuildProperties>

       <Message Text="TestSuccess=’$(TestSuccess)’" />

  5. I suspect there is a race condition here when using multiproc MSBuild…  The TestSuccess property is just a helper property that checks whether TestStatus is BuildPhaseStatus.Succeeded – it being false doesn’t necessarily mean that tests have failed – they could just still be marked as BuildPhaseStatus.Unknown, since the logger hasn’t had a chance to update the status yet.  You could verify this by printing out TestStatus in addition to TestSuccess.

    If this is the issue, the best approach I can come up with would be to write you own custom task to check the test status which sleeps for a while if it is BuildPhaseStatus.Unknown.

  6. As promised, here are some more details on other SP1 changes for TFS Build.&#160; 3. Detect test results.

  7. Not sure if this is because I am on TFS 2008, but simply specifying the following in a <PropertyGroup> achieves the same.

    <TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>