How good was that build?

  • Article

Software teams need to continually check the current condition of their bits in order to ensure they are in at least a testable if not yet shippable state. Nearly every day, one or more people on the team seek answers to a few questions that are easily asked.  “How good is that build?”  “What was the last good build?” “What changes were made and what bugs were fixed in that build?”

Easily asked, but not necessarily easily answered. In this article I hope to help you quickly understand the quality indicators available in Team Foundation Build and how you can use them to answer your own questions about the quality of your completed builds. I also aim to begin filling a subtle gap in the recently published details about the AssociateChangesetsAndWorkItems activity.

Three indicators of the quality of a completed build

Your Team Foundation Build system provides three primary indicators about how good a completed build is:

  • Build status (automatic)
  • “Build goodness” (automatic)
  • Build quality (manual)

How build status and “build goodness” are evaluated

Before we explore all three indicators in detail, I think it will help for you to first see how the system generally evaluates the two automatic indicators:

These outcomes in the build...   Result in these quality indicators...
Result of code compilation Evaluated by the MSBuild activity Recorded in IBuildDetail CompilationStatus property Result of tests Evaluated by the MSTest activity Recorded in IBuildDetail TestStatus property Result of other steps in the build process Did all post-compilation and post-test steps in the Default Template (DefaultTemplate.xaml) succeed?   Build status Build goodness
Succeeded Succeeded Yes   Suceeded Succeeded Good
Succeeded Succeeded No   Partially Succeeded Partially Succeeded Good
Succeeded Unknown or Failed Yes or No   Partially Succeeded Partially Succeeded Bad
Unknown or Failed Succeeded, Unknown, or Failed Yes or No   Failed Failed Bad

 

There are a few exceptions that can cause the build to be evaluated differently than is shown in the above table.  They are covered in the details below.

Build Status

Build status is the most prominent indicator of build quality in your build system.  You see it in both the build explorer and the build results window.

How build status is determined

Build status is evaluated in the Default Template (DefaultTemplate.xaml). In a build based on DefaultTemplate.xaml, build status is determined as follows:

  • Suceeded Succeeded
    • The MSBuild activity did not record a value of Failed in the IBuildDetail CompilationStatus property.
    • The MSTest activity did not record a value of Failed in the IBuildDetail TestStatus property.
    • Various other activities did not fail. These activities include, but may not be restricted to gathering test impact data (GetImpactedTests activity) and publishing symbol data (PublishSymbols activity).
  • Partially Succeeded Partially Succeeded
    • The MSBuild activity did not record a value of Failed in the IBuildDetail CompilationStatus property.
    • One of the following occurred:
      • The MSTest activity recorded a value of Failed in the IBuildDetail TestStatus property and Fail Build on Test Failure was not selected when Automated Tests were specified in the build.
      • One of various other activities failed. These activities include, but may not be restricted to gathering test impact data (GetImpactedTests activity) and publishing symbol data (PublishSymbols activity).
  • Failed Failed
    • One of the following occurred:
      • The MSBuild activity recorded a value of Failed in the IBuildDetail CompilationStatus property.
      • The MSTest activity recorded a value of Failed in the IBuildDetail TestStatus property and Fail Build on Test Failure was selected when Automated Tests were specified in the the build.

If you develop a custom build process template, you can use the SetBuildProperties activity to force the build status to be whatever value you want based on your own workflow logic. If you use this approach, then the logic described above is disabled for any builds based on your custom build process template.

The build status for each build is stored in the data warehouse via the IBuildDetail Status property.

The effect of build status

Build status is displayed in the build results window and in Build Explorer. This indicator is used by the system to determine which retention policy to apply to a completed build.  For more information, see Create a Basic Build Definition.

“Build goodness” (or “last good build”)

I actually made up the phrase, “build goodness” because nowhere in the product does this phrase appear.  But as you will see, the concept behind build goodness is important.

How build goodness is determined

Build goodness is evaluated by Team Foundation Server internal logic, and thus cannot be customized. The rule is simple.  The build is “good” if both the IBuildDetail CompilationStatus property and the IBuildDetail TestStatus property are set to “Succeeded”. Otherwise, the build is “bad”.

Unlike build status, there is no attribute of a completed build in which build goodness is indicated.  Rather, a reference to last good build is recorded in the  IBuildDefinition LastGoodBuildLabel and  LastGoodBuildUri properties.

The effect of build goodness on the AssociateChangesetsAndWorkItems activity

Build goodness has one very important effect, which is that it is used by the AssociateChangesetsAndWorkItems activity to determine which completed builds are associated with recent changesets and their associated work items. Specifically, the AssociateChangesetsAndWorkItems activity:

  1. Attempts to locate the last good completed build from the same build definition. If the last good build is located, it continues to the next step.  Otherwise, it stops here.
  2. Compiles a list of all the changesets applied to items mapped in the workspace defined in the build definition.
  3. Associates the changesets, along with any work items to which they are linked, with the completed build.

Build quality

Build quality is simply a string that someone on your team uses to characterize the quality of the completed build. 

How build quality is determined

You manually set this value, as explained in Rate the Quality of a Completed Build. You can also customize the list of build quality values, as explained in Add or Remove Build Quality Values.

The effect of build quality

Build quality is displayed at the top of the build results window of a completed build.  It is also displayed and can be used to filter the view in Build Explorer.

Closing

This post is the result of one of several key concept article ideas that crystalized in my mind as I researched and wrote Team Foundation Build Activities. I also want to express my thanks to Ed Blankenship, whose comment inspired me to write this blog post.  He promptly pointed out a subtle gap in the section on AssociateChangesetsAndWorkItems, a gap that I hope this article partly fills.

I aim to blog additional experimental drafts such as this one, and then eventually publish the material on MSDN. Is this kind of information helpful?  Is the above explanation of how the AssociateChangesetsAndWorkItems activity works sufficient?  Or would you like to see the behavior laid out in more detail with examples and graphics? As always, I welcome your feedback.

Andy