Managing Orcas Beta exit criteria by dogfooding TFS

  • Article

I've written a few times about our usage of TFS while we develop newer versions of Visual Studio and TFS.  It's one of my favorite topics so I'm excited about a recent update that we've made to our system to help us manage the end game of our beta release of Orcas.  Part of getting ready to do a major release of our flagship product is ensuring that many different criteria are met by the various teams that contribute to the overall release.  Since much of our execution and accountability is with the product units within DevDiv (as I believe it should be) we manage many of these "exit criteria" at the product unit level and then roll up status into a divisional view. This is something that TFS is well suited for. 

To start with, we created a new work item type to track each exit criteria for each product unit.  We currently have 28 exit criteria categories for beta 1 (things like performance results, test pass rates, and stress pass rates).  To learn more about some of these read Brian's recent series on Managing Quality (Build, Automation, Performance, Stress, Dr. Watson).  Each of these criteria are measured by each of the 24 product teams delivering some components to the release.  These work items are assigned to someone on the product team who is responsible for managing and reporting back on these criteria. 

The interesing fields of this work item type are as follows:

  • Title - common name of the exit criteria (one of 28)
  • Product unit - one of 24 teams that participate in shipping Orcas
  • Assigned to - who is the program manager or tester responsible for managing the specific exit criteria?
  • Due date - when will this be signed off on?
  • Modified criteria - has the product team modified the divisional criteria for their team?
  • Criteria modifications - if so, what changes were made to the critera?
  • Sign off status - a drop down with the following choices
    • Meets Expectations: a team has determined that they have met the requirements (or the modified criteria) for this exit criteria.
    • On track (Doesn’t meet expectations):   although a team has not yet met the requirements, they believe that they are on track for meeting the requirements by the due date specified.
    • At risk (Doesn’t meet expectations): a team believes that they are not on track for meeting the requirements by the due date specified
    • Blocked: a team has not met the requirements for the exit criteria and are unable to make progress due to factors outside of their control
    • No data: this is the default state of the work item and indicates that no update has been made
    • N/A: this exit criteria does not apply to this portion of the product

From there, we wrote a report which translates the states for each team and criteria to show us an all up view.  Since I represent only the Team System teams, I cut out the results for other teams:

Each of these squares in the grid can be clicked on to drilldownto the work item details.  Since this is build on TFS, we have information in the work item about history and the links necessary to track and communicate our plans across our various teams. I utilize this regularly so that I can understand the hotspots as we close down our beta release.  We will use this information during our product unit reviews as we plan for future milestones and the eventual RTM release of Orcas.  Knowing early on where our hotspots are will help us ensure that we have a plan to get green across the board by the time we ship.