I will share a straightforward method to help development teams understand and communicate their ability to address the bugs in their areas. Not all product teams in Microsoft use a method such as the one I will describe below, but many do and there is much to be learned from this exercise.
The advantage benefits of this method:
- A simple method for multiple feature teams to have a product-wide consistent conversation about their bugs
- Enables one to see the impact of other activities on bug count
- When historical data is available, allows one to estimate how future milestones will affect the bug count
[UPDATE 2010/12/18 – A draft version of the text was originally posted by mistake. The version below has been revised to bring it closer to the intended final version]
First, we need to pick a unit of time for this method. The most convenient is weeks instead of individual days. There are two benefits. First, there are fewer values to plot. Second, we don’t have to account for handling weekends in the calculations.
The number of bugs that are to be fixed at the start of a given week (Monday) is that week’s BUG COUNT. To compute the bug count for week n is simple. Take the bug count for week n-1 and add the net effect of all activities that affected the bug count.
How we compute the “net effect” (we’ll get to this later) is the strength behind the analysis.
A STATIC SYSTEM
We start with the simplest case. We have start with a bug count of 87 on week 0. Everyone is on vacation. Nobody is entering new bugs. Nobody is fixing them.
The chart below is the current model. It is nothing more than the BUG COUNT for each week.
As a convention, cells with red text with a gray background indicate calculated values. Cells not marked in this way are considered input that we can modify,
The table – its form will change throughout this post – is the numeric model of the bug count.
To assist in visualizing the predicted bug count, I will use a simple column chart with the table. Below, you see it matches the table – the bug count is static and does not change week-to-week.
Every week people enter new bugs. This is the INCOMING RATE.
For now I will simply assert that 40 new bugs are filed every week. I include it the model by adding a row called incoming. As you can see below, the bug count is growing linearly – 40bugs/week.
BUT HOW MANY ARE KEPT?
When a feature team looks at the new bugs filed, not every bug is kept. Some of them are dismissed immediately.
The reasons can be any of the following:
- The bug is a duplicate of an existing bug
- The underlying issue was recently fixed in a build later than than the bug filer is using
- The person who filed the bug has a misunderstanding about the issue or performed incorrect steps – the bug report itself is invalid
- The bug has nothing to do with the code – it is not actionable by a developer. So, We do not count it a “code defect”. An example of bugs we might exclude are those relating to out-of-date documentation.
- The bug describes a behavior the filer of the bug considers erroneous but that the feature team identifies as an intended behavior.
It’s not uncommon that 50% of incoming bugs are discarded for such reasons. This ratio of kept versus the incoming is the KEEP RATIO. For example, if we keep half of all bugs filed then then ratio is 0.5 and so we apply it to the weekly incoming number.
Instead of growing 40bugs/week, the BUG COUNT grows by 20bugs/week. The situation is still bad – it continues to trend up – but at least the trend is not as steep.
WHAT IS THE CAPACITY TO FIX?
Now we account for the bugs fixing activity of the developers.
Although the number might seem low, we start by assuming that a developer can fix 1 bug each day. Thus the rate is 5 bugs/week for 1 developer. With a team of 6 developers, the capacity to fix bugs is 6 * 5 = 30 bugs/week. This is the FIX CAPACITY.
Now we include the FIX CAPACITY to the model. I’ve added a net change row to make the change per week obvious.
The model reveals the trend is now shrinking – down by 10bugs/day. This is good, the trend is desirable. Still, the team will not hit zero bugs by week 6.
NOW WE MODEL
To MODEL the bug count is to take into account the activities of the team and see how it affects the bug count. In practical terms by considering the behavior of the team – we will directly modify the weekly values for incoming, keep ratio, and fix capacity and then see how those modifications affect estimatin.
EXAMPLE: THE “TEST PASS” ACTIVITY
Some teams may schedule a “test pass”. In a test pass, the test team will formal run through every set of test cases and scenarios they can think of.
Suppose that the test pass occurs in week 4. Based on some historical data (which would be best) or some guesswork (which is acceptable to start) we can state that the incoming rate will be 2x the normal one and we will keep 75% of the bugs in that week. The reason for the higher KEEP RATIO is that we believe that the test team is catching more real issues with the product via their concerted effort during this week.
Accounting for the test pass, the team is further from the goal because of the influx of bugs from week 4.
EXAMPLE: RAMPING DOWN DEV & TEST ACTIVITIES
The focus of dev and test in a phase shift. For example, after the test pass the test team may not be looking as actively for bugs – they may be moving on to planning activities for the next phase of the project. We adjust for that effect by reducing the incoming rate in week 5.
Likewise the dev team may be moving from fixing bugs to writing design documents for the next milestone so we need to account for that as well – in this case we’ll say that a single developer is moving from bug fixing to planning. Thus the FIX CAPACITY goes from 30 to 25.
We are still at 57 by week six!
EXAMPLE: ASKING FOR HELP
The model reveals that the team is not going to hit zero bugs by week 6.
At this point the team requests for developers to be temporarily reallocated from other parts of the product to this one. The general manger will rightly ask how much of a difference will this make. And because we have a flexible way of modeling the FIX CAPACITY – we can show the effect and clearly explain the computation.
Suppose the team request for:
two additional developers for w0 and w1
only 1 additional developer for w2, w3, and w4.
It would affect the numbers like this …
Look at week 6. Week 6 starts with 22 bugs, at the bottom the net change will be –20 bugs. This means week 7 begins with 2 bugs, So now we have a reasonable expectation that with additional help we will be done on early in week 7.
BUG MODEL EXCEL WORKBOOK
You can find the XLSX file used in the calculations above on my skydrive:
- Where do the numbers come from? You’ve got to look back to the history of the project – there will always some guesswork and estimation involved. As you collect data week by week you can refine your models as needed.
- This a simple model. As presented I only wanted to give you a flavor of how to start your own analysis and modeling.
- It isn’t always necessary. This as an analytic tool to help teams think about their project. It may be too much overhead for some teams. But having overseen multiple teams for many years at Microsoft, I think it can bring some useful rigor to project execution.