Metrics are very important to the success of an engineering team. They help with data-driven decisions. Today let’s focus on QA metrics, although having metrics for all the important indicators we use to ship products is essential to any engineering team. For QA metrics, some of the most important ones are around # of test cases, % automated, % run, % passing, and % code coverage. But recently there’s been a big push for traceability. In Visual Studio’s TFS, we have work items for many different artifacts in the product lifecycle like requirements, test cases, and bugs. So my team started using metrics to track items like how many test cases do we have per requirement, how many bugs have we found per test case, etc. Now obviously you just can’t have these individual items in TFS, you actually need to make sure to link them together. Sometimes making sure everyone in the team is consistent in how they enter their data (like links between test cases and bugs) is the hardest part of getting an accurate metric. So when the first report rolled out and one team showed that they had 97% of their test cases associated with bugs, we commended them for their efforts. That means they went through all their test cases, even legacy ones, and associated them to bugs found. This was a big accomplishment, or was it? Do you see a problem with this?
At first, we honestly didn’t. But it only took a few days and some different perspectives by some incredibly smart people on my team to ask the question, is having 97% of our test cases associated to bugs a good thing really? What does that metric really mean? It means we find a ton of bugs while doing our test passes. Again, is that a good thing? Well, not really. Our job as quality assurance engineers is not to find all the bugs but to help stop them from occurring to begin with. Our test cases shouldn’t be written to prove there are bugs in our code. They should be written to prove there are no bugs in our code. So having bugs associated with 97% of our test cases in one of our projects means there are way too many bugs in our code and we need to push quality upstream.
Granted, if we were to say that 97% of our test cases were reviewed to see if there were bugs associated with them, then we would be measuring amount of work and progress towards overall traceability. This is what we wanted from this metric, but that’s not what it tells us. It tells us that we don’t do enough testing upfront in the product cycle and that we are using test cases to catch bugs and not to verify that none exist. Guess what? We have a lot more work to do here. And fortunately, by having metrics and critiquing them appropriately, we have exposed places where we need to improve as a QA organization and as a product team.
What metrics does your team collect? Are they the right ones? Watch out!…and think twice about what behavior you are trying to drive!