Leveraging Automated Tests to Tighten up Your Bug Process


  • The Why: Apart from being expensive in terms of time and effort, every time a bug is handed off between Dev and Test is an opportunity to drop the ball or for confusion to creep in.
  • The How: Capture bug repro steps as an automated test that any developer can run without assistance from test.
  • The What: Reduce bug hand-offs and interdisciplinary interactions to move a bug to the “Closed” state.

The cost of finding a defect in test and resolving the issue via a bug is an expensive process in terms 'of time and effort. The ideal solution is to find bugs prior to check-in with TDD or gated check-ins. But we live in a world where that doesn’t always work and the bug is still a part of life.

Within the Package Conversion Manager 2012 project I am currently engaged in we have put a lot of effort into test automation through TDD and other forms of testing. We’ve essentially implemented the Agile development model that my colleague John Socha-Leialoha has described here. As the project matured we settled on a bug process that I have found to be a bit more efficient than what I have experienced on other more traditional or waterfall projects. It’s this process that I wanted to highlight in this blog post.

Traditional Bug Process

If you’ve worked on a traditional waterfall style software project you are probably familiar with a certain bug process. A tester finds an issue then files a bug. Invariably there is some back and forth on the repro steps with the dev. Once the bug is taken and fixed the developer resolves the bug and assigns it back to the originating tester to verify the fix. The tester executes a test or set of tests to ensure that the issues was addressed. If actually fixed it’s resolved as closed. Below is a diagram that shows a typical bug process minus the triage step.  In blue we see the steps taken by a tester and red indicates the steps taken by a developer. In the diagram below we see two transitions or hand-offs. It is these hand-offs that I want to avoid as these are very expensive and each hand off is an opportunity to drop the ball or for confusion to creep in. Keep in mind that the process shown below is a best case scenario for a waterfall style project.

image

In some cases there can be additional hand-offs. During my testing years I was all to familiar with the “no repro” trick. As a result there was a substantial amount of back and forth with a bug as I attempted to convey the complexities and nuances of the actual issue via a text field in the bug form. In the below diagram we see a less than ideal situation where the number have hand-offs have been doubled.

image

Simplified Bug Process

Within the PCM project our testers will capture each bug in an automated test that I as a developer can easily run to validate my fix. If the automated test that discovered the bug is a high level acceptance test with a great deal of leverage I will replicate the failure in a unit test that is targeted at that specific area. These tests are created using the Unit Test framework (we use it for all forms of tests) found in Visual Studio / TFS. By having all automated tests be part of the main product solution it is easy for me as a developer to run all tests or a specific test. This process hinges on the ability of the developer to easily run any automation created by a tester. The main benefit here is that once the tester has filed the bug and linked it with an automated test the tester is essentially done with this bug. The automated test serves as clearly defined and very specific repro steps. Gone are the days of a bug being resolved as “No Repro”. The diagram below shows the simplified process that we are using.

image

As you can see we have effectively reduced the number of hand-offs to one greatly reducing the overhead and risk associated with traditional processes shown above. This has had the effect of creating a lighter weight and crisper bug process that keeps the tester focused on expanding test coverage and product quality and not on my dirty laundry.

Team Dynamics & Culture

It is technically feasible that any team could use this process, however it is likely that any team that is in a position to be successful with this approach has a certain culture with regard to test. Such a team will probably maintain a culture where the test team is in a position to place the responsibility of running test automation to verify the fix with the development team. Not every team works to put test in the driver seat on such issues which can server as a blocker to implementing the streamlined process described above.


Comments (1)
  1. Chris Koniges says:

    Very nice recap of the process.

Comments are closed.

Skip to main content