Repro This!

Pop quiz: A tester finds and logs a bug. Some days (weeks (months)) later a developer picks up the bug and runs through it. Lo and behold, the bug does not occur. What should the developer do with this bug?

Most common answer: Resolve the bug as “No Repro” and send it back to the tester. Bzzt! Wrong! An electric shock to that part of your body you care most about. The bug jolly well does repro or I wouldn’t have logged it. Use the build I was using when I found the bug and I guarantee it will reproduce.

Less common answer: Resolve the bug as “Fixed” and send it back to the tester. The bug used to happen but now it doesn’t, so I guess this answer is technically correct. You still get a cattle prod between your shoulders, though. The bug may in fact be fixed, but it’s just as likely to still exist and merely be hiding behind some other bug.

Extremely rare answer: Install the build on which the bug was reported and verify it does indeed repro on that build. Next, binary search the daily builds (you do build your full product at least once each day, right?) until you find the build at which it stopped repro’ing. Binary search through the checkins for that build (a simple process if you run every checkin through a full build) until you find the checkin that “fixed” the bug. Figure out why that set of changes caused the bug to no longer repro. This is the first point at which you have enough information to validly resolve the bug as Fixed. Of course, it’s just as likely that the bug is still lurking somewhere. If you find this to be the case, your tester will be more than happy to help you find it again.

Resolving a bug “No Repro” just because it doesn’t repro on *your* machine is a sure way to make your tester grumpy.

*** Comments, questions, feedback? Want a fun job on a great team? Send two coding samples and an explanation of why you chose them, and of course your resume, to me at michhu at microsoft dot com. I need a tester and my team needs program managers. Great coding skills required for all positions.

Comments (10)

  1. If testers are only filing bugs that are reproducible, then why use a bug system that allows a resolution of "not reproducible" for those bugs? It’s pretty confusing (as a user) to have a system that offers an option you should never pick. Even if you document that this should never be done, you’re going to have people that don’t read the documentation and you’re going to have new developers come in and use the system and not know about this.

  2. David Fries says:

    Michael, does the burden of proof fall on the tester, or on the developer? Shouldn’t we be encouraging testers to go through these steps when they get a No Repro bug back, rather than simply closing the bug?

    Nicholas, there is such a thing as a No Repro bug. Sometimes you install a bad build, or otherwise get your machine into a bad state that no customer should ever be in. In this case, I’d say it’s legitimate to say that the bug is No Repro, since it doesn’t repro on a "clean" machine.

  3. The Braidy Tester says:

    Definitely the tester should be going through the steps when they get a bug back. Not just for a No Repro, either. Whatever the resolution the tester had better verify that the bug really is fixed or can’t be reproduced or should be designed that way ("By Design" is another bane of testers’ existence) before closing it out. A tester who doesn’t isn’t doing their job.

    I think the burden of proof is shared between the tester and the developer. The tester needs to make an effort to ensure the bug really does reproduce, minimize the repro steps, and poke around the edges of the bug and determine whether the bug really is the bug or if it’s just a symptom of some larger issue. The developer needs to make an effort to understand the bug report, actually go through the repro steps before resolving it (even if the developer fixes the bug they had better understand it before writing code that makes it (appear to) go away), and poke around the edges of the bug to determine whether the bug really is the bug or if it’s just a symptom of some larger issue.

    Yes, both the tester and the developer need to poke around the bug’s edges. Each has a different perspective on the application. User-visible functionality is not always indicative of under-the-skin plumbing. Two bugs that appear to be related may not be, and vice versa.

  4. David- For a bad build, the bug may be invalid or against your build system or won’t fixed because it doesn’t affect customers. And for that bad configuration you might choose to won’t fix because you don’t intend to support that configuration. Hopefully though those bugs are still reproducible if you get that same bad build or put together that same weird configuration. I don’t think you can just dismiss a bug as not reproducible if you don’t try it under the same circumstances as the tester.

  5. Thank you!!!! Now MS needs to list to this for all of the bugs that I report… it’s scary how many "no repro"s I get back….

  6. AndrewSeven says:

    Is "No Repro" a black hole, a subset of resolved?

    Maybe a not currently reproduceable status is needed, something that sends it back to the tester to see if he/she can still reproduce it.

    It is probably more efficient to get together as a group (I don’t want to say meeting) and look at the bugs that cannot be reproduced.

    Just as a bug may be a side effect, it may be fixed when the cause of the side effect is fixed.

  7. Stesmo says:

    I had a dev who’d just bulk reassign all of his open bugs back to the tester every 2-3 weeks to see if they repro.

    There are sometimes very good reasons why a bug will be ignored by a developer for weeks or months. However, in my experience, the developer that waits for weeks/months to repro a bug are actually hoping, praying it will be Not Repro if they just wait long enough.

  8. I.M. Testy says:

    If a bug goes for weeks or even months before being addressed by a developer and the developer resolves the bug back to testing, then you deserve what you get.

    The tester is responsible for driving the issue to resolution. It’s not simply enough to throw your report into the coffer.

    I can’t think of a single good reason why a bug will be ignored for months. The longer a reported defect goes unaddressed, the greater the probability of that bug being masked.