Don’t Live With Broken Windows

  • Article

Pragmatic
programming says “Don’t Live with Broken Windows”.  The hopefully non-copyright
infringement summary of this idea is that if a broken window in a building is
left un-repaired for too long of a time, that it leads toward more breakdowns
in the building (more broken windows, graffiti, trash, etc.) until the building
is so far gone that the owner gives up on fixing it.  The analogy to
programming is that if you don’t fix bad design or bad code early, that the
neglect leads directly to software rot.

I’ve
seen the same problem in software test.  I’ve seen projects where early in the
project, an automated test fails.  Investigation is done, and it is determined
that it’s a “test bug” that will be fixed “soon”.  Later in the project, the
test manager looks at the overall status, asks about the test, and is told that
the errors in that part of the system are “test bugs that will be fixed soon”. 
Since the tests “didn’t find any bugs”, the TM is happy.  Late in the project,
the Project Manager asks about the failures, and is told about the test bugs
that will be fixed “in the next release”.  Finally, the product ships with
excellent test results – but no one notices that there were more errors
reported as the project progressed.  Two months later, a customer reports a bug
in this area.  The support team runs the relevant tests and finds that the error
was detected by the original automation.  After a lot of screaming and yelling
(and $$$), the bug is fixed and a QFE is issued.

This is
a true story, but not necessarily an extreme.  Testers should be
perfectionists.  They should not tolerate uninvestigated errors, and should “fix”
broken tests immediately.  I often see test pass results of 98 or 99%.  While
these are excellent pass rates, if there are 100,000 tests, this pass rate
means that at least 1000 of the tests failed.  Every single failure needs to be
understood – perhaps the failures are errors in test – perhaps they are bugs,
but in corner cases that a customer would never hit.  Chances are, however, that
there are real bugs that need to be fixed.