How'd Ya Miss That?

Have you ever missed an obvious bug? One of those sit-up-and-smack-you-in-the-face problems that you nevertheless still somehow managed to not see?

I remember one where my team was adding product licensing, so that you had to have a valid registration key in order to run the application. The developer's private build looked good and the changes were checked in. (At the end of the day, of course. Just before the dev went on vacation, of course.)

The next morning we discovered that licensing plain did not work for users without administrative rights. It didn't work on the almost-released-OS we were testing either. Oops.

I remember another time where my team was testing code samples for a product's software development kit (SDK). Again, everything looked good - each sample compiled clean, with no errors or warnings. As soon as we released to customers, however, we learned that several of the samples crashed on launch due to null references and uninitialized data. Oops again.

How did we miss such obvious tests? I mean, who wouldn't think to actually*run* the samples they had just compiled? Wasn't that an obvious thing to do?

Who wouldn't think to test licensing under the new OS that was such a big deal? Wasn't that an obvious test?

Who wouldn't have caught the bug you missed this morning? It was completely obvious, right? <g/>

The "obvious"-ness of something is completely dependent on context. Licensing was missed in part because to that point we hadn't found any differences between the released OS and the not-quite released OS, and we unconsciously expected that to continue to hold true. As for the uncaught null reference, I can tell you that there are plenty of developers for whom the need to run their code once it compiles is not at all obvious! And I'm sure you have a very good reason for missing that nasty bug that escaped you this morning.

Problems often seem more obvious in retrospect. This is one reason I run FxCop and use checklists like my Did I Remember To list (now in the process of being expanded and annotated on my DDJ blog as You Are Not Done Yet posts). Reviews of all sorts (spec reviews, design reviews, code reviews, test reviews, ...) are useful in part because each participant comes from a different context and so sees different issues. Our customers come from yet different contexts and so find those "obvious" bugs we still manage to miss.

The next time you miss an obvious bug, don't beat yourself up about it. Take the opportunity to start your own personal list of things you commonly miss, your own personal Am I Done Yet checklist. Add to it each time you miss something else. It may never stop growing - mine hasn't, anyway - but that simply means you haven't stopped growing either!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.