A few days ago, while working with one of our unit test suites, I discovered that one of my test utility classes had a bug. The bug didn’t cause any tests to fail, but it caused a few tests to become false negatives – they didn’t fail, but they actually didn’t test what they were supposed to test either. Embarrassed, I corrected the test utility class (it was a very simple fix), all unit tests still succeeded, and now the tests that previously didn’t test the correct behavior actually do that.
At that point, I happily checked in my changes and went along with my life.
Later, it occurred to me that I couldn’t be certain that the fix I had implemented would stay that way. What if someone else comes along and changes the test utility class back to the way it was before? I know from experience that it isn’t going to break any tests, although some tests do rely on specific behavior of the test utility class.
When I first wrote the test utility class, I thought it was so simple that it didn’t need testing in itself. When I fixed it, it was still simple, but if I could make the mistake once, I could conceivably make it again, so I ought to protect myself against this sort of regression. As Gerard Meszaros writes in xUnit Test Patterns, you may definitely have a need for writing Test Utility Tests, and that was obviously the correct thing to do here, so I wrote a test that verifies that the test utility class behaves as expected, and I can now feel much more confident that my ‘real’ tests will not produce false negatives.
So, following Roy’s pattern language, anti-pattern #2:
Forces: The bug is in your test code, so you just want to fix the test and get on with the ‘real’ feature you are working on.
You don’t think test code needs testing, even when you just discovered a bug there. Instead of writing a test that reproduces (or exposes) the test bug, you just fix the test code and hope it’s going to stay fixed forever.
How to avoid: Apply the same mindset to your test code as you do to your production code: If you discover a bug in the test code, write one or more unit tests that reproduces the bug, and then you can fix it. The ensures that no test regressions will occur duing a later test refactoring.
It would be interesting to read about other unit testing anti-patterns, so I’d hereby like to start a little relay race and pass the baton to Ayende, whom I much admire, in the vain hope that he will divert us with unit-testing anti-pattern #3, and subsequently pass on the baton to someone else…