Hack the Build: Use the Whole Test-First Whale
Jomo Fisher—
Try lots of things to improve quality. Keep doing what works, stop doing what doesn’t.
Writing code with Test-Driven-Development (TDD) takes about 30% longer than writing code without it. This was my experience on MSBuild. Still, as I noted here and here, I’m a fan of TDD. Code quality tip #2 explains why…
Tip# 2: Take Maximum Advantage of Developer-Written Tests
Subsistence hunters use every part of their kill. This makes sense because it’s hard to catch a whale. If you expend that kind of effort, you need to maximize your return. If you don’t then you’ll freeze all winter while your neighbor down the coast is toasty warm next to their blubber-oil stove.
I don’t think of TDD as just a goal in itself. I think of it as a platform for a set of code-quality drivers. What follows is a set of ideas for how to get the maximum use out of your developer-written tests. We used all of these, with success, on MSBuild:
Everyone Shares the Tests—We checked the test source code into source control alongside the shipping production code. Everyone worked with and contributed to the same set of tests. As a developer, this allows you to create features that are virtually impossible for your well-intentioned team-mates to break.
You Can’t Check in until All the Tests Pass—In addition to using the GUI test driver for development, we have a single batch file that reports a failure if any tests are failing. Before you can check it, you have to pass the whole battery of tests. This keeps the code in source control pristine.
Drive Up Code Coverage—With ideal TDD you should be able to hit every line of code in your product with unittests. In practice, this turned out to be more like ~87% of the code for MSBuild. You can judge the relative health of your unittests (and your TDD process in general) by looking at code covered by the unittests. Try this: set a break point on every line in one source file, run your unittests under the debugger. Every time you hit a break point, remove it. When you’re done, the remaining break points indicate unittests that you’re missing.
Every Bug Has a Test—When fixing bugs, first create a test that fails. Then fix the bug and see the test pass. Bug regressions (bugs which return after they’ve supposedly been fixed) are very expensive. If you use a bug tracking system that assigns a number to every bug, consider sticking this number in the method name:
[Test]
public void Regress99999_EjectedCDInjuresUser() {...}
This way, you can always track the test back to the original bug that it was supposed to address.
Refactor Mercilessly—People sometimes use a word called “churn”. It’s a derogatory term for changing a lot of code at once. Thing is, changing a lot of code is often the right answer. Avoiding a needed refactoring can cost you more down the road. Once you have a good corpus of unittests, take advantage and refactor whenever it makes sense. If you’re a manager or team-lead, let your team know it’s OK to refactor as much as they need to.
Mutation Testing—Think of unittests as an immune system for your code. If there’s a bug in your code, then a test should fail. The idea behind mutation testing is to intentionally inject a bug in your code and then run unittests to make sure at least one fails. There are plenty of ways to create bugs in your code. For example, if you have this:
if (condition) {...}
Then replace it with,
if (condition && false) {...}
Did a test fail? If not, add a unittest that does fail. If you can’t, the code is probably redundant and can be removed.
We automated the process of systematically injecting bugs into the code and running unittests. Our solution used a hand-written C# parser to generate the mutation set. I haven’t tried it, but there’s a free tool called Nester which is supposed to automate this for you for C# files.
This posting is provided "AS IS" with no warranties, and confers no rights.