Unit test speed.

  • Article

If you're thinking about doing Test-Driven Development, you may ask yourself how long it should take for your unit tests to run.  Faster is always better.

The time it takes to run tests has various effects.  When the unit test suite slows down:

  • You become reluctant to add more tests, because it might get even worse.
  • You put off running the tests, making the Red-Green-Refactor cycle longer, therefore:
  • You bite off bigger chunks (by making more complex tests) in your TDD cycle.  The problem becomes harder, and you spend more time making & recovering from mistakes.
  • Your unit test coverage worsens, allowing more bugs to lurk in your code
  • You confidence decreases, giving you more gray hair, so:
  • You hesitate to refactor, for fear of breaking code.  Your code quality worsens and you can’t go fast.
  • With worse code structure, it’s tougher to test things in isolation.  It’s tougher to target specific behaviors you want to write.  Tests traverse more code, so they are slower.  The cycle reinforces itself
  • Oh, no, it’s DOOMSDAY!

There’s a yahoo group on TDD, frequented by PhlIp, Ron, and others from the Agile community.  One bit of guidance that I’ve seen there is that unit tests should run in less than a minute.  When they start to get slow, and you find the above bullets are happening, it’s time to refactor your code, refactor your tests, and do perf. improvements of both.

One thing to keep in mind is scale of project: at Microsoft our teams are larger, our ship cycles are longer, our plans are grander.  I can’t imagine the full unit test suite for all of Visual Studio running in less than a minute.  But I could see it happening per-team, or even smaller.  I figure that when code is decoupled & has good unit tests, you know which tests you need to run for which class.

Running the full set of unit tests + integration tests before checkin is a good idea, and part of XP.  Keeping the whole checkin verification process under 15 minutes is good, too.

I don’t work in a situation like the one I’ve described, but we’re getting better.