Just because you can test it doesn’t mean you should

A wise test lead once said to me, “do as little as possible while still ensuring quality”.  He wasn’t giving me tips on how to be a slacker =)  I think what he wanted to say was there always seems to be more work than there are people so it pays to be as efficient as possible,  especially when prolonged testing can hold up shipping a product.

 

One thing I’ve seen at Microsoft is sometimes the tendency to want to test everything, which isn’t a bad thing as long as you don’t have to ship =)  Hence the title – “just because you can test it doesn’t mean you should.”  If say you have several components on a stack calling each other, it makes sense to just test at the highest layer to automatically get coverage for lower components.  Some people also call this end to end testing.  When you test each component separately, you may have the following issues:

  • Duplication of effort.  Tests of lower level components doing the same testing as end to end tests.
  • You may not be testing a component the way it will be used in the product.  For example, your tests call an API passing it X, but the caller in the product will only ever call the API with Y.
  • Holes in coverage.  Because you’re testing components in isolation, you may miss main stream customer scenarios.

 

So my recommendation?

  • Test at the highest layer whenever possible.
  • If the top layer is not yet available, then temporarily test the lower layer with the intention that they will be replaced with end to end tests.
  • Focus testing on main customer scenarios first taking a top down approach.  This maximizes coverage and at the same time focuses on scenarios customers will hit first.
  • Use code coverage tools to drill down on coverage of components.  If code is not being used by top-level components then question whether the code should be there at all.
  • Push devs to do unit testing of their components.

 

Of course there are times when component testing is called for.

  • Component code cannot be easily hit by end to end testing.  An example is negative  or error testing.
  • A component is consumed by multiple clients or it's a public API that anybody can consume.  In this case you should treat this component as the highest layer.
  • Performance tests should ideally go through the top layer and each component logs their own performance numbers.  But if you want to test things like throughput it might make sense to go through the component directly.
  • Any testing where going through the top layer would just be too slow or not possible like  security testing, stress testing, fault injection, etc.  For example, you may need to ensure an app can open 100K different files for fuzz testing but going through the UI would be painfully slow.

 

What about MVVM (Model View ViewModel) you say?  This is where you have a layer (ViewModel) that encapsulates data and functionality needed by the UI allowing you have to have a very thin UI layer, even making it easy to replace the UI layer if needed.  Some folks prefer testing against the ViewModel as opposed to going through the UI.  As somebody who has done both, I can tell you that testing against the ViewModel is much easier and UI testing can be a pain.  But in my opinion, the easy way is not worth the risk.  We saw several bugs slip through because the thin UI layer that supposedly didn’t have any bugs of course did.  Going through the ViewModel has its uses like expediting certain operations but I don’t recommend exclusively testing against it.

 

So to close, test end to end first, and component test only if you need to.  Lazy testers are efficient testers, unless they’re  trying to outsource their work on craigslist =)  Feel free to comment if you have any thoughts on this topic.

 

-Samson Tanrena