Just because you can test it doesn’t mean you should


A wise test lead once said to me, “do as little as possible while still ensuring quality”.  He wasn’t giving me tips on how to be a slacker =)  I think what he wanted to say was there always seems to be more work than there are people so it pays to be as efficient as possible,  especially when prolonged testing can hold up shipping a product.

 

One thing I’ve seen at Microsoft is sometimes the tendency to want to test everything, which isn’t a bad thing as long as you don’t have to ship =)  Hence the title – “just because you can test it doesn’t mean you should.”  If say you have several components on a stack calling each other, it makes sense to just test at the highest layer to automatically get coverage for lower components.  Some people also call this end to end testing.  When you test each component separately, you may have the following issues:

  • Duplication of effort.  Tests of lower level components doing the same testing as end to end tests.
  • You may not be testing a component the way it will be used in the product.  For example, your tests call an API passing it X, but the caller in the product will only ever call the API with Y.
  • Holes in coverage.  Because you’re testing components in isolation, you may miss main stream customer scenarios.

 

So my recommendation?

  • Test at the highest layer whenever possible.
  • If the top layer is not yet available, then temporarily test the lower layer with the intention that they will be replaced with end to end tests.
  • Focus testing on main customer scenarios first taking a top down approach.  This maximizes coverage and at the same time focuses on scenarios customers will hit first.
  • Use code coverage tools to drill down on coverage of components.  If code is not being used by top-level components then question whether the code should be there at all.
  • Push devs to do unit testing of their components.

 

Of course there are times when component testing is called for.

  • Component code cannot be easily hit by end to end testing.  An example is negative  or error testing.
  • A component is consumed by multiple clients or it’s a public API that anybody can consume.  In this case you should treat this component as the highest layer.
  • Performance tests should ideally go through the top layer and each component logs their own performance numbers.  But if you want to test things like throughput it might make sense to go through the component directly.
  • Any testing where going through the top layer would just be too slow or not possible like  security testing, stress testing, fault injection, etc.  For example, you may need to ensure an app can open 100K different files for fuzz testing but going through the UI would be painfully slow.

 

What about MVVM (Model View ViewModel) you say?  This is where you have a layer (ViewModel) that encapsulates data and functionality needed by the UI allowing you have to have a very thin UI layer, even making it easy to replace the UI layer if needed.  Some folks prefer testing against the ViewModel as opposed to going through the UI.  As somebody who has done both, I can tell you that testing against the ViewModel is much easier and UI testing can be a pain.  But in my opinion, the easy way is not worth the risk.  We saw several bugs slip through because the thin UI layer that supposedly didn’t have any bugs of course did.  Going through the ViewModel has its uses like expediting certain operations but I don’t recommend exclusively testing against it.

 

So to close, test end to end first, and component test only if you need to.  Lazy testers are efficient testers, unless they’re  trying to outsource their work on craigslist =)  Feel free to comment if you have any thoughts on this topic.

 

-Samson Tanrena


Comments (21)

  1. jader3rd says:

    I disagree with you on the MVVM part. Tests which target the ViewModel can be executed orders of magnitued faster, and be much more thorough than testing the View, plus are debuggable in ways which are impossible with View testing. Yes, there needs to be some tests which ensure that the View is wired up correctly to the ViewModel, but that's it. That's the only thing the View should be doing, is wiring itself up to the ViewModel; anything that's actually interesting to test should live in the ViewModel or lower.

  2. PK says:

    As you say Test at the highest layer whenever possible, then how do we unit test and do TDD?

  3. Matt Honeycutt says:

    While I do agree that some tests are low-value, I disagree with the "test at the highest layer possible."  You should indeed *start* testing at the highest layer possible, but if you're practicing TDD correctly, those tests will drive the creation of new lower-level components, and you'll write tests that drive the implementation of those lower-level components.  The higher-level tests should, for the most part, not explicitly exercise functionality of the lower-level components.  

    The Dependency Inversion principle tells you that your higher-level components (your policies) should not depend on the lower-level components (details), and that's for a really good reason.  If you find that your higher level tests are explicitly covering behavior of lower-level components, you're going to run into maintenance problems from high coupling.

    That said, I do believe that there's tremendous value in integration tests.  I just feel that their purpose is to guide you towards the next set of unit tests to write. 🙂

  4. Appreciate the comments folks.

    @jader3rd, per the article, I do mention that one of the ViewModel's uses is faster execution.  I just haven't had good results with exclusively testing against it.  A mix of UI/ViewModel is usually needed for sufficient coverage and expedient execution.

    @PK, I should've expanded more in the article regarding pushing devs to do unit tests. TDD/unit testing should still be done by developers.  But this article is geared towards testers who do the component/end to end testing.

    @Matt, same point as with @PK, TDD/unit testing are definitely recommended but they're typically developer tasks.  This article is for the tester who gets the components from the developer who should've already done unit testing.

    Regards,

    Samson

  5. jader3rd says:

    "the developer who should've already done unit testing."

    It's this attitude which causes a lot of bit rot. If the developers tests aren't part of the daily test runs they become bloat, overhead and dead code. The unit testing should never be done. It needs to constantly happen.

  6. @jader3rd, I wasn't implying that unit tests are only run/written once.  In fact my team runs them daily and with every checkin.  A commendable practice is to checkin unit tests at the same time as features.  Of course this doesn't preclude devs from adding more unit tests later on.

    Regards,

    Samson

  7. Jimmy says:

    As the code grows older it starts to rot. The only way we can improve is to change it. How do we know our changes won't break it and cause regressions… Because we tested it.

    ..how can we trust our tests – because we only wrote production code to make a test pass.

    When you start being selective, you lose this confidence and have holes where regressions could occur.

    If you need to take the short-term gain to ship something asap then at least be aware you have taken on long term compromise.

    Just be aware of the trade offs.

  8. tcmaster says:

    If you are talking about unit testing, I would say that the idea of "Test at the highest layer whenever possible" is quite dangerous. It might encourage developers to ignore tests during development since it's too low level. When it comes to complex system with different logic layers, it's always make sense to test each layer separately and then comes the integration tests. This at first sight will cause duplication of effort, but it's actually not, because the aspect of the tests are different. The lower level the test is, the more focused is it on the individual behaviors of the components. While the higher level tests are more context related. For example, in the low level UT, a test might just confirm that the parameter must conform to some rule; while in the high level test, it's stated like a specification, which says something like some kind of input will be rejected…

  9. Me says:

    Perhaps this might be the explanation for Microsoft's perennial QC issues…

  10. Kraiven says:

    Hmm! Actually I'm a bit disappointed that you're an MS employee. I really like MS software and make a living developing mainly in .Net. It is well known that low-level continuous unit testing, and preferably TDD, is the best way of ensuring quality, reducing the number of bugs and thus speeding up the whole release cycle. Microsoft, through MSDN, conference attendance, Visual Studio testing frameworks etc push this idea themselves, so I'm more than a little surprised at your post – I disagree with you on almost every point, you're just plain wrong and I'm wondering whether this career-limiting post was well considered.

  11. foolish-galatian says:

    Quoth Kraven:  "It is well known that low-level continuous unit testing, and preferably TDD, is the best way of ensuring quality, reducing the number of bugs and thus speeding up the whole release cycle."

    I think Windows Live Tester's point (with which I agree) is that high-level continuous integration testing is both easier and more effective at accomplishing the goals you cite.  Use TDD at the integration level rather than the unit level.

  12. foolish-galatian says:

    I would go further than Windows Live Tester and say that unit testing is a waste of time if the code is susceptible to automated testing at a higher level.  Unit testing requires mocks, which, in a complex application, are painful to implement, painful to maintain, and ineffective.  When you test code that consumes data, use real application code to build the test data.  When you test code that creates data, exercise real application code to consume the data.  Your tests should use real application code to create test users, test sessions, etc.  If an application is well-designed, it will have high-level constructs (such as result from the command pattern) to facilitate such testing.  Then you don't need to modify or re-write your existing tests every time the shape of your data changes or every time you refactor a component.  You only need to modify existing integration tests when your high-level interfaces change.

  13. Kraiven says:

    Sorry foolish-galatian but "Use TDD at the integration level rather than the unit level." and "unit testing is a waste of time if the code is susceptible to automated testing at a higher level." imply that you don't know what unit-testing or TDD are even about.

    "Unit testing requires mocks, which, in a complex application, are painful to implement, painful to maintain, and ineffective."

    Wrong on all counts: unit-testing using mocks isolates the component under test, identifies coupling and pushes good component design.

    "If an application is well-designed, it will have high-level constructs (such as result from the command pattern) to facilitate such testing."

    And where does this good design come from? Oh yes that's right ,good quality unit testing that test components in isolation and helps enforce the SOLID principles of good OO design.

    Testers are not developers, and frankly your comments and those of the OP illustrate just why. I suggest keeping to your area of expertise and leaving the development teams to theirs.

  14. foolish-galatian says:

    Kraiven, I've been a professional developer for 21 years and have had good success with the principles I cited.  BTW, Windows Live Tester's position is strong, defensible, and well-articulated.  Even if it's provacative, it should not be in any way career-limiting.

  15. Kraiven says:

    foolish, Hmm! And I have worked in companies that adopt the approach you are citing also. The requirement for a fully functional vertical layer of functionality to be available before testing can even start results in testing being pushed way to the end of the project life-cycle; ultimately squeezed, often into nothingness, due to development overrun (probably resulting from the components which have not been properly tested and which are therefore buggy as hell and extremely fragile); leading to under-tested poorly designed code with maintainability problems to make you weep. Thankfully I no longer work in that type of team.

    WLT (and you) is not being provocative, but is voicing opinions outside of their area of expertise (and wrong opinions at that). Automated integration testing is fabulous for, well, integration testing, but if this is what you are relying on then your software is doomed. Unit-testing is about quality and good design, if you are not doing this then your software is doomed. Whether voicing incorrect opinions on a public forum is career-limiting or not is not my call, but I hear MS can be pretty ruthless ;-).

  16. Pablo says:

    I would also add that the dev time burn rate is extremely high with many TDD approaches, and the cost/benefit tradeoffs tend to be quite unfavorable. In my view, the TDD paradigm is already a bit tired.

  17. I have a question for all the commenters:  How many of you are straight up developers and how many of you are actual QA/STE/SDETs?

    I ask this question because this article was written from the point of view of the latter, not the former.  Unit test development is the responsibility of the developer, everything else is the responsibility of the tester and the developer should make sure that every unit test is checked in simultaneously with the production code so that it can be executed as part of the basic verification/acceptance tests (BVT/BAT) that are executed before the testers start doing their work.

    For informational purposes, Microsoft uses a variety of technologys to determine what testing is accomplishing, including scenario testing, targeted functionality testing using both blackbox and whitebox methods and they use code coverage technology to determine how much of the actual code their tests are exercising on any given test pass and they've created some amazing internal tools to help them determine how much of any given code change has been touched by smoke testing prior to any final bits being checked into the enlistment.

    As someone who spent almost 14 years at Microsoft as a tester, what the author wrote makes a lot of sense because really, you cannot test everything, there's just never enough time to test it all, and if you did try to test it all, the code would be well beyond obsolete by the time it ships.

    Thanks!

    Mike

  18. foolish-galation says:

    I am a mature developer, and I believe the author is a mature developer.  TDD is not a panacea, nor is a software project doomed without it.

    I would also point out that I said I agree software should be tested at the highest level possible.  That doesn't mean you wait for a complete vertical slice of functionality.  (But, if a vertical slice IS available, you use it.)

    More importantly, I think the O&M phase of the project life-cycle needs to be taken into consideration.  Automated integration tests don't die when the software goes into production.  They actually become even more important in maintenance mode, when the software is updated/enhanced, often by developers who aren't familiar with it.

    In my opinion, unit tests usually only need to be run once and so are of limited value.  You only need to rerun a unit test if you modify the specific piece of code that it is testing.  A suite of automated integration tests is far more useful and should be run at every check-in.

  19. Michael says:

    Someone in our team forwarded this article in our company. This was my response:

    The author is inherently lazy 🙂 His basic argument is that Integration tests alone are enough, and there is limited need for unit testing.

    I think both unit tests (low level) and integration tests (high level) are equally important.

    Yes, integration tests are difficult to achieve, as the architecture needs to be designed such that interfaces to external systems are mocked out. Like the author says, they prove that the design meets the requirements – something unit tests alone can't do.

    Unit tests however are important for the following reasons. If we write code to achive full code coverage of each component, which is can be achieved with less coding using a tool like Pex, then this will assist in quickly highlight when the build is broken. Something integration tests may not pick up, as by their nature are completed last and do not cover all code paths. They are especially useful when you have developers working in parallel. We found unit tests especially useful on our latest project, as they on many occasions when we wrote code that broke the other person's code.

    Since we are working in an Agile manner, every new sprint will invariably cause us to refactor or change the functionality of our code. Yes, writing both high level and low level tests is more work up front, and may seem like you are duplicating effort, but in the long-run will save more time by less bug fixing during UAT. We have seen this before when we spend a long time on bug fixing then testing, then bug fixing again before a release, there is one word to descirbe this – waterfall.

  20. Long View on TDD says:

    To be able to refactor code, unit/programmer tests need to be independent of the software architecture.  To allow this, the tests need to be written at the highest level practical.  Anything is possible, but intelligent people can do trade offs and determine what is most practical.  

    To be able to refactor, to do evolutionary design, one needs to be able to change the underlying structure of the code and rely on the existing test suites to validate that nothing has changed.  If one needs to change both the code and the tests, then the cost goes up and the benefit of the tests go down.

  21. foolish-galatian says:

    Kraiven:  "unit-testing using mocks isolates the component under test"

    That's exactly the problem.  Testing a component in isolation is of very limited value.  When you use mocks and stubs, you're essentially not testing the component in a realistic way.

    Michael:  "integration tests are difficult to achieve, as the architecture needs to be designed such that interfaces to external systems are mocked out"

    Integration tests are easier to achieve than unit tests.  You don't need mocks and stubs, because you're using the code to test the code.  (And integration testing is more effective, for the same reason.)

    Unit tests only tell you when you break the piece you're working on.  (As I said before, you only need to run a unit test when you're changing the specific code it tests.)  Integration tests tell you not only when you break the piece you're working on, but also when you create unintended side effects in other areas, which, in a complex application, happens a lot and can be very difficult to detect until it's very expensive to fix.

    Michael:  "every new sprint will invariably cause us to refactor or change the functionality of our code"

    Which also forces you to rewrite your unit tests.  But integration tests do not have to be changed nearly as often.

    As for the code coverage argument, integration tests give you much more bang for the buck and can achieve the same level of coverage with many fewer lines of test code.