Testing Ideals

  • Article

I have been thinking about what an ideal software development process might look like:

  • The product team would understand exactly what their customers want to do and how their product could help their customers do so.
  • Developers would understand exactly how to translate that vision into code.
  • Developers would write code which does exactly what they want it to, code which has zero defects.
  • Installing the application would work every time with zero problems.
  • Once it is installed the application would update itself automatically and transparently; these updates would install with zero problems every time.
  • The application would continually self-test itself and automatically report any failures to the product team.
  • The application would guarantee zero loss of data or productivity due to failures on its part.
  • Customers would have a trivial process for reporting issues and providing suggestions to the product team.
  • Customers would allow the application to report failures to the product team, and report every issue they find and suggestion they have to the product team, and have zero qualms about doing so.
  • The product team would have either a fix or a "Nope and here's why" back to the customer within some short timeframe after receiving the report.

Note that testing does not appear in this list. Testing is necessary today at least because:

  • The product team does not understand exactly what their customers want to do.
  • Developers do not understand exactly how to translate that vision into code.
  • Developers do not write code which does exactly what they want it to.
  • Developers do not write code with zero defects.
  • Installing the application does not work every time.
  • Applications do not update themselves transparently. (Yes, many applications update themselves. I haven't yet seen one which does so transparently, so that I do not realize it has done so.)
  • Updates do not install correctly every time.
  • Applications do not continually self-test themselves.
  • Applications do not guarantee zero loss of data due to failures on their part.
  • Applications do not guarantee zero loss of productivity due to failures on their part.
  • Customers do not have a trivial process for reporting issues and providing suggestions to the product team.
  • Many customers do not allow the application to report failures to the product team.
  • Most customers do not report every issue they find and suggestion they have to the product team.
  • Most customers do not have zero qualms about submitting crash reports (in part because the reports may contain personal and confidential information (like the full contents of that steamy love letter to your Significant Other)).
  • Product teams do not have a fix back to the customer within a short timeframe after receiving a problem report.

I do not think this has to be so.

One way to fully test a software application, and by implication find every bug in that application, and so eventually have bug-free code, would be to build the tests into the application. (Thanks Roger for suggesting this.) This is the intent of Design By Contract: Pre- and postconditions are defined for every method, and every method checks its preconditions before it does anything else and checks its postconditions after it has done everything else. This could be extended outside individual methods by creating daemons which periodically verify the consistency and correctness of the system as a whole.

While this is a start it does not make the application fully tested. Incorrect functionality would not be found, as the pre- and postconditions and consistency checks would be verifying only what the developer thinks should happen, which will not necessarily match what the other members of the feature team - or the customer - thinks should happen. Nor would this catch performance issues, or security holes, or usability problems.

Many of these however could be found by design and code reviews. Static analysis tools like lint and FxCop and PreFast can find other types of errors. Dedicated application of root cause analysis, where the product team analyzes every issue and takes steps to eliminate its cause, could largely prevent these and other defects from ever recurring.

Even with all of that testing still seems necessary. Today we test our products in order to gather information our management uses to make business decisions about our product, such as whether to continue working on it or to ship it in its current state. One reason we do that is because undesirable business consequences tend to result when customers attempt to use software whose level of unfinishedness goes beyond their tolerance levels. One cause of this customer satisfaction is the loss of productivity they experience when they have to take time to redo work which was lost when the software crashes, when they have to take time to report problems, when they have to take time to install patches.

I believe this is achievable. For example, I have not yet lost data from Microsoft OneNote despite it crashing on occasion. Web controls tend to update themselves fairly transparently. Microsoft's Online Crash Analysis makes submitting crash data simple (although not transparent).

Some of the pieces are here at least in part. The others seem eminently feasible. What are you doing to help this ideal world come into existence?

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.