World Class Testing

  • Article

I've highlighted my take-aways from the "World Class Testing" section of Managing the Design Factory by Donald G. Reinertsen.  It's an insightful book whether you're optimizing your product line or designing a new product.  It's packed with empirical punch, counter-intuition, and practical techniques for re-shaping your engineering results.

Viewing Testing as an Asset

  • View testing as an asset not a problem.  If you don't, you'll likey have an underresourced and undermanaged test department.  
  • Testing is typically 30 to 60 percent of the overall dev cycle - treat it as a major design activity.
  • The mismatch between theory and real-world is often unexpected - test results have inherently high information content

Ways to Optimize Testing

  • Distinguish between design testing and manufacturing testing.  Manufacturing testing is done to identify mistakes in the manufacturing process. 
  • Design testing is done to generate information about the design.
  • Test at the level in the product structure where you can find defects most efficiently.
  • Identify what you're optimizing in your testing - expense, unit cost of impact, performance, or time. 
  • Use economic analysis to help choose what to optimize.
  • To reduce cost, eliminate duplicate testing, test at the most efficient subsystem level, automate testing processes, and avoid overtesting the product.
  • Avoid overtesting, by branching test plans.  If the product fails certain tests, follow different paths.  Don't blindly run the tests where the rest of the tests no longer have significance.
  • To reduce the unit cost impact of testing, you can eliminate product features that exist only to make the product easier to test, and use testing as a tool to fine tune product costs.  Sometimes a design improved through testing, is cheapter than a do-it-right-the-first-time design.
  • To optimize product performance, you can either increase your test coverage or enhance the validity of your tests.  When you increase testing coverage, it's more practical to test probable applied use vs. all the possible permutations (which is usually statiscially impossible or inefficient).  To improve your test validity, generate the same types of failures in your labs, that you see in the field.
  • To decrease testing time, increase the amount of parallel testing, use relability prediction, decrease testing batch sizes, or monitor and manage queues.  To use reliability prediction, begin downstream activities when you can predict that you will likely achieve your targeted reliability for the product.

What I like about this particular book, is that it doesn't prescribe a one-size fits all.  Instead, you get a pick list of options and strategies, depending on what you're trying to optimize.  It's effectively a choose-your-own-adventure book for product developers.