Testing and Quality


I’ve been thinking about testing and quality again lately. Across the industry, testers call themselves “quality assurance”, or “quality engineers”, and testers say they “measure quality”. I’ve had serious thoughts recently whether the two terms  (“quality” and “testing”) are as related as most people seem to think.

Testers don’t measure quality (and they certainly don’t assure it). The role of test is to provide information – but that information is just data – data about how the product was engineered. Things like code coverage, test pass rates, and bug rates are just status about the engineering effort.   Product quality – or perhaps better put – customer quality is different. Customers don’t care if you had 90% code coverage. Customers don’t care what your test pass rate is, and don’t care how many bugs they found. To them, quality is more emotional – quality products make them feel good, and help them accomplish a task in a new or exciting way. Somewhere along the line, someone decided that engineering quality was the same as product quality. Granted, the choices testers make to measure engineering quality are based on experiences in customer reactions, but it’s a crapshoot how much the the efforts align.

Graphically, you have two different measurements. In the worst case, they miss completely. For example, think of a level 5 cmm company that delivers bad software – they did it because they missed the boat on matching their process to measuring customer quality.

image

In a perfect world, the two measurements match exactly. In reality, I bet that engineering quality and customer quality match somewhere in the 25-50% – maybe 60% at the most.

image 

To get customer and engineer quality to match, you need measurements that can predict how the customers will perceive quality. This dictates that the best measurements would be from customer feedback during product development – things like beta feedback, newsgroups, and customer surveys may be more valuable than you think.

So what do you do with the engineering metrics? You keep them. If the role of test is to provide information, the right set of measurements can be the perfect tool for assessing risk. All that stuff like code coverage and pass rates that the customers don’t care about – they don’t care, but you still need to do it. If I buy a new car, I don’t care how many parts were replaced on the assembly line as long as it looks and works perfectly when I drive it away. Even though I don’t care about how the car was engineered, my perception of quality is affected by the quality of the engineering process – whether I know it or not.


Comments (4)
  1. Shrini says:

    Alan,

    I have difficulties in understanding various manifestations of quality as you have articulated here ….

    Product Quality – Intrensic to the product

    Engineering Quality – Quality of all (engineering) processes that produce the product

    Customer Quality – Eventually what customer cares about (will he buy it? will he recommend it?)

    Except customer quality (which is a time-bound relationship between the user and the product), other two forms of quality appear to indicate that "quality is part or an attribute" of "process", "product"…

    I just left a comment to your previous post quoting Michael Bolton that … "quality is a relationship (1-1) but not an attribute of a thing …

    What do you say?

    Shrini

  2. Alan Page says:

    Let me put it differently.

    I’m saying there are two kinds of quality: Engineering quality, and "Experience" quality. Engineering quality is a measure of how the product was built – Experience quality is what the customer sees.

    I don’t think engineering quality is process quality – i.e. I dont care how the process works, but I do care about the measurements. For example, I suppose there’s a process to code coverage, but what I care about is the data.

  3. I’m beating a dead horse here (or at least beating an old post ), but I have a story to share about quality

  4. 上一篇談 Tester 的職責是什麼, 這一篇談另一個觀念, Tester 不應該被叫做 QA (quality assurance), 因為基本上, Tester 沒辦法真的確保 "Quality".

Comments are closed.

Skip to main content