Make Your Performance Work Count: The 20% Guideline


I recently read Steve Seow’s book, Designing and Engineering Time: The Psychology of Time Perception in Software. Steve is an expert on performance engineering at Microsoft, and really understands the human element and psychology behind software performance.


A really interesting take-away is that users do not notice changes in performance below a certain threshold. Research has shown that for durations under 30 seconds, users do not detect differences up to 18%. Steve’s rule of thumb is that users really won’t notice performance changes of +/- 20%. So if you are looking at setting goals for a performance improvement, or considering performance against a competitive product, your goals should be to gain at least 20% over the baseline measurement.


Likewise if you are considering whether or not to fix a performance regression, 20% is a good measurement to consider. If the regression is less than 20%, users likely won’t detect the degradation, so you may be better off spending your effort elsewhere.


Of course this is a slippery slope. If each release you allow a 20% degradation, users certainly will notice and stop using your software. So it is important to maintain and meet performance goals that will keep your users happy with your software. One of my performance testing customers ends his emails with this quote:



“If it is fast and ugly, they will use it and curse you; if it is slow, they will not use it.”


-David Cheriton in _The Art of Computer Systems Performance Analysis_


Ed.

Comments (8)

  1. pazu says:

    Funny. Unfortunately Microsoft is using this rule again again so from W95/W98/W2K/WXP/WV now we have 1.2*1.2*1.2*1.2 = 2.0 slowdown, don’t we ? He ? Me noticed !

    (Not speaking my hardware is from that time sooo faster).

  2. edglas says:

    Hah, that is funny! I almost put a caveat in the post about accumulating 20% degradations, of course it adds up to greater than 20%. And no, this is not a wide-spread, in use rule at MS. 🙂

    BTW, you’ll like Win7, it has a bunch of perf and resource usage improvements over vista.

    Ed.

  3. Guy kolbis says:

    Ed Glass from the MS Load Test Team wrote a post: Make Your Performance Work Count: The 20% Rule Basically

  4. cflare says:

    It really depends upon the application.

    Research under 30 seconds doing what?

    At 30 seconds, 20% is 6 seconds. Typing on a keyboard, sure; waiting for a program to load, probably noticeable.

    So the real rule should be, does the amount of work justify the improvement?

    This rule seems fair in the world of OS and clunky mouse users. However, in Business applications (where the user repeats many micro tasks indefinitely), that 20% builds up over time when you have 80 reports to run, or 500 forms to fill out. It gets even worse when that 20% is because of added in keystrokes to a user-end process. Making a repetitive task takes more strokes, is unforgiveable.

  5. Performance & the 20% Rule I was reading the post from Ed Glas , Make Your Performance Work Count:

  6. nguyer says:

    Very interesting! Thanks for the tip.

    By the way, I have been trying Windows 7, and it does seem fast. I was not impressed by startup/shutdown, but sleep/resume was like lightning! Really looking forward to it.

  7. edglas says:

    clflare,

    The rule strictly applies to human perception, so it only applies to performance measurements that the user experiences directly. When I considered some typical response times this made sense, +10% would be hard to notice, I think I’d start noticing at +20%:

    time +10%   +20%

    2     2.2    2.4

    5     5.5    6

    10    11     12

    30    33     36

    The specific case you pointed out was 30s. Really 30s is unacceptably long for any response time. I stop paying attention and start on something else at about 10 to 15s. This is also backed up by Steve Seow’s research, I have another post coming on that topic. That said, I do think going from 30 to 36 would be noticable, less than that probably not.

    A good example of a performance measure this applies to is the time it takes a web page to load. What really counts there is the first point at which the user can interact with the page and read it, it doesn’t necessarily have to be completely loaded.

    A case you point out and I agree with that it does not apply to *at all* is anytime the measurement may something like a function time, say String.Concat that the user doesn’t experience directly. If String.Concat took a 20% hit it would have a far greater performance consequence than 20% for some operations since it will be called many many times.

    The other interesting case you brought up is something like typing. Steve Seow categorizes these operations as "instantaneous", and there is interesting research on this as well. The threshold that humans perceive an operation as instantaneous is 100 to 200 ms. If you go over 200ms, it is no longer instantaneous. So I agree with you on that as well.

    Ed.