Yet another experiment in motivating people to find and fix bugs


Everybody has probably heard about some project where management decided to motivate testers and programmers by rewarding testers for finding bugs and programmers for fixing them. In the absence of high ethical standards, this can devolve into the situation known to Dilbert fans as I’m gonna write me a new minivan.

I experimented with this idea once, over a decade ago. Well, not exactly this idea, but a variation of it: For each bug in my code which I fixed, I paid the tester who found it.

The risk here would be that I would intentionally resolve bugs as INVALID (or WONTFIX or anything other than FIXED) in order to avoid paying the penalty for fixing them, but I like to think that I made my decisions on their technical merits rather than based on my pocketbook. Since the tester saw the bug resolution, there was a degree of oversight: If I used bogus bug resolutions to avoid the bug fixing penalty, the tester would just reopen the bugs. The amount of the reward varied based on the quality of the bug, so that encouraged testers to focus on finding good bugs instead of weenie ones.

I think I paid out a few dozen bug rewards. (It was a small project.)

Comments (11)
  1. Chriso says:

    Haha nice. Though, wouldn’t that carry the risk that small bugs would get overseen in favoring the bigger ones, giving a higher payout?

  2. Richard Neish says:

    Sounds like Donald Knuth’s rewards for bugs – http://en.wikipedia.org/wiki/Knuth_reward_check

  3. Mark (The other Mark) says:

    Chriso, it would depend on the scaling of the payout, I would think.

    However, I also believe Raymond left out the most interesting portion of this idea- What did he feel the result was? If he no longer does this (“I experimented once…”) I think we can assume the value to Raymond is lower then the cost of paying the rewards for every project. Which, of course, I find totally reasonable- I’d hate to self-fund a program like this on a large project!

    Question for discussion- Would the proper method of rewarding QA and Developers for bug free code be based on the number of post-release bugfixes requested by Tech Support? That is, QA and Dev are free to file bug reports and fixes as normal, but each issue that causes a significant enough number of calls to Tech Support that Tech.Supp. asks for a bugfix (Tech Supp does track and provide that info for your environment, right? :-) reduces the “bug-free” bonus for QA and Dev.

    To be fair, I’ve never worked under such a environment, but I think it would be interesting to do so. It is one of the metrics I like to use on my self-assessment at review time.

    [It was a small project where I was the only developer, so when a bug was found, it was clear who should pay out… Assigning blame is harder when the project has N developers. (The shared pool idea is interesting though.) -Raymond]
  4. Adam says:

    I’ve seen a system of two pots was tried.

    The developers started off with x amount in their pot and for every bug, money was removed and the testers gained from it – their pot started at 0.

    It worked because the devs didn’t want to create bugs, so tested a little more before releasing.  The testers wanted money so they tested even more obscure test cases in order to find bugs.

    If there was a massive difference (ie the devs actually managed to have something in their pot at the end of it), the testers were given consolation prizes (as developer’s succeeding means testers not).

    It was cool – we used money, sweets, time off, etc in our pots.

  5. Brad says:

    Hmm, how did you agree on what constitutes which "value" of bug? Severity?

    Sounds like an interesting idea, but the majority of bugs I’ve fixed in the last few months were pretty low priority bugs.

  6. Simon Cooke says:

    Hehe.. I did that back when I was at Sierra. I handed out $64 at one point to one tester. (It was a small piece of code, and I Knuth-doubled it, starting at $1.).

    Except I doubled it after every bug I found too. And got WAY over confident. :)

    It was worth it though.

  7. Jonathan says:

    Mark (The other Mark):

    This does happen inside Microsoft at a product’s level. The support organization provides extensive statistics per-product (# of incidents, minutes per incident, drill down according to root cause, etc), and it boils down to support cost. Then the product team – and their seniors – can compare revenue vs. support cost. A high ratio (more than 8%, I think) means the product has problems: Poor quality, poor usability, etc. A team can then invest in areas that have high support cost, in an attempt to reduce the support cost there. Our team has done that with great success.

    It doesn’t get drilled down to the individual dev/QA person though. If it did, it would push people away from doing complex area to safe, easy ones. Also, the delay is to long – if a release cycle is 1.5 years, and customers’ deployment cycle another 2 years, by the time you get the support calls the original devs have more often than not moved on.

  8. Mark (The other Mark) says:

    Thank you for the insight, Jonathon.

    In my area, products are generally either simple enough enough that only one developer works on them, or the components are separate enough that it’s fairly easy to tell who the problem belongs to. I sometimes forget that it’s not that easy for everyone.

    One of my biggest personal issues is that I often feel my products don’t receive sufficient testing before they go live.

    Additionally, much of the Help Desk, including their manager, works not far from me. I make it a point to stop by and chat for a few minutes a few times a week.

    As such, I’m very sensitive to the support costs of not catching a bug, and also very interested in various programs which help to uncover bugs.

    Which is why I was hoping for a quick and simple clarification from Raymond. I’m not sure why I was hoping for that, but I was. :-) However, reading between the lines, I get the message “I found it valuable, but seldom work on projects of the scale this would work for”.

    [I found it an interesting exercise but seldom work on projects where I’m the only dev. -Raymond]
  9. Jonathan says:

    Inside Microsoft, it is quite difficult to have good connection between the product team and the support personnel, which are scattered all around the world. Our team takes great pride in having accomplished that. I envy you for being as close to your helpdesk, and your helpdesk for being close to you.

    Off-topic: You typo’ed my name "Jonathon" – can you think why? Other native English speakers have also made this mistake, and were never able to explain why when I asked. I’ve been trying to find out for year!

  10. Mark (The other Mark) says:

    Thank you, Raymond.

    Jonathan- I had to actually look twice to see the typo. And, in fact, I had to correct it when I wrote your name at the beginning of this paragraph.

    I don’t know what to say, other then it’s an alternative spelling, probably similar to Mark versus Marc. A quick jaunt over to Bing says I’m in the minority. I wonder if it’s just English speakers from the Southern United States, or some other smaller subset subset of native English Speakers.

    I chose Southern United States because, at least where I grew up, the name Jonathon (And/or Thomas, Jonathon Thomas and Thomas Jonathon being a common pairing of names) was probably a reference to Thomas Jonathon Jackson, AKA "Stonewall" Jackson. He was a military leader in a failed revolution that occurred here in the US who is particularly considered a hero in certain areas, for various reasons probably too complicated and way too off topic for this particular forum.

  11. Rayfl says:

    Yet another good post Raymond, thanks.

    I used to do this in my marketing role (not, not bug checking).

    I offered £10 for every mistake that employees could find in literature or on the website – as long as they were the firs to find it.

    It was amazing how it motivated people.

    But the real benefit was the fantastic training it delivered – employees reading our own brochures carefully and understanding what we wanted them to!

    Ray

Comments are closed.