Characteristics of great Beta Testers

Things have been particularly busy here for the Encarta Test team as we roll out the Beta for our next version, Encarta 2005. This seems like a good time to talk about characteristics of a good Beta tester, and is something that I would like to elaborate more on in the coming weeks.

I would categorize great Beta testers into one of two groups.

1. Power User Testers. These are the beta testers who could certainly excel if they were to choose a full-time career in Beta testing. They're naturally driven by curiosity about new technology, and enjoy pushing the latest software to it's limits. These are the same testers that tend to be very active in the Beta discussion newsgroups, and enjoy the problem solving aspects of a interesting software bug. Testers in this category are the  “power users“ that we all know who have an aptitude for learning new technology quickly.

2. Less-Technical testers. These are testers who don't care much technology for the sake of technology. Instead, they're focus is on applying technology when it's needed and useful. They've got some great suggestions about how to improve products for common people. Picture our Moms and Dads that didn't grow up using computers, but still find interesting uses for them. Some of the most compelling suggestions for feature improvements come out of this group.

As you might guess, most Test teams at Microsoft are stacked with Power Users. We can script and code our way through our most frequent tasks, and know every keyboard shortcut of our heavily used applications so we don't need to waste time grabbing the mouse. We share these traits with our Power User beta testers. While the “core“ of our Beta tests rely on the feedback and bugs entered by our technically savvy Power Users, some of the greatest feedback comes through “suggestion“ bugs entered by our Less-Technical testers. These are the people who help us see the other side of the coin- that not all of our users find it clear when we intersperse phrases like “GB” and “MB” within the same document.

Perhaps the greatest tester of all is one that is a Power User, but still maintains the ability to see software as it's used by less-technical people. I often have problems “remembering“ what it was like to first begin using computers, but I think I share that problem with most of the software industry. In the mean time, we'll be using some of the non-technical feedback from our Beta testers to help us remember.


Comments (5)
  1. Nicole Simon says:

    puuh I am relieved. :o)

    Because of my interests I was eager to read about how testing was at MS and was very disappointed to see "okay, it is like a developer, but instead of developping the app, you develop tests’.

    Your post is the first to recognize this ‘other’ group and gives me back the hope, that I am not totally crazy if I ask programmers to add a forgotten shortcut on the third screen from the left or ask for another arrangement of menue entries because the do not feel like they should be.

    I use a comparison which goes with your two groups: Power users are the ones who tell Mom and Dad that they just need stick the key in their hand into the car to get it started.

    They find this very ‘basic instruction’ from their point of view, but for Mom and Dad it is very complicated, because they don’t know what ‘car’ is supposed to be, stand 5 meters away from a big something with doors. They try to do as they are told: Stick the key – where?

    This is where your second group comes in. They remember (or better: know) what Mom and Dad will probably face.

    Or other relatives. I just brought my aunt online, who is 60. She worked on her new laptop and was totally frustrated not only that the computer did ‘wrong’ but that I – without seeing what she did – could tell her exactly what she did ‘wrong’.

    She did nothing wrong. The programm itself had usability problems and wasn’t suited for ‘Mom/Dad/Aunt’. :o)

    btw, there was an edition of an older encarta on the laptop and she had typical difficulties with it – I think you have a long way to go gg No, just mocking a little bit.

    One thing I really appreciate: Usabiltity and interface are consistant across products/productlines and therefor enhance learning curves.


  2. Greg says:

    Hi Nicole- I enjoyed your analogy quite a bit. Every time a user doesn’t "get" how to use the software, it’s a failure for our design. Power Users all realize that a key goes into the ignition… but what if the car itself is locked? Many of the Mom/Dad/Aunt users become confused and frustrated with the software (or car). When events happen unexpectedly, it can really throw our less-technical users into a loop.

    There was an interesting post on jobsblog about a Microsoft Recruiter getting hired into a testing role years ago. The post is worth reading- take a look here. (Unfortunately, I agree with Gretchen and Zoe’s conclusion that this is increasingly difficult to do at modern Microsoft).

  3. Nicole says:

    The analogy is also a great test, to see if someone understands ‘Mom and Dad’. If someone does not understand them, how is he able to understand any demands customers make?

    Programming is always about ‘what the customer wants’ and not what the programmer thinks what is best for them.

    And yes, I know the post, this is how I got to know your blog :o) The question about the salt shaker is a great one; it gives me as a reader the possibility to think about getting out of the usual routines / boxes.


  4. AT says:

    IMHO, I will be to nice to collect feedback from Regular Users instead of Power’ed.

    Why ? Becouse most likely software will be used by Regular people.

    But it’s hard to get anything valuable in return.

    Possibly some kind of legal spyware collecting information on user clicks, opened topics, problems. Something that usability group does, but in a large scale.

Comments are closed.

Skip to main content