The Acid Usability Test

Designing software is fun fun fun. During the OneNote project we had countless hours of brainstorming design sessions full of laughs and great ideas. There’s really nothing quite as intellectually stimulating as a tough intellectual problem, 3-5 smart, articulate people, and a whiteboard. Ideas come and go. Analogies drawn. Metaphors extended. One of my goals in such meetings to is to inject hilarity where possible, and some of the deepest laughs I have ever had have come from those sessions. Surfing the “froth” at the tip of the idea wavefront in sessions like these is incomparable.

Then, the test comes. The idea we have laughed and cried over and finally reached an agreement on is prototyped, and we put it in front of “normal people”. These heathens then have the gall to not understand how to use the feature we have designed, or not discover it among all the other stuff, or otherwise demonstrate through their confounded normality that our ivory-tower design sessions have produced a masterpiece of hokum. The program manager “owner” of the design drops his or her tail, and sulks through the remaining sessions (usually 6-8 in total) glumly verifying that yes, seven normal people out of eight could not make heads or tails of their masterwork.

By the next morning, the feedback has been internalized, the design ideas flow again, and we try once more to come up with the design which somehow meets all the criteria, fits within all the constraints, actually solves the problem at hand, and this time will in fact be usable by the dreaded “user”.

Iterating in this way, we can work out most of the fundamental errors in a product’s design. But usability tests are not a panacea. In fact they can be quite dangerous. Although they ostensibly test normal people, these kind souls are people who come to the hallowed halls of Microsoft in Redmond. They are very aware that they are going to be using something new, and they should be on the lookout for changes – plus they really want to be helpful, so they take extra special care to examine what’s in front of them. This is all quite unlike true normal people, who don’t have time to notice software and just need to get their work done.

Another bias that usability tests introduce is the “optimization for first use” phenomenon. You’ve probably seen this. There’s a feature you use which seems innocuous the first time you use it, but as you use it more and more, it bugs you that it takes so many steps to get it done. Or that it seems so verbose, with a lot of user interface for something which is really quite simple. What has happened is that the usability test showed that people who have never seen a feature before have trouble with it in the first hour of using it. So the designer makes the feature hold your hand through the process. That improves the results in the test, but ruins the feature for people who know what they are doing.

Remember Microsoft Bob? That was an extreme. People who didn’t know anything about computers could use Bob to write a letter. But Bob was so optimized for the first time user that even those utter novices would quickly tire of its helpfulness and want to move on to something that got out of their way and just let them do their thing.

Comments (13)

  1. Woo Seok Seo says:

    GUI sites is great!!!!! 😉

  2. Woo Seok Seo says:

    If I want to be a member of your team, what should I do?

  3. I’ve probably checked out a whole bunch out there (as many as scoble?), but I currently have just 30 blogs on my aggregator. Now you’re on it. <tongue-in-cheek>Congratulations.</tongue-in-cheek>

  4. If you’d like to work on OneNote, you can apply to Microsoft and specify that. We tend to have a full team because we’re a popular place to work though.

  5. rick says:

    I’ve enjoyed your posts about OneNote so far and look forward to whatever else you offer up.

    Your mention in this post of "optimization for first use" in particular reminded me of Carl Franklin’s post (url below) a short while ago where he pondered development of UI Layers that would provide access to more features, content, and complexity as the user progresses in use of an application. His focus was on availablity of design tools for implementing such an approach and think that absolutely merits some thought.

    It seems to me that implementation of this concept would allow truly excellent new applications like OneNote to get us started with first use optimization and either guide us gradually to features we haven’t used or possibly be aware of the types of actions we’re taking and offer to show us features that might be related.

  6. Reminds me of Mac Word 4.0 There was a "beginner" and an "expert" mode for menus. People were confused by this, and thought the app had lost features from version 3. Even those who understood what was going on switched to expert soon enough because they needed some feature or other that was only on the expert menus. I am not sure that limiting visible features does all that much to help people understand an application – people are quite good at skipping over things they don’t understand – you don’t gain that much by hiding them. The way we solve this now is to provide a UI that lets beginners succeed, but also provide ways to tweak the behavior (options), shortcuts, and other direct ways to let people who know what they are doing succeed. There is no ideal solution for any design problem however – only optimizations. As I tell my team, "designing is deciding who gets screwed".

  7. Peter Torr says:

    Design meetings a f-u-n. We also have a lot of laughs over in Visual Studio… and then feel the agony of those usability tests. Oh, the pain. Make it stop!

  8. Usability is very costly because it always need to involve an enough number of people. Therefore, our team is studying an automatic usability approach, including the consideration of information architecture on the system design. The approach will be different from what Melody Ivory did before.