You never know until you test it with real users


Speculate all you want about what users expect, what they want, what they care about. There’s no substitute for actually running experiments to find out. Those who haven’t already been following Jensen Harris really ought to be, because he talks about user interface design in a highly practical way. Consider this entry on deciding how the ribbon should behave:

These rules came directly out of watching people use the product in the labs over the last few years. Other rules we thought would be important (such as always collapsing from right to left) turned out to not matter at all.

Or this one on the special rule for lingering:

What we found was that an extremely common scenario involved people using an object, clicking away from it to make the selection handles go away (to get a better look at it) and then clicking right back on the object to continue formatting it.

Notice that prior to discovering and then implementing the “lingering” rule, users who used Contextual Tabs were frequently confused. But once the rule was created, the usability results are solid.

Comments (16)
  1. Andy C says:

    Jensen’s blog should be on every developers must read list. Much like your own, Raymond.

  2. J says:

    My favorite part of the blog are the comment posts of people saying stuff like "That’s a dumb way of doing it–nobody uses their computer that way!  You should do it THIS way instead:  <random speculation>".

    Experimental results apparently don’t matter to a lot of people.

  3. PatriotB says:

    Now if only Jensen would observe someone saying "why the heck doesn’t this application use the Windows title bars?!"

  4. TAG says:

    In addition to changes in products – you can also change people.

    The best customers I had – were people who sow computer for first time. Once I will them how everything works – they become happy users.

    The worst customers I had – users of completive product. They were trained to do so many stupid things – so they asked for some additional useless features. Once I have to add a button that does nothing (not even showing popup), simply because folks were trained to push it to get results.

    So – real users are often unrealistic.

    For usability studies I wish you used folks who never sow computers. If they will be able to figure out everything without relying on assumptions computer user know – then it will good for everybody.

  5. jon says:

    One word : Clippy.

  6. Raymond II says:

    I guess sending developers home with the product is asking too much.

  7. Tony Cox [MSFT] says:

    "I guess sending developers home with the product is asking too much."

    You’re missing the point. Developers using their own product simply doesn’t uncover many usability issues. The reason is obvious – they developed the product, so they know how to use it.

    Trust me, I’ve watched usability studies on my own products, and it’s extremely eye-opening. Users get confused by things you thought were completely obvious, or try to use your product in ways which suggest a completely different mental model than the one you used when building it.

    You just can’t argue about this stuff by anecdote. You have to do a real study. Often the results confound expectation.

  8. 8 says:

    Jon, indeed… did Microsoft dogfood Clippy?

    Did no-one try placing the taskbar vertically? Did no-one try installing a font? Did no-one notice the toolbar context menus don’t work when there’s no page loaded? Did no-one got frustrated that the recycle bin starts in the right place and warps to the other side when arranging icons? Did no-one ever notice the X-Windows style (focus on hover) gets X-Windows style wrong? (doesn’t always follow the don’t raise, just focus configuration). Did no-one get the bug where WMP starts itself and goes to the Media Guide, then when you close it, it comes back, and back, and back, and over and over again? Did no-one got his usb stick mapped to a drive already used by a network share? Did no-one try to record more then 60 seconds at a time with sound recorder? Did no-one ever overlap a solitaire window doing its win animation and go back again (shows lack of polish imho, bad for a program of that age)? Did no-one try to change the colour pallete when using visual styles? Did no-one find the keyboard mouse enabled again even after disabling it and other glitches when using hibernate? Did no-one find it weird that to add a standard (jetdirect/lpr) network printer you have to select local printer, then add a new port in another wizard and go back to the previous wizard? Speaking of wizards, whats the logic in the configure my network wizards? I really didn’t expect it to open another wizard after finish. Did everyone find their keyboard layout settings in the locale settings?

    And the list goes on and on…

  9. required* says:

    "If you limit yourselves to test subjects who’ve never used a computer before, you’re eventually going to run out."

    Not if you farm people for that very purpose.

  10. PMStalker says:

    If you limit yourselves to test subjects who’ve never used a computer before, you’re eventually going to run out. (Admittedly, it might take a while, but…)

    Besides, new users are (usually) impressionable and open-minded, if somewhat terrified of making the machine explode; the users you really have to worry about are the crotchety ones who’ve used computers since the dawn of the vacuum tube, and expect nothing in the behaviour of the machine to change. These people need to be given good reasons *why* the changes were made, or they’ll never accept them.

  11. 8 says:

    Tony Cox, I know what you mean, and I don’t want to brag but I know what’s wrong with my dialogs the moment I design them, because I design from the new user perspective (I’m the first new user to my own program). I sortof complain about my own dialog while its not even finished yet. Though I refuse to completely rely on this instinct because I’m too familiar with all major desktop environments. I’d like to pass it on, but you kinda have to "catch it", I guess.

    I think this is related to other GUI experiences, so I’ll share them if you’re interested.

    I can’t work with WinCommander, it just looks very cluttered to me. Many people laugh about it when I say that, but it’s true. Explorer (with the folder pane on) is fine, I’ve setup Nautilus to look the same. Ofcourse WinCommander isn’t from Microsoft, but I bet you’ve seen it, and I bet you can use it.

    I also couldn’t create a table in Word until I learned it from a book (I was limited to basic layout, figures, clip arts and wordarts, everything else was beyond me). But I learned StarOffice all by myself, however (and learned that a word processor isn’t made for DTP when I tried to push it). A huge difference is that page layout is under layout, not file. Layout is divided into 3 basic parts: page, paragraph and character. This just makes a lot of sense to me, it gives a more coherent feel.

    But the new OpenOffice.org 2 frightens me. Especially Impress which is completely different. I can’t adapt to it.

    MS Access was the hardest of all applications to learn. I never could learn it by myself. I’ve learned database normalisation and SQL syntax, but not with Access. I was also unfortunate to have to look at the "hostile" appearance of Office 2003 (it makes it harder to see, and the high contrast of dark blue and white in Word degrades my concentration), I couldn’t disable it because of administrator policy (on the display properties). Anyway, I had to learn it because it’s mandatory in the IT course I’m doing, and the classmates had a laugh at me because they know me as a Linux guru (but that’s not really true, I’ve ran Windows for much longer then Linux) and there’s usually nothing in software or networking that they can teach me, except this. Good thing the Office website has all the help I ever need. It has excellent step-by-step guides, and I couldn’t have done anything without them.

    Also I couldn’t learn Photoshop by myself (and never got around to learning it), but I was able to learn the GIMP by experimenting. I’m still not an expert on that yet, but I can get the job done. (And to be honest, going from gimp 1.x to 2.x was a pain.)

    And if I haven’t used Windows in a long time, I just forget how it works, from a user perspective (funny, because I can’t seem to get win32 out of my head :) and have to basically re-learn it (simple things like changing taskbar & start menu settings for example).

    So maybe you should try using a completely different desktop environment for a while. That’ll give you the "new user experience" for sure, and perhaps give an insight. I’d recommend Mac OS X, because that’s *way* off from Windows. You don’t neccesarily have to agree with the design decisions ofcourse.

  12. Raymond II says:

    Tony Cox [MSFT]: "You’re missing the point."

    Yes, that’s because I chose to respond to the general idea stated by the title of the entry. That is, my frustration with poorly tested products, which have glaring errors that would get caught by simply using them as if you were the customer (Windows NT 4.0 SP6a springs to mind).  Anyway, I’ll make up for it below, hopefully..

    "Developers using their own product simply doesn’t uncover many usability issues. The reason is obvious – they developed the product, so they know how to use it."

    Now we’re into definitions of what a developer is, can or cannot do, based on what makes sense within some business culture.  It’s possible to play both roles, and many do it with success, maybe because some of them have to think for themselves to "survive."

    If you’re making music, you have to be able to switch between the listener and the composer.  It requires self-discipline, being critical and having an understanding of what it is you’re creating.  It requires distance.  Taking a step back.  The musician is the developer and the user.

    I feel old-fashioned. :-)

  13. Lionell Griffith says:

    The end user has a real job to do:  create content for the purpose of communicating real information to people who give a damn.

    What I see here is change for the sake of change in order to get still more upgrade payments.  All the end user seems to be good for is to be a source of those payments.

    Focus on the results to be accomplished and make the effort to create the results effective and efficient.

    PS: To create content one must be able to SEE the content.  Eye candy is NOT content.

  14. 8 says:

    Eye candy can become very distractive

  15. Anonymous says:

    It’s silly to think that only practical usability results are what drive the UI development. Surely sales and marketing has a lot to do with it, maybe even contrary to what usability tests say. If the product looks the same its not going to sell too many upgrades. There’s also the legal factor. Make it look unique so copy-cats are obvious, etc.

  16. Darren Winsper says:

    When Netscape were developing Netscape 6, a lot of "interesting" UI decisions were made by marketers, causing immense frustration to the developers that had a sense of UI design.  Netscape 6 was a buggy product released way too early, but Mozilla 1.0 was reasonably solid and the biggest complaint about it was that the UI sucked.  Firefox, on the other hand, was designed with an eye to being simple and usable and look where that has taken them.  I’m sure Microsoft have learned similar lessons over the years.

Comments are closed.