Usability Stockholm Syndrome


One of the many ways we test designs with real people is through usability testing. Although in Office 2007 we’ve greatly expanded the range, scope, and types of testing we’ve done to include everything from remote testing to extremely early deployments to longitudinal studies, we still do our share of standard usability tests.


What is a standard usability test? Well, normally it works like this: a test subject drives to Microsoft and comes into our labs. The usability engineer responsible for the test greets them, answers any questions they might have, and then sits them in front of a computer which is running the software or prototype being evaluated. The usability engineer runs the test from an adjoining room, while the designers and program managers responsible for the feature being tested either watch in person or from their offices via a video feed.


In many tests, the test subject is given a set of tasks to complete, and asked to verbalize their thoughts as they go: what their expectations are, what they’re looking for, if they’re getting frustrated… things like that. Other times, people are given more open-ended tasks, such as “make a document exactly like this printout” or even just “make a nice looking resume.” Sometimes we have people bring their own work to Microsoft and they complete it in our testing environment.


Most times, the usability subject has filled out a screener ahead of time which helps us judge how much of an expert the subject is at using the software being evaluated. The point is not to exclude anyone, but to help us analyze the results–we do test everyone from ultra-novices to super-elite power users.


When a test is done (usually between one and two hours later), the subject is given a software gratuity as thanks for donating their time, and the cycle of improving the design begins anew.


Back when I first was exposed to usability early in my Microsoft career, my expectation was that people were really going to be super-critical. After all, the software is usually in a pretty rough state during the tests and this was people’s one chance to really let Microsoft have it and let out their rage at things not working as they expected them to.


But it turns out that this impulse is generally wrong. In fact, people tend to be much less critical of the software designs they’re testing than they probably should be.


I think of this as a form of “Stockholm Syndrome,” in which a hostage becomes sympathetic to his captors.


Number one, people are coming to our labs, as a guest of Microsoft. There’s a little piece of human nature which says you don’t go to someone’s house and then insult them. They come to our place, we’re giving them free stuff–no wonder they subconsciously want to please us a little.


Secondly, people have an innate tendency to blame themselves when they can’t complete a task, instead of blaming the software. You hear a lot of “oh, I’m sure this is easy” and “I’m so embarrassed I can’t figure this out.”


Maybe this comes from taking tests when you’re in school, knowing that every question has a correct answer and if you get one wrong, it’s your fault. Maybe computers are still so complex that people feel like they should have to undergo training in order to use them correctly, and failing a task in usability plays on that insecurity.


Whatever the cause, this tendency to not criticize the software is a major risk to the results of standard usability testing. Our usability engineers are well aware of this and take great pains to ask test subjects to be critical and reassuring them that “it’s not a test of you, it’s a test of the software.”


But there’s always the potential of skewed results, and this is one of the reasons we’ve supplemented standard testing by initiatives in which we watch the software more in the real world–including technology to perform tests remotely on people’s home computers, far away from the bright lights and cognitive din of our on-campus labs.


Interested in participating in usability research as a participant? Visit http://www.microsoft.com/usability.

Comments (31)

  1. Rosyna says:

    Is this why anyone thought that the disappearing menu items in Office were a good idea? You know the ones.. the menus only show the "used" menu items and then there is a double arrow at the bottom to show all menu items. Ugh.

  2. J says:

    "But it turns out that this impulse is generally wrong. In fact, people tend to be much less critical of the software designs they’re testing than they probably should be."

    Surely you know by now that the way you can get people to be more critical is to have them post anonymously on an internet forum?  The only problem is that in addition to them being critical, you’ll have them being rude and arrogant, so maybe it’s not a perfect solution.

  3. Adrian says:

    One company I worked at would sometimes do usability comparisons with our product versus the competition.  You always knew when these were happening, because all the signs on the buildings and around the campus were covered with slipcovers to avoid biasing participants.  They were only told after the test which company had invited them.  We also did testing in different parts of the country, to get a more realistic cross section of users.

    Few usability conclusions were based on test subjects’ opinions.  Most were based on behavior.  Did they find the option?  Did they complete the taks?  The easy way or the hard way?  Yes, people will blame themselves instead of the software, but their actions will show you where the real problem lies.

  4. Marjut says:

    Maybe you should consider using an outside company for testing? In that case people would tell more about how they truly feel and not be afraid of hurting anyone’s feelings in the process.

  5. clap clap says:

    Slashdot trolls arrived early today, must be spring…

  6. Max Howell says:

    The comments to this blog have gone seriously downhill since the ignorant hoards of the Net found it.

  7. Mwahaha says:

    Stockholm syndrome, as in the town name in Finland.

  8. Confuzzled says:

    You must be confusing that for Helsinki in Holland

  9. Nidonocu says:

    I think this blog needs moderated comments turned back on again. :/

  10. kish829 says:

    Some comments need to be cleaned.

  11. Nick Murray says:

    "this tendency to not criticize the software"…

    Jensen, this is hard to believe. I’ve had enough bad experiences with software (both Microsoft and non-Microsoft) to know that it is *usually* the fault of the software designer, not me. The users you describe here appear to be novices, whereas many of the comments posted in response to this, and other, Office GUI blog posts, show users with a high level of skill, and equally high expectations.

    The ribbon and associated UI improvements will make a vast amount of difference for novice users who don’t know where to find things. I just hope that people who know what they are doing won’t be left wanting.  

    In many cases, the current Office UI actively blocks users from doing things. For example, when you build a rule in Outlook, you have a 3-deep stack of modal dialogs on the screen. This makes it impossible to scroll up and down in your Inbox or any other folder looking for email addresses to add to a rule. Many of my contacts are NOT in the "Global address book".

    Modal dialogs are just evil.

  12. jensenh says:

    I deleted out the offending and lewd comments.

    Sorry they persisted for as long as they did.

  13. MaxPalmer says:

    Jensen,

    Is there much usability testing done outside of the US?

    Always interested to learn how you do usability testing. Thanks for the continuing feed of information and sorry to hear about all the problems with the comments system.

    Max

  14. Al says:

    Interesting way of putting it – Stockholm Syndrome.

    I see a similar reaction in some projects. Initial acceptance testers love feature A, but months down the track the same users start to hate it.

    Maybe people just love any new thing, but after a while, they get over it. Then they really start to evaluate something properly, without the self induced hype.

  15. jensenh says:

    Max,

    There are some countries in which we do more than others.  For example, we do a lot of usability research in Japan.

  16. TIm S. says:

    Jensen, I have to say that I completely agree with your "Stockholm Syndrome".  We’ve seen identical reactions in the usability testing that we do; we’ve just never thought of giving it a name.

    One thing that we started doing which has helped us immensely towards getting back more honest feedback is telling the participants that we (the testers) had nothing to do with the design of whatever it is that they might see (that may or may not be true).  They’re not going to hurt our feelings in any which way.  

    This has had a remarkable impact in getting more "blunt" comments from our users.  A few of them will mention off hand to us "well, since you didn’t have anything to do with this, lemme tell you something…"

    Incidentally, we don’t call our participants "subjects" – we call them "test participants" or "users".  "Subjects" makes them sound like white lab mice that we’re experimenting on.  And as you state, we’re not testing them, why call them something that’s tested on?

  17. jensenh says:

    Tim,

    That’s very interesting!

    Incidentally, we don’t really call our participants "subjects" either.  We call them "participants."

    For some reason, I thought calling them "subjects" in the article made it more clear to people what I was talking about. 🙂

  18. Dave Wollrich says:

    A means of avoiding this syndrome (by whatever name is being used) is to do the testing in a neutral atmosphere, that is, an office site that is not specifically at a Microsoft facility.  There is no point (and it would show a lack of integrity) in hiding who is doing the testing, but I suspect you would get less of the "sympathy" factor involved if it was not at a Microsoft office.

    Another approach (which has other dividends as well) would be to have the user test both the Microsoft and a competing product (they do exist, don’t they????).

  19. Josh says:

    <blockquote>”One thing that we started doing which has helped us immensely towards getting back more honest feedback is telling the participants that we (the testers) had nothing to do with the design of whatever it is that they might see (that may or may not be true)”</blockquote>

    Maybe I should be a lawyer, but what I always say is, “I didn’t make all this stuff you are going to be looking at, so don’t worry about hurting my feelings…” They assume that means I didn’t make ANY of it, but I am just saying I didn’t make ALL of it 🙂 Anyway, I feel better about it because I am not lying to the participant, which I tell them up front I am not going to do.

  20. Have you tried to do this in a less controlled environment? For example, set up a Terminal Server and let users access it for a week or so. For that time, give them a few practice tasks to accomplish with the new piece of software. A task should maybe take them a maximum of an hour to do, and you’d collect direct feedback from them afterwards via a questionnaire. After they had the chance to practice using the new piece of software, invite them to your location and do a conventional usability test. I think your results would be a lot better though, because the user wouldn’t be discovering things for the first time, but rather would try to actually be somewhat efficient in doing the task.

    Personally, when I got Beta 1, I was very thrilled and loved the UI without questioning it for the first few weeks. After I had started really using it though, I started questioning how certain things worked.

  21. hwaite says:

    Slightly off-topic.  Hopefully not warranting of deletion.  I understand the value of honest criticism and imagine that it is well worth all the hassle and expense of recruiting users, poring over videotape, etc.  Vocal users are far more helpful than polite ones.  This being the case, I can’t understand why Microsoft has made it so difficult to report issues (at least for those of us that are not part of an official usability study).

    I’m a software developer and consider it a professional courtesy to report bugs or even usability issues when one stumbles across them.  I recently encountered a minor bug in Internet Explorer and figured I’d submit it via the Microsoft bug tracking site.  Couldn’t find one.  Hunted around on the web for a bit and learned that there’s a ‘Feedback’ button in the ‘Help’ menu.  The trail leads to a support phone number.  Next thing I know they want to charge me for tech support!  Or “maybe you can send a physical letter,” I’m told.  Am I missing something here?  Seems like you’re throttling one of the most efficient conduits of product feedback.  Is it because there is just too much chaff?

  22. Will Pearson says:

    I’m just wondering whether this is related to Orm’s theory on Demand Characteristics.  Participants who have volunteered want to be "good participants".  Maybe they feel that being critical of a product, rather than just saying good things about it, is not what the people running the tests want them to do.  It would be interesting to do some pre/post testing interviews to try and figure out exactly what participants viewed the purpose of usability testing as.

    I do think you’ve got a point about testing though.  Regardless of how much people are told the test isn’t about them they always seem to think it is.

    Will

  23. Jensen Harris: An Office User Interface Blog : Usability Stockholm Syndrome The term Stockholm Syndrome describes the situation where a hostage becomes sympathetic to his captors. When Jensen Harris had to conduct his first usability test at Microsoft,

  24. Jensen Harris: An Office User Interface Blog : Usability Stockholm Syndrome The term Stockholm Syndrome describes the situation where a hostage becomes sympathetic to his captors. When Jensen Harris had to conduct his first usability test at Microsoft,

  25. Jensen Harris: An Office User Interface Blog : Usability Stockholm Syndrome The term Stockholm Syndrome describes the situation where a hostage becomes sympathetic to his captors. When Jensen Harris had to conduct his first usability test at Microsoft,