More Than Just the Two-Way Mirror


I’ve
revealed in the past
how many of the Office 12 UI decisions we make are
influenced by what we learn in collaboration with our usability and research
team.

Personally, I find usability tests scary.  You have an idea, discuss it
with others, work it into a prototype and get excited about it.  You want
it to work.  And then you pull back the curtain, put the idea in front of
real people and all of a sudden you know whether the idea works for real or not. 
It’s totally humbling.

It’s like when you’re dieting–you can feel good about the food you eat or
exercise you do, but nothing brings it into focus like stepping on the scale and
facing the result.  If the result is bad, it’s back to the drawing board. 
If the result is good, you set it aside, plan to retest, and start focusing on
where else you can improve.

Over the next weeks, I’m going to write in more detail about our usability and
research efforts around the Office 12 UI.  To give you a taste, here’s a
snapshot of all the special research projects we have getting started over the next
month or so.  The following bullets are excerpted from a status e-mail
message written by Tim
Briggs
, one of our usability leads.

  • Office 2003 Benchmark (10/19 – 10/20): 30+ people coming through
    the Office Design Lab, 10 at a time to complete tasks in previous versions. 
    We’re collecting success/failure, time on task, and satisfaction so that we
    can compare to the Office 12 Benchmark later.

     
  • Eye Tracking 1 & 2 (10/19 & 11/11): Using the eye tracker to give
    us better understanding of how people use and browse the current visuals so
    we can then refine and study the new visuals.  We’re looking especially
    at how people use the chunk titles, the contextual tabs, and the MiniBar.

     
  • Card Sort II (early Nov.): A year ago we had people organize
    commands into buckets to help us think about how to generate the tabs. 
    This time we’ll give them our buckets and see how well they can sort the
    commands into them, especially while coming up with names for some really
    sticky ones.

     
  • Internal Longitudinal Study (Now – Dec.): We’re getting people
    from the company set to install and start using both current builds and Beta
    1.  All are outside of Office, most are from non-product groups like
    Marketing, Sales, and Legal, and will not only be sending feedback but
    participating in the Office 2003 and Office 12 Benchmarks so we can compare.

     
  • The “Truman Show” (Now – Dec.): A brave soul from the “real”
    world has agreed to let us take away his Office 2000 and replace it with
    Office 12… and do site visits… and work here on campus for a while… and
    have conferences via Live Meeting with us… and let us analyze his personal
    instrumentation data… and send us a daily journal of his experience. 
    He’s a local guy with a small personal business and excited
    about this challenge.  As are we.

     
  • Office 12 Benchmark (Nov. – Dec.): We’ll look at the whole range
    of use from OOBE (“out of box experience”) with newcomers to 2 weeks/2
    months use with the internal longitudinal people.  Like the Office 2003
    Benchmark, the tasks are selected to give us some key comparisons we can see
    in the big Office Design Lab data and more personally through 1:1 standard
    lab sessions.

     
  • Extended Usage Study (Now – RTM): We’ve started deep engagement
    with a large group of users from a local company.  We’ll be rolling out
    Beta 1 to all of these people.  As this kicks off, it is the first
    major opportunity to closely monitor the rollout, training, adoption, and
    acceptance of the new UI over a long period of time.  We’ll be doing
    persistent visits, monitoring instrumentation data, and collecting 1/1
    feedback.

     
  • Beta Survey & Visits (Dec. – Feb.): As part of the overall
    beta plan, we’ll be adding to the benchmark data with a series of site
    visits to customers to observe first-hand “use in the wild.” 
    Members of the design team will be partnered up with customers so that we
    can follow up more individually starting in December.  Well also be
    following up with surveys to drill into productivity and satisfaction.

     
  • This is just the known, scheduled research. There are lots of other
    efforts going on too like building up the list of instrumentation metrics,
    3rd party validation studies, continued “Send a Smile” tracking, continued
    consolidation of usability findings from Word, Excel, PowerPoint, Outlook,
    and Access, and oh so much more.

Jensen here again.  All of this is in addition to the normal ongoing
“behind two-way glass” usability tests that focus on individual mechanisms or
features.  And of course it’s added to all of the beta feedback we get through newsgroups, customer
visits, e-mail (and even here in blog comments!)

As you can see, we have a lot going on as we strive to learn more and more
about the UI and how to continue improving it.  It’s a large effort, but I
think it will be worth it.

I’ll fill you in on the details of some of the more interesting studies in
the coming weeks.

Have a great weekend everyone!

Comments (14)

  1. Really interesting to see how much effort you put in.

    How do you like the Eye-tracking stuff? I do some work with a UK usability company, and the eye-tracker is increadible.

    There’s so much data and analytics to be gained from there.

    My clients don’t think they are making the best use of their data currently; I get the feeling there’s not much hard-science and indsutry-standard ways of analysing the results.

    But maybe that’s just me? It’s impressive technology, at any rate.

    Cheers,

    Andy

  2. Jay Zipursky says:

    Can you tell us how many usability engineers are involved in all those activities? It looks like it’s a huge undertaking.

  3. sloan says:

    Have you also equated a "best possible" time metric to complete a task? Based on estimates of cognitive reaction times, the time it takes to move a mouse and hit a target… I think everything you are doing is great, but I’d also be interested in how your results compare to theoretical perfection. 😉

  4. Wayne Walton says:

    I fine this facinating. I have done some work with site usability in the past, but nothing so deep.

    Will generalized results be made avaliable, via Microsoft Research? I find much of what I find there very useful.

  5. TC says:

    What is: "Send a Smile" tracking ?

  6. Paul DeGroot says:

    Interesting, Jensen.

    I’m someone who still uses Office 2000 because Windows XP broke the comments and reviewing features, and they weren’t fixed in Office 2003. I don’t expect them to be fixed in Office 12 either, since some of these bugs were logged and reported in the public Knowledge Base, but they haven’t been fixed. So I assume they’re not considered serious by the Office team, and won’t be fixed. However, they’re showstoppers as far as I’m concerned. I do a lot of editing and spend a lot of time writing comments for other users and reviewing their comments in my work.

    Here’s a quick list of my beefs (these have all been reported at least twice directly to people in the Office team, by the way, another reason I’ve concluded they won’t be fixed. MS knows about these and knows they’re showstoppers for us, but hasn’t addressed them.) Note, by the way, that we normally work in outline view, so if you try this, you should be in outline view.

    + The Delete key doesn’t work on comments in the reviewing pane. Press Delete, nothing happens. I can’t imagine what feature this implements, and it was recorded in the KB as a bug, but it hasn’t been fixed.

    + The Backspace key doesn’t work when text is selected in a comment. Select text in the reviewing pane, press Backspace, nothing happens.

    Note that Delete WILL delete if text is selected, and Backspace works if text ISN’T selected, so there’s a bizarre mirror image of disfunctionality here and I gotta believe that this is a trivial fix.

    + When viewing a document in which revisions have been tracked, if you hover the mouse over deleted text with a comment on it (marked with a strikethrough) the tooltip that appears shows you the deleted text, and not the comment. This is bizarre. I can see the deleted text in my document, so showing it to me again in the tooltip (and again in the reviewing pane, if it’s open) isn’t useful. What I want to view is the comment about the deleted text, which typically is an editor explaining why the text was deleted. The only way to view the comment is to switch into a different view and look at the balloon (in which, by default, you can now see the comment, but not the deleted text. Who thought that up?).

    If you want to know how this should work, simply see how Word 2000 does it. Delete text in a file viewed in outline mode, enter a comment on top of the deleted text, and see how it is viewed on a mouse-over. Also, see how the Delete and Backspace keys work in the reviewing pane, both with and without selected text.

    This is a showstopper for me, because I really hate to use software that constantly reminds me that it’s buggy and disfunctional. When I’m using a version of Word later than 2000, I get this reminder about every 10 minutes, and have to switch views, hunt through the (also buggy) reviewing pane, and perform various workarounds to get where I want to go. (Another, less serious isssue: what usability genius told you that they wanted the reviewing pane to hold not only comments, but every single trivial edit? Why isn’t there a way to view only comments in the reviewing pane, without having to change the view of the main text to Final, thus hiding the markup to which the comments often apply?)

  7. Richard says:

    How much wieght is put on to how new users work with the apps vs. experienced users?

    While everyone starts a beginner, consistent users are not going to stay that way for long.

  8. Charles says:

    For that matter, how much is done with people of different age groups, abilities, and equipment?

    I use a pointing tablet in place of a mouse, and I would really like the things that require accurate pointing to grab something (table cell borders in Word, cell corners in Excel) prior to a drag to have wider ranges of proximity in order to trigger the grab. Perhaps this would be a preference.

  9. There are many ways we collect feedback about the new UI. I’ve detailed some of the ongoing usability…

Skip to main content