More Than Just the Two-Way Mirror

  • Article

I've
revealed in the past how many of the Office 12 UI decisions we make are
influenced by what we learn in collaboration with our usability and research
team.

Personally, I find usability tests scary.  You have an idea, discuss it
with others, work it into a prototype and get excited about it.  You want
it to work.  And then you pull back the curtain, put the idea in front of
real people and all of a sudden you know whether the idea works for real or not. 
It's totally humbling.

It's like when you're dieting--you can feel good about the food you eat or
exercise you do, but nothing brings it into focus like stepping on the scale and
facing the result.  If the result is bad, it's back to the drawing board. 
If the result is good, you set it aside, plan to retest, and start focusing on
where else you can improve.

Over the next weeks, I'm going to write in more detail about our usability and
research efforts around the Office 12 UI.  To give you a taste, here's a
snapshot of all the special research projects we have getting started over the next
month or so.  The following bullets are excerpted from a status e-mail
message written by Tim
Briggs, one of our usability leads.

  • Office 2003 Benchmark (10/19 - 10/20): 30+ people coming through
    the Office Design Lab, 10 at a time to complete tasks in previous versions. 
    We're collecting success/failure, time on task, and satisfaction so that we
    can compare to the Office 12 Benchmark later.

     

  • Eye Tracking 1 & 2 (10/19 & 11/11): Using the eye tracker to give
    us better understanding of how people use and browse the current visuals so
    we can then refine and study the new visuals.  We're looking especially
    at how people use the chunk titles, the contextual tabs, and the MiniBar.

     

  • Card Sort II (early Nov.): A year ago we had people organize
    commands into buckets to help us think about how to generate the tabs. 
    This time we'll give them our buckets and see how well they can sort the
    commands into them, especially while coming up with names for some really
    sticky ones.

     

  • Internal Longitudinal Study (Now - Dec.): We're getting people
    from the company set to install and start using both current builds and Beta
    1.  All are outside of Office, most are from non-product groups like
    Marketing, Sales, and Legal, and will not only be sending feedback but
    participating in the Office 2003 and Office 12 Benchmarks so we can compare.

     

  • The "Truman Show" (Now - Dec.): A brave soul from the "real"
    world has agreed to let us take away his Office 2000 and replace it with
    Office 12... and do site visits... and work here on campus for a while... and
    have conferences via Live Meeting with us... and let us analyze his personal
    instrumentation data... and send us a daily journal of his experience. 
    He's a local guy with a small personal business and excited
    about this challenge.  As are we.

     

  • Office 12 Benchmark (Nov. - Dec.): We'll look at the whole range
    of use from OOBE ("out of box experience") with newcomers to 2 weeks/2
    months use with the internal longitudinal people.  Like the Office 2003
    Benchmark, the tasks are selected to give us some key comparisons we can see
    in the big Office Design Lab data and more personally through 1:1 standard
    lab sessions.

     

  • Extended Usage Study (Now - RTM): We've started deep engagement
    with a large group of users from a local company.  We'll be rolling out
    Beta 1 to all of these people.  As this kicks off, it is the first
    major opportunity to closely monitor the rollout, training, adoption, and
    acceptance of the new UI over a long period of time.  We'll be doing
    persistent visits, monitoring instrumentation data, and collecting 1/1
    feedback.

     

  • Beta Survey & Visits (Dec. - Feb.): As part of the overall
    beta plan, we'll be adding to the benchmark data with a series of site
    visits to customers to observe first-hand "use in the wild." 
    Members of the design team will be partnered up with customers so that we
    can follow up more individually starting in December.  Well also be
    following up with surveys to drill into productivity and satisfaction.

     

  • This is just the known, scheduled research. There are lots of other
    efforts going on too like building up the list of instrumentation metrics,
    3rd party validation studies, continued "Send a Smile" tracking, continued
    consolidation of usability findings from Word, Excel, PowerPoint, Outlook,
    and Access, and oh so much more.

Jensen here again.  All of this is in addition to the normal ongoing
"behind two-way glass" usability tests that focus on individual mechanisms or
features.  And of course it's added to all of the beta feedback we get through newsgroups, customer
visits, e-mail (and even here in blog comments!)

As you can see, we have a lot going on as we strive to learn more and more
about the UI and how to continue improving it.  It's a large effort, but I
think it will be worth it.

I'll fill you in on the details of some of the more interesting studies in
the coming weeks.

Have a great weekend everyone!