Real People Doing Real Work with Office 2007 (Part 1)

Today’s Guest Writer: Rich Grutzmacher

Today I present guest writer Rich Grutzmacher, Program Manager on the Office User Experience team. He helped coordinate one of the long-term, real-world studies conducted on early Office 2007 builds.

For eight months, we followed the progress of a group of twelve people in a Fortune 500 company here in the Pacific Northwest who used Office 2007 to perform their daily work. These people graciously agreed to install Office 2007 Beta 1 on their main work computers and to let us track their feedback and thoughts over a long period of time using the software.

The participants in the study were not software engineers or in any way associated with high-tech; they were just normal people who use Office to help get their jobs done.

During this extended usage study, we personally interviewed and videotaped each participant every 2 to 3 weeks to understand how they were using Office 2007 to perform their everyday work tasks. The participants in the study also sent us feedback using Send a Smile whenever they encountered a problem or discovered something about the product that they liked.

Given the issues one encounters while using early beta software, we considered nominating these participants for Sainthood. Their willingness to endure the pain of using a beta product for an extended period of time to perform meaningful work provided us with valuable insights in how typical Office customers learn and use the Office 2007 user interface.

In this series of articles, I will share with you some of the lessons we learned from this study and how the information was used in combination with many other sources of customer research to improve Office 2007 before Beta 2 got out the door.

Now before you jump out of your chair and scream, “ONLY TWELVE USERS,” let me explain a bit more about the purpose of this study.

Yes, the sample size for this specific study was small. And yes, the study only included people in one company in the Pacific Northwest. It is true that for these reasons the results of this study aren’t statistically significant.

But the purpose of this study wasn’t to provide quantitative results. Rather, the goal was to employ ethnographic research strategies to allow us to understand qualitatively how the new Office 2007 user interface fit into daily work lives of typical Office customers.

There are several factors which contribute to the overall Office user experience which cannot be accurately measured through satisfaction surveys or through two-hour structured studies in our usability labs.

A big part of our research strategy for the new user interface is the concept of triangulation: collecting data and feedback from many sources, with many different kinds of studies, and with widely varying sample sizes and time periods. This particular study was designed to watch how Beta 1 fared not just in the first hour or the first week, but over months and months of use.

We went to great efforts in selecting participants for this study who encompass a range of characteristics found in Office customers and who use Office applications to perform a broad range of work-related tasks. For instance, we wanted some people who mostly used Word, and others who lived in Excel or PowerPoint. We looked for people with varying proficiency with Office and people in different kinds of jobs.

This selection process, combined with the high-touch methodologies we used, gave us an incredibly valuable insight into how the user interface changes would fare in the real world–eight months before we would have usually even started to gather this kind of feedback. And because we got the feedback early, we were able to address deficiencies in the product and then measure the impact of the improvements in future betas.

Next time: Thoughts about training and migration tools from the study.

Comments (13)

  1. Francis says:

    It would be interesting if you had provided all users with Office 2007 AND 2003 and noted when they reverted to the latter. What tasks was the older user interface more efficient for?

    Beta 2 is better for some activities but worse for others.

  2. randuze says:

    Would that be a true test though?  I’m willing to bet that, most of the time, the average user would revert to the old UI out of familiarity rather than anything to do with efficiency.

    For example, a number of our users initially turned off "personalized menus" in Office, WinXP, and "personalized favorites" in IE because it was so unfamiliar to them.  Later they realized how efficient it was to not have to scroll through commands/favorites they rarely used.

  3. Francis says:

    It depends on whether users are willing to learn or give up without trying to.

    In Access 2003, I toggle between Design and Datasheet View with the keyboard, but in Access 2007, I use the mouse. Without comparative analysis, nobody would notice this difference, let alone pose the question "why?"

    Is it because I have not learnt the new keyboard shortcuts (familiarity) or because the new shortcuts are poor (inefficiency)?

    In this case, it is the latter. (ALT+V, ENTER vs. ALT+H, Y, 1, H)

  4. danruze says:

    uhhh…no one realized that randuze.  Personalized menus are widely considered a bad call (been discussed on this blog).  No one turned them back on.  People realized how inefficient it was to have to re-read and learn the menu everytime something on it changed.

  5. Mike Dunn says:

    While not pleasing to statisticians, 12 users isn’t small for a usability study. It’s been my experience that you’ll start seeing patterns of usability problems after testing with just 6 people.

  6. randuze says:

    Hmmm, they didn’t? Odd… I wonder why so many of our users said so in our post-deployment.  🙂  

    What I was trying to say is should learning curve be a factor in measuring the efficiency of new ways to perform common tasks?

  7. GoodThings2Life says:

    I agree with Mike Dunn… I’ve recently deployed a new web site tool for the college I work at, and with only a small handful of users I quickly discovered things I needed to change before moving forward with it. The more users I’ve added to it, the less productive feedback I’ve received because they all said the same things. Granted, I’m not complaining… it means fewer things to resolve! 🙂

  8. GoodThings2Life says:


    That’s certainly a true statement that given an option, most people do revert to what is familiar rather than what is efficient or effective. Many people do, in fact, dislike change of any form and will always prefer older ways of doing things even if those ways are archaic and outdated. However, that’s a problem in itself, since allowing such complexity in a test would skew results.

    I’ve been using Word since 4.0, and each version I’ve had to relearn something, and I will often catch myself (even today) trying to use a method that– like the double click of the top-left menu to exit– no longer exists. I’m still learning where things are in Office 2007, but overall, once I find things, I realize they are much MUCH easier to use now than before.

    Jensen has mentioned time and time again that it’s not just about making the interface more efficient, but also about making it more intuitive and making features more discoverable. It is something they have done well in doing.

  9. Francis says:

    Older ways are not always "archaic and outdated," e.g. when car manufacturers ordained that headlights should be controlled by a tiny stalk on the steering column (replete with 4 buttons, 8 dimensions of motion, and 12 iconettes) instead of a simple knob on the dashboard.

    In the case of Office 2007, I do agree about the interface being easier to use–it’s the knob! (as long as you are mouse user.)

  10. Stefan KZVB says:

    Please write also how users’ Autotext entries are converted from older Word versions and what to tell them on the new mechanism provided for that functionality in Word 2007. Thanks!

  11. Today’s Guest Writer: Rich Grutzmacher
    Rich Grutzmacher is a Program Manager on the Office User Experience…