Measuring Results

A couple of us on the UI team were having a conversation about the comments to

yesterday's post
.  Someone pointed out that it appears we've discovered
a community on the internet of people just as obsessed about UI as we are around
here. 🙂

I wanted to build on yesterday's conversation by helping you to understand more about how
we measure results.

When you boil it down, we have a pretty straightforward set of high-level
goals for the Office 12 UI redesign.  Help people use more of the
functionality in Office.  Help people create better-looking, richer
documents.  Save people time in doing the tasks they frequently do. 
Make sure people can be productive right away using the new version.  Help
normal people get results that only power users could get before.  Give
power users the a richer set of powerful tools to go beyond what was possible
before.  Stuff like that.

So, how do you know when a feature is right?  There are so many tools at
our disposal.

The first line of defense is the people who work on the product.  I install a daily build of Office
literally every day.  As soon as a new piece of functionality is in, you
have to try it out and get that visceral feeling about "does it feel right?" 
Sometimes, no matter how good your spec seemed or your prototypes were, the
first time you play with it you know it feels wrong.  You never get a
second chance to form that first, instant impression of "is it right or not" and
I put a lot of value in my initial impression.

Of course, there are way more people than me using interim builds.  And
included in that set are the crustiest elite power-users in the world: Microsoft
employees.  You would think that Microsoft employees would be open to
change and sympathetic about "work in progress", but some of the most advanced
Office users in the world work here, and if you get in the way of their
productivity, you're going to hear about it.  These people represent the
upper crust of people who use a cross-section of just about every feature in
Office.  So they provide a constant stream of opinions representing people
who are already experts in using previous versions of Office.

We do usability tests, as I've talked about many times.  The test subjects
range literally from people who have no experience with productivity software at
all to experts who make a living writing Office add-ins.  We do tests in
Redmond, of course, but we also do remote testing in sites all around the United
States and in our labs in Europe and Asia.

A common misconception is that a usability test is all about data--that we
receive a 100-page report full of graphs and tables and we average the data and
it makes design decisions for us.

It's not actually the raw data though that makes usability so compelling. 
Most of the time, it's the "a-ha" moment you have in watching someone with a
different background and way of thinking from you use the software.  Often
times, within the first 5 minutes you can see that you've failed and you don't
need a sheet of data to tell you that.  It's a humbling experience to sit
and watch people struggle.  And your job as a PM is to figure out
why and to fix it.

Yesterday's story about "Eat Dismiss Clicks" is an
example of this.  One can argue the theoretical implications of focus
issues until the cows come home, but watching people all around the world, of
all different skill levels, fail again and again in the same way tells you the
design is wrong.  When you repair the design and then see the same
diversity of people succeed at the same tasks, you know you've done the right
thing.  It's not about some computer spitting out data, it's about watching
the experiences people have interacting with the software.  Watching their
faces, hearing what they have to say.

Data from usability comes into play when answering questions like "what features
are the hardest to find" or "where does it take people longer to do something
than it used to."  For instance, we can benchmark how long it takes people
to make particular kinds of documents and see where people do great and where
they struggle.  We can then take a more in-depth look to see why
people are struggling in certain areas and make improvements.

The next source of feedback are all of the many people who have been using the
product over the last six months.  We have MVPs and a program of technical
professionals who have been giving us feedback during all this time.  For
the last three months, we've been receiving feedback from thousands of beta
testers.  We have rollouts of Office 12 in businesses in three continents,
having people use it every day to get their job done and telling us what works and what
doesn't.  Having thousands of vocal Office users providing a constant
stream of feedback gives you a good idea of
what people like and where parts of the product need some more thought or some
more work.

There's all of you here on the UI blog as well, and people writing about Office
12 all over the internet.  We read the things you write, noting what you
like and the questions you have.

And of course, the last piece--yes, we do have data.  Through the
Experience Improvement Program
, we can look at aggregated statistics about what
features people use, how they use them (keyboard, mouse), when they tab switch
in the Ribbon, and a lot of other things.  This provides a general
"heartbeat" of how the overall project is doing and complements all of the
anecdotal feedback we get from people using the product.

So, how do we measure results against our high-level goals?  We have to
synthesize all of those inputs.  Yes, there is a ton of data.  We get
bushels of anecdotal opinions.  We talk to partners and beta users. 
We watch people use the software here and in their place of work.  We watch
our parents use it.  We watch our children use it.  We talk to people
at the grocery store, or on the airplane.  We look at what you write here. 
And we try to stay true to our design tenets around simplicity, efficiency,
predictability, respect for screen real-estate...

All of these things combine to provide the true picture of how we measure success.

Comments (23)
  1. Sendell says:

    Questions regarding Office UI <-> Windows UI:

    1)Does the Windows UI team have more/less test data to work with?

    2)Is it fair to assume that something successfull in Office will also be successfull in Windows? Did you ever had contradicting data? examples maybe 🙂

    3)Do you (or anyone else of the office UI team) take any active steps to convince the Windows UI team to implement of some interesting conclusions (like "Eat Dismiss Clicks", technical problems aside).


  2. Dan McCarty says:

    FWIW, I’ve always thought that the focus model that dismisses a list or dialog AND does something back in the parent app is wrong. Kudos to you guys for recognizing what needed to be done and doing it. Working around the Windows focus model isn’t easy.

  3. Fox Cutter says:

    As luck would have it I’m currently at MS so I get to play with the beta. The first thing I tried was the focus issue, and it worked exactly the way I expected it to. It was great. In fact I really like all the changes, even the options have been simplified, in fact the only thing I don’t like is that the minibar comes up when you right click, but I found how to turn it off.

    It’s odd coming into Word and seeing how much it has changed, then discovering that everything just seems to work. You’ve obviously put the usability data to good use, as everything seems to be easy to get to and right where I need it.

    This is probably the first time I’ve used Office, and it really just worked. I’m just total impressed.

  4. Christopher Hill says:

    Is there any way for ‘normal’ users to give feedback directly on the Office UI? For example, I discovered today a little discrepancy – in Word, Excel and PowerPoint 2003 there is a useful feature called ‘Compress Pictures’ which can be found in the ‘Format Picture’ dialog box, and on the Picture toolbar.

    The feature was added recently to Publisher 2003, in Service Pack 1, but unfortunately the button was only added to the Picture toolbar and not to the ‘Format Picture’ dialog box as well. This caused confusion for my users. It’s small things like that which I used to report to, but this email address no longer functions…

  5. jensenh says:


    For Office, visit this page: and click Make a Suggestion. They are directly reviewed by the product team.

  6. Simon Jones says:

    One thing I have found, using Beta 1, is that some commands really need to be repeated on all the tabs of a ribbon.

    For instance in Outlook email composer, Signature is on the Insert tab but you have to switch back to Write to find the Send command which is usually the very next thing you need. You end up switching tabs, inserting a signature and then having to immediately switch tabs back to where you came from to send the email. You’ve effectively doubled the number of clicks to insert a signature, a common task, from two to four. (Previously = Signature, this sig | Office 12 = Insert, Signature, this sig, Write)

    Send (or Save & Close for other item types) really ought to be at the far left of _every_ tab because they are needed so often.

  7. ajr says:

    "the crustiest elite power-users in the world: Microsoft employees."

    "the most advanced Office users in the world work here"

    "These people represent the upper crust of people who use [..] Office."

    Ah Ah Ah

    that was a good one, Jensenh.

    honestly do you beleieve that ?

    ok now i understand.


    /in my opinion/

    this is where your problems is

    Get out of there, Jensenh. take a walk around the world, go see office being used outside the cosy redmond facilities. Get out of America for that matter.

    Go see how people whom think differently then you do think about office, go see how they use it.

    Don’t just have them report to you Go there, your self, take a look over your shoulders.

    (that is exactly what *teachers* do all day long, which is why you’re missing a very big portion of your users)

    You may very well be suprised.

    Actually , as a ten years + teacher , I can assure you you will be suprised.

    No wonder you guys have such narrow-minded point of vu.

    Now wonder the look I try to bring to you seems out-of-this-world : your world is soooo narrow.

    You are narrowing your Field of Though by posing at the very begining of your thought process, "I am the best"

    Don’t just do tests, you very well know that measuring something changes the value of the subject your are measuring (basic science statement). Go into buildings where people actually are working their daily shift with office. don’t give them exercises to do. Just watch them.

    You didn’t stop to rethink your (thought) processes, you meerly stop to assert what your are doing and justify it with very funy statement as "the crustiest elite power-users in the world: Microsoft employees."

    If you think of yourself as elite, then whom can actualy tell you "NO YOU’RE WRONG" ?

    When does the fact leaves your sight for your ego?

    When you talk about yesterday’s story, you actually make exactly the same mistake again.

    You don’t even consider your are using wrong measurement tools, wrong decision processes.

    You just state blatently "they’re the right tools" without even trying / considering other options (how more narrow can such process be?).

    You justify your position /without/ even trying to dig "outside of your box".

    let me take an exemple:

    "we can benchmark how long it takes people to make particular kinds of documents and see where people do great and where they struggle."

    Just *who on earth are you* to dictate to the world thy shall make this or this kind of document?

    This is like GM declaring: we’ve measured that people mostly turn to the right. So from now on, you can only turn right. Should you need for any reason to turn left, well, just turn right three times.

    Of course in the car business GM is not the only maker, so he cannot say that. In the office business (unfortunately for the users all around the world whom do not think like you guys) there is just you, and your narrow thinking.

    "All of these things combine to provide the true picture of how we measure success"

    oh yes I agree completely: it does indeed show "the true picture of how we measure".

    It also *does not* proove it to be right, accurate, pertinent.

    In fact it prooves it to be self-indulging, quite pompous, and very pretentious.

    You actulaly think you have a vision of all word users out there, where there are people whom do not agree with your decisions.

    but hey, you microsoft guys are the elite.

    Thanks for the goof, jensenh.

    There are many thing upon which you and your team are elite, that’s for sure and obvious.

    But there are also many things upon which you know nothing. You so much know nothing that you are incapable (thus do not have the capacity) to even recognize that you don’t know.

    It’s sad realy.

    /in my opinion/

  8. Stig says:

    /argh.. delete previous comment 🙂 /

    It is actually very good to see that you guys perform all these tests. O12 will quite certain be a great step forward.

    Keep up the good work and keep the posts comming.

    As to comment to AJR, the previous poster, did you actually read the whole thing or was you put of by the power user statement? Their internal users the first line of testing before proceeding with other internal and external tests.

  9. ajr says:

    yup i did read the whole thing.

    from Jensenh:

    "Soon after you install Office 2003 on your computer, a balloon pops up asking if you would like to "Help Make Office Better." If you click on it, you are given the opportunity to enroll in something called the Microsoft Office Customer Experience Improvement Program. If you opt-in, anonymous data about how you use Office are uploaded to Microsoft occasionally in the background.

    If you’re the curious type, you might have wondered where your data goes. Well, today I’m here to answer the question: it goes into an Excel spreadsheet I have sitting on my desktop."

    /in my opinion/

    How ever the sheet is fed, not ever will an excel file give it’s reader a clear vu on customer experience.

    My point is that how ever elaborate the tools put in place are, it will never be as efficient as going to watch a user over his shoulder do his every day work. *never*

    if you give exercises to users to see how they do this or that , if you ask a user to use this or that tool, you are corrupting the tests.

    This is logicaly obvious.

    Nothing can replace RealWorld Experience. That’s why it is called Real World.

    the work of statisticians is to try and replace RealWorld by models. but these models are never accurate, he whom thinks they are, looses the ability to be pertinent.

    I mean Jensenh is actually saying "I have the world user experience in an excel file sitting on my deskop"

    It’s funny to read but frightening to know that Jensenh is actualy conveinced so.

    It’s easy to see what mistakes can be made from such an assumption

    ("we’ve measured that most people turn right")

    I ask that Jensenh releases himself from those assumptions. That he steps aside from the realworld as he sees it, to discover the RealWorld as seen by every individual users.

    Kill the Clipper campagne was a violent way of the users telling the developpers "you may found that cute and funny, but we find it enoying and disruptive". That campagne showed the enormous gap between what dev think the users think, and what users actualy think.

    When Jensenh sets off to make word as usable to newbies as to power users, he makes assumptions (about their previous knowledge), but does not verify them.

    The very basis of his work (on that part) is flawed.

    But futhermore, he doesn’t (at least not publicly) agree that he *could* be wrong in his assumptions.

    I aggree for having worked with developpers that one cannot and must not second guess his work every step of the way. BUT ALSO one must always consider the possibilities that go against the basis on which he worked.

    I am not challenging the decisions themselves, I am challenging the hypotesis (eg: users need the click to be eaten) AND i am also chalenging the measure tools used.


    Why do i chalenge the way Jensenh works?

    Who am i to challenge that?

    I’m a teacher.

    I teach word usage to powerusers as well as to newbie.

    Last year I tought hundreds of ours to hundreds of users.

    I know how much each and every single one of them *think* differently.


    thinking and acting are two different things:

    user A may open a menu because he think it will bare the option alpha-gama he needs


    user B may open the same menu because he doesn’t now where is the option and what option he needs to accomplish whatevr task

    Both user need to dismiss the menu, but they think differently


    With my experience and pedagogy, I am able to see pass that chaos of multiple users and see a certain order.

    That order comes from what is commonly called "the look and feel" of the application , married to the cognitive approach users have that look’n’feel

    The objective of the "look and feel" of the application should work at that, build a certain order uponwhich users can rely to do their chaotic tasks (since no two users do the same thing, despite whatever that excel sheet says)

    That order is what spawns my pedagogy.

    That order is what *should* /in my opinion/ spawns Jensenh’s decisions.

    So yes from my little tiny area of expertise, far away from jensenh’s, I challeng his methodologies.

    as to answer the why question:

    Challenge is good.

    It enlarges our field of thought, of perceptions.

    That helps the both of us to think differently, thus widen our approach, thus get a little closer to "the Real World".

    Reading Jensenh’s approach to what he think of users, helps me craft even more my pedagogy, I only wish to return the favor.

    It’s respect, and sharing.

    because I honestly believe that knowledge has value only when shared.

  10. Ti says:

    I give support and appreciation to Jensen.

    Air, I feel you may be not reading his posts carefully. Jensen uses various ways to collect the data, and that’s only one of the many ways. He mentioned that again on this very post.

    I am just glad that because of him, we all benefit from knowing the great improvements and there are so much being done to to improve it. I have no doubts that this version will be great as long as your team keeps up the good work and keep the bugs low.

    Like he said, there are something you need to feel it first. May be we should give him a chance until we try out to use it and then comment on it further?

  11. jensenh says:


    The Outlook tabs have been substantially redesigned since beta 1 to address many of the issues you bring up.

    In the meantime, you can always add the Send button to the Quick Access Toolbar so that it is always available.

  12. Simon Jones says:

    Thanks. I’m sure Beta 2 will be even better than Beta 1. I’m looking forward to seeing it.

    Yes, I’ve added stuff to the QAT in Word, Excel & Outlook (I don’t use PowerPoint or Access that much).

    Simon Jones

  13. a jar says:

    ajr: I would absolutely, totally and utterly hate to have you for a teacher – you seem to be completely unable of listening (or reading, in this case), instead you keep babbling some obscenely long self-righteous claims long after everyone stopped caring.

    Jensen: sorry

  14. sitoso says:

    ajr: you lost me at "Ah Ah Ah".

    jensenh: I love the new UI, I can’t imagine going back to pre Office 12 UI now. And I think the "Eat Dismiss Click" is a good idea, having tried it out a few times myself since your last post 🙂 I didn’t notice it "on my own", because I always click the down arrow again to dismiss the menu, but I know not everyone does – and everyone isn’t tought by ajr either.

  15. John Greenan says:

    Just one question. Typically I consult for investment banks. The usual desktop I work with has twin 3.0Ghz processors and 2GB of RAM. It’ll also have Bloomberg and Reuters datafeeds. Now, does anyone at Microsoft (“some of the most advanced Office users in the world work here”) have these datafeeds and test with them? I bet that there is no one fitting that bill. And yet the financial servies industry is surely one of the heaviest users of Excel. I bet not because if there was someone at Microsoft with those datafeeds you would sure as day follows night work on improving the RTD implementation, which is (with due apologies to Tim Getsch and others) not really enterprise class.

    I’ve actually raise questions about this in the past with Tim and placed requests on the MSDN forums. Maybe this is not a big deal, but RTD could be awesome, instead it’s just ok.

  16. MSDNArchive says:

    While I think all this discussion about whether Microsoft employees are elite Office power-users or not is just trolling, I’ll add a nice "proof by contradiction". To start, I don’t claim that any one person at Microsoft uses a specific feature more often or more extensively than anyone outside of Microsoft. That might make you a power-user of that features.

    Do you think there’s a feature in Office that you know about that no one at Microsoft does? I doubt it. Do you think there are features that the collective Microsoft Office team knows about that you don’t? More than likely. And not only that, but the people at Microsoft know how those features work, what trade-offs were made when designing them, and what features are being added or deprecated in this version, and some of them know exactly where Office is going for the release after that.

  17. jensenh says:


    I know that the financial services companies are some of the most thorough users of Excel and, suitably, the Excel team spends a lot of time learning from people who work in these companies. In fact, it seems like they’re almost on-site every few weeks. And they’re very eager to pass on information about how the new UI impacts these financial services info workers (with lots of suggestions, many of which we have done.)

    If you have specific questions/comments, Dave Gainer’s Excel blog would be the place to talk to the product team directly.

  18. BruceCris says:

    AJR: I just read your comments here. You seem so jealous. Did you once get fired from Microsoft? As a usability/UX profesional I can tell you that Jenson is doing it the right way. He can’t be everywhere at everytime for god’s sake. Lighten up brother, Usability testing is a science, and the testing itself is an art. Knowing what questions to ask and how to discern the responses/observations is critical in making applications easier to use. JENSEN IS DOING IT RIGHT for the betterment of you. Lighten up man, you look like a ranting idiot. Good job Jenson, it sounds like you and your team are doing the right thing.

  19. Jeff says:

    Honestly, I think the guy has a point.  Because I’m interested in usability, I’ve kept a list of all the annoyances I’ve found in Word (which I use for approximately 75% of my job), and how I’d fix them if I had the chance.  (Some of the more obvious ones:  the absolute idiocy of a spellchecker that’s unaware of context, or the awful autocomplete that actually stops working as you type in more of the text)  I’ve never seen any of these acknowledged, much less addressed; instead, we get a toolbar (oh, sorry, "ribbon") borrowed from Netscape 8 and a bunch of "lipstick on a chicken" graphical cruft.  And the sense I get from this is that Microsoft has no idea how their software is being used.

Comments are closed.

Skip to main content