Quality Is Usability

We have a number of ongoing, long-term projects designed to help us test the
overall usability and learning curve of the new Office 12 user interface.

One of the most important tasks has been a full deployment of Office 12 Beta 1
within a local company.  We chose a dozen people with all different job
types (but none of them computer or software related) and replaced their Office
2003 with Office 12 Beta 1.

On the surface, this doesn’t sound that novel.  Obviously, if you want to
find out how well a product works, you need to get it in front of people. 
What is unusual is how early we started this study.

If you’ve been in an Office beta program, you know that Beta 1 is typically
extremely rough.  It’s designed to give people a glimpse of where the
technology is headed but not really suitable for daily work unless you’re
willing to put up with a lot of bugs.  We typically do not get the quality
of Beta 1 far enough along so that we would feel comfortable with "normal"
people using it.  It’s designed for our hard-core MVPs, partners, and
technology enthusiasts.

That said, we knew that if a long-term, longitudinal study with real people was
going to impact the product, it had to happen early enough for the feedback to
be acted in the product.  By the time Beta 2 is out, we’re mostly focusing
on quality, performance, and fit-and-finish issues and we don’t typically
revisit the design of a major component of the product.

Thus, in early November, an unsuspecting group of volunteers at a local company
got their rock stable, menus-based Office 2003 replaced with Office 12 Beta 1.

We’ve learned at enormous amount from this study, and I’ll be sharing many of
the results with you both anecdotally and statistically in the months ahead. 
The overall feedback has been very positive and, in the places there are have
been problems, the feedback has been totally actionable.  We have made many
improvements to the product to address feedback garnered from all sources about
Beta 1.

The single most interesting takeaway for me, personally, has been seeing the
incredibly strong link between the quality and the usability of the product. 
Normal people simply can’t tell the difference between something that’s broken
and something that was designed poorly.  Most of the "usability" feedback
we get is actually bug reporting.

When the quality sucks, the usability sucks.  And it’s not just the obvious
things like crashes and data loss; the fit-and-finish ultimately makes or breaks
the usability.  When a menu disappears too quickly, or the selection
handles aren’t intuitive, or when a feature doesn’t work in a certain
scenario–all of these may be accounted for as bugs which will be fixed in a
later version, but these things will add up and taint your usability numbers.

One of the hardest parts of synthesizing all of the usability feedback we get is
knowing when to write off things as "probably fixed in a later build" without
neglecting real issues.

Comments (12)

  1. blunose says:

    Sorry this is a bit off topic but I’ve been pitching this idea to every windows UI guy I can. What are your thoughts on modal dialogs?

    I am of the opinion that modal dialogs should always be modal with respect to their parent window only. This is why MDI was broken all along. If you wanted a modal dialog you would prevent document switching. If I had 2 unrelated docs in word open, and I tried to save one as the other I could not edit either document.

    Mac OSX already does this. Modal on a parent window also would allow you to minimize the parent window without having to resourt to the ugly minimize all hack wich is the ‘Show Desktop’ button on the windows taskbar.

  2. ChrisC says:

    bluenose wrote: "Modal on a parent window also would allow you to minimize the parent window without …"

    1) Uh no, I don’t think it would.

    By definition the user must deal with the modal form before the parent responds to anything (dispite the results of the OS fn ‘minimize all’).

    2) I don’t think it should

    Where should the line be drawn? Can they maximize too? If they min or max can they then do a restore to see the screen again?

    I think a form which displays a modal form should basically only be expected to respond to WM_PAINT events (i.e. redraw itself after something moves over it or is drug across it)

    Not only is it off topic, but I doubt it would be Jensen’s area (directly at least).

    See the blog "The Old New Thing" to get some really cool history of how windows came to be what it is… maybe you’ll figure out who to talk to about changing to the ‘new modal’ (I’m guessing the windows team itself)


    -Chris C.

    "The Old New Thing" @ http://blogs.msdn.com/oldnewthing/

    And there are 1,045 posts there as of today so it might take you a while – if you don’t care about how to code it, this will go a lot faster. To speed this up try google-ing with a site-specific search (to do this add "site:http://blogs.msdn.com/oldnewthing/" [without the dbl quotes] to the END of your keyword list)

  3. blunose says:

    I apologize for sounding somewhat incoherent. This can happen when you think you’ve got a great idea :). I read The Old New thing as well, and I am perfectly aware of the intricacies of changing the windows UI model at this stage in the game.

    To Chris C. :

    What are your specific objections to allowing minimization and maximization and ‘gasp’ restoring of the parent window? I am coder so don’t be afraid to give examples.

  4. ChrisC says:

    To blunose: "What are your specific objections"

    Example #1: Resize & Maximize

    If there are controls on the form which respond to the resize event then you have a case where code in the app would be fired off in response to a user action – dispite the (correct) architecture assumtion that the modal dialog must be responded to before this happens.

    Example 2: In general (Opening a file)

    Once the "Open" command has been selected, users have no choice but to select a file for opening (or to cancel the operation). While attempting to open a document, the user cannot interact with the existing document (for example, scroll it around to look for some text that would give a clue as to what file to open next).

    But hey the user might want/need to scroll around, right? Why not let them?

    If you let them scroll around, what do you do when they decide to save it*? Save is also a modal operation – but you already have a modal operation (Open) in progress!

    (* = Assumes that the app saves the current view of what the user was doing and always re-opens to that setting; otherwise there’d be nothing to save from just scrolling around 🙂

    My $0.02 = I can code around a large number of things, and will deal with this too if MS changes it… I just think that the concept isn’t broken, so they should leave it as it has been for years.

    (I was really unhappy with the ‘minimize all’. When introduced it broke a business app I was responsible for – of course it shouldn’t have, but the UI design was crap, which was my manager’s fault, which meant it was my fault, even though it wasn’t. Been there yet? 🙂

    Email me at earthlink dot net if examples 1&2 don’t do it for you or don’t make sense. (I don’t want to annoy the people subscribed to this blog who aren’t coders 😉

    -Chris C. (jccompton)

  5. Dan McCarty says:

    "the feedback has been totally actionable"

    actionable: adj. Giving cause for legal action

    So, in Microsoft-speak, what does "actionable" mean? 😉

  6. Centaur says:

    Speaking of usability…

    A few days ago, a friend had a question about Track Changes in Word. Specifically, she pointed out that Word assigns different colors to changes made by different authors, and that the first author invariably gets the red color. She complained that it was quite inconvenient to author documents while writing in red.

    I consider myself an expert Word user, but I could not suggest any halfway decent solution. I learned that the color map is not customizable through Word UI. I wasn’t even able to find the place in Word binaries where it stores the color table.

    In my opinion, Track Changes users would benefit from being able to customize the map from authors to colors, on a per-user basis (that is, in a group of three users A, B and C, A could see C’s changes in red, while B could see the same C’s changes in green, and would not have to agree on a common color coding).

  7. ChrisC says:

    Centaur wrote:

    "… the first author invariably gets the red color. She complained that it was quite inconvenient…"

    At first it didn’t surprise me that this was true.

    Then I thought about color blindness and now I am surprised it isn’t customizable in some way through some means. (have you looked with this in mind, maybe it is in a weird area?)

    I would think the user would be able to pick the color of author 1, author 2, etc.

    The by-name idea you seem to imply/suggest, would add too cost to the UI testing+QA of adding a feature most people don’t know exists.

    (In fact I’ve received docs with small stray pieces of markup in it and when I ask the sender complains they can’t get rid of it)


    -Chris C.

  8. MSDNArchive says:

    "Actionable" in Microsoft-ese means "We can do something about it" or "fixable".

  9. Centaur says:

    Chris C.:

    > I would think the user would be able to pick the

    > color of author 1, author 2, etc.

    I would, too. However, the user only gets to pick a color for insertions, deletions, formatting changes, and comments. For each of these, one can pick a fixed color, or have them allocated depending on the author, with first author being red, second being blue, and so on. So, either all insertions by different authors are the same color, or the majority of the document body is red.

    As for complexity, Excel has a default color map for charts, and it is customizable. The same code could hopefully be reused in Word.

    Speaking of features people don’t know exist, well, that’s 90% of all Word features, and most people use Word as they would use Notepad, or Write, or, in many cases, a mechanical typewriter (case in point: aligning text with spaces, or emphasis using CAPS (not the All caps style but actual capital letters)). Ain’t it sad…

  10. Alex says:

    Please have a look — a Word screenshot. 🙂


  11. Today’s Guest Writer: Rich Grutzmacher

    Rich Grutzmacher is a Program Manager on the Office User Experience…