API usability and the cognitive dimensions framework

Brad Abrams posted a
slide deck that I use to talk about the cognitive dimensions framework which we use
at Microsoft to evaluate and describe API usability. I thought I would take this space
to provide a little more explanation about how we use the framework.

The cognitive dimensions framework presents a set of dimensions that describe aspects
of an API that impact its usability. For example, one of the dimensions is 'Abstraction
Level' which describes the types of abstractions that the API exposes. Another
dimension is 'Working Framework' which describes the mental load that the API places
on its users. An API may be found to be unusable because the abstractions it exposes
are too low level. Or it might place too much of a mental load on its users, requiring
them to keep track of multiple objects in their head as they write code to perform
a task. Or it could suffer from a combination of both problems, exposing multiple
primitives that the developer has to keep track of in their head as they write code
to accomplish a given task.

By evaluating an API with respect to each of these dimensions, you can form a picture
of what the API looks like overall with respect to the framework. We like to create
pretty graphs to do this. For example, here is a graph of an evaluation of some fictional
API:

 

The black dotted line in the graph represents the API being evaluated. The spokes
in the 'wheel' represent each of the different dimensions. End points (the middle
and outer edges) on each spoke represent the end points on the scale for each
dimension. Thus the point on each spoke where the black dotted line crosses it
corresponds to the point on the scale for that dimension that the API has been valued
at.

The most common way that we use to perform such an evaluation is to perform a usability
study in our usability labs here at Microsoft. We invite developers from outside of
Microsoft to come in to the labs and use an API on a given set of tasks, over a number
of hours. We observe each developer as they work from behind a one way mirror and
make notes about what parts of the API work well and what parts don't work well. We
usually bring at least five developers in for at least two hours each for every study
we do. So after ten hours worth of watching developers write code against the API,
we spend time reviewing and analysing the data we collected and producing our evaluation
of the API with respect to the cognitive dimensions framework.

However, there really isn't much point in going to all this bother if all we end up
with is a graph like that shown above. It's not very helpful to provide the team developing
the API with a graph showing how their API fits on the cognitive dimensions - there
is nothing on that graph to tell them whether or not the API was judged usable or
not. There is no baseline with which to make a comparison. That is where the individual
developer personas come in to play.

We have defined what the three developer personas (Opportunistic, Pragmatic and Systematic)
would judge to be the ideal API with respect to the cognitive dimensions framework. In
other words, we have created profiles for each of the personas that describe
for each dimension, the ideal point on the scale for that persona. The personas define
quite different working styles so the profiles have ended up being quite unique. Thus
we now have a way to judge whether or not an API is usable by comparing its evaluation
with the profile we produced for the developer persona that best matches the target
audience for that API. We can do this comparison in the same graph:

 

The blue line on the graph represents the profile of a particular developer persona.
For example, it shows that this persona prefers APIs that expose aggregates (the outer
edge on the abstraction level scale) and APIs that have small working frameworks (the
inner edge, or middle, of the working framework scale). If you compare the points
where the blue lines cross each spoke with the point where the black lime crosses
each spoke, you get an idea for how well the API matches the persona's ideal API.
Hopefully you can see that this particular API doesn't really hit the mark for this
persona (there are big differences between the points where both lines cross on most,
if not all, of the spokes in the graph).

Fundamentally the cognitive dimensions framework provides developers with a common
vocabulary with which to talk about and discuss API usability. The benefits of such
a common vocabulary are twofold. First, the language shapes the distinctions that
are considered important to attend to. By explicitly describing each of the dimensions,
it is more likely that they will be attended to. Secondly, the language allows developers
to discuss issues using terminology that they can assume will be understood by others.

We've found that the cognitive dimensions have really helped us get to the root cause
of issues that we have observed in the usability labs. For example, in one study,
we observed lots of developers spending a large amount of time in the help docs looking
for code samples that would show them how to accomplish a given task. The first interpretation
of this data was simply 'Fix the help docs!'. However, when we used the cognitive
dimensions framework to describe the issues, it became clear that the reason the developers
weren't successful when they were searching through the help was because what they
were looking for in the help simply didn't exist. The API they were working with exposed
a set of abstractions that were at the wrong level for these particular developers.
They expected a particular type of abstraction to be exposed by the API but since
it wasn't, they couldn't find anything about it in the help docs. As a result of
this, the API team redesigned this API to expose abstractions more in line with what
developers were expecting. When we retested the API, it worked much better.

The dimensions in the framework have been adapted from decades worth of work done
at the University of Cambridge on programming language usability. So far we've used
the framework successfully around the company to evaluate APIs and to help to design
usable APIs. It's still evolving though, so it would be great to hear your
comments and feedback on this. I'd love to know about other frameworks or methods
for evaluating and describing API usability also.