HOWTO: Design and run an API usability study

A few people have asked me about how I design and run API usability studies. I'm running an Indigo study this week so I thought I would describe the steps that went into setting this study up.

The study is a follow up to the Indigo study we ran in October and which identified a number of usability issues, the biggest of which was the extensive use of attributes. Since then, the team redesigned parts of the API and were keen to get fresh usability data on the new design to see if the problems were resolved. To get the new study kicked off, I met with Steve Swartz (one of the Indigo architects) and David Palmann (a program manager on the Indigo team responsible for usability) in December 2004 so that they could describe the design changes to me.

We discussed the earliest date that we could run the follow up study. The implementation of the new API design wouldn't be complete until some time in February and then with the necessary testing that needs to take place, it looked likely that the earliest a stable build would be available wouldn't be until some time in March. We tentatively scheduled early March to run the study.

One of the first trade-offs to make in setting up a study is deciding when to do it. You need to make sure that you run the study with a version of the API that looks pretty much the way that the team have designed it but you don't want to wait until the API is completely implemented and tested since there probably won't be much time to respond to any changes suggested by the usability study. We thought that we would be able to secure a reasonable build of the new Indigo API by early March which would still give the team enough time to respond to any usability issues.

With a rough timeframe in mind, I drafted out a plan for the work that I would need to do to prepare and run the study. Typically the very first thing that I like to do is to get hold of a build of the API such that I can spend some time coding against the API to get a feel for the API and to identify any potential usability problems in the API. This build needn't be the exact same build that I will use in the study itself, just something that I can get up and running and playing with. At this point in time (late December), a build wasn't yet available so I had to put off being able to play with the API until some time later in January.

This would leave me with at most four weeks to play with the API and prepare the study if we were to run the study in early March. Normally this would concern me since one risk is that once I start playing with the API, I discover that it is pretty complex and that preparing the study materials is likely to take a significant amount of time. Since I am typically running/designing more than one study at a time it can be quite difficult to find the time to prepare the materials for a complex API when there are other studies going on.

In this case though I wasn't worried since I already had materials ready from the October study. I just needed to update those materials based on the new changes. The study materials that need to be prepared are basically an API introduction document that each participant gets before they come into the lab and the task list that participants work on when they are in the lab. The API introduction is a very important document - it gives participants enough of an intro to the API so that when they come in to the lab they have some idea of what to expect and what they will be working on. However, the intro needs to be tuned to the needs of the study. It's important that the intro gives participants enough information that they feel comfortable working on a task in the usability lab, but not so much that we just tell them what to do. Very often I end up writing this document myself from scratch based on my experiences using the API for a few weeks. I try to look back on my experiences using the API and try to document answers to questions that I had while getting to grips with the API. Quite often these will be things that I needed help from the API team to figure out.

We try to send the intro document to participants at least a week before they come into the lab so if I wanted to run the study in early March., I really needed to have a draft of the document complete by mid February so that it could be reviewed before being delivered to participants. I was able to reuse the document that was sent to participants for the October study and edit this to reflect the new API. In late January I spent an hour with Steve Swartz making changes to the original document.

At this point I still hadn't used the new API myself. I found that when I was working on the changes to the tutorial document that Steve had suggested there were still some niggling questions that remained which would have been easily answered had I been able to use the API and explore for myself. So I waited until a reasonably stable build was available for me to play with. In mid-February David was able to provide me with a Virtual PC image containing a build of the API that I could use. This helped me complete the edits to the intro doc so that it reflected the new API.

Around the same time, other projects I was working on required some attention. There were UI reviews for Visual Studio Team System and Joey Lawrance had just started his internship with us. He and I were working on a pretty big study that was to take place throughout February and this took higher priority. This meant that our original plan of running the study in early March wasn't feasible so we had to postpone the study for a couple of weeks. But I had a build of the API now so I could get at least think about getting started on the task list.

In my next posting I'll talk about how I go about designing tasks for an API usability study.