Test Has Lost Its Way

In a blog post, Brent Jensen relays a conversation he had with an executive mentor. In this conversation, his mentor told him that, "Test doesn’t understand the customer." When I read this, my initial reaction was the same as Brent’s: "No way!" If test is focused on one thing, it is our customer. Then as I thought about it more, my second reaction was also the same as Brent’s: "I agree. Test no longer cares about the customer." We care about correctness over quality.

Let me give an example. This example goes way back (probably to XP), but similar things happen even today. We do a lot of self-hosting at Microsoft. That means we use an early version of the software in our day-to-day work. Too often, I am involved in a conversation like the following.

Me: I can’t use the arrow keys to navigate a DVD menu in Windows Media Player.

Them: Yes you can. You just need to tab until the menu is selected.

Me: That works, but it takes (literally) 20 presses of the tab key and even then only a faint grey box indicates that I am in the right place.

Them: That’s what the spec says. It’s by design.

Me: It’s by bad design.

What happened here? I was trying out the feature from the standpoint of a user. It wasn’t very fit for my purposes because if I were disabled or my mouse was broken, I couldn’t navigate the menu on a DVD. The person I was interacting with was focused too much on the correctness of the feature and not enough about the context in which it was going to be used. Without paying attention to context, quality is impossible to determine.  Put another way, if I don’t understand how the behavior feels to the user, I can’t claim it is high quality.

How did we get to this point? The long version is in my post on the history of testing. Here is the condensed version. In the beginning, developers made software and then just threw them over the wall to testers. We had no specifications. We didn’t know how things were supposed to work. We thus had to just to try use the software as we imagined an end user would. This worked great for finding issues, but it didn’t scale. When we turned to test automation to scale the testing, this required us to better understand the expected cases. This increased the need for detailed specifications from which to drive our tests.

Instead of a process like this:


We ended up with a process like this:


As you can see in the second, the perspective of the user is quickly lost. Rather than verifying that the software meets the needs of the customer, test begins to verify that the software meets the specifications. If quality is indeed the fitness for a function then the needs of the user is necessary for any determination of quality. If the user’s needs could be completely captured in the specification, this might not be a problem, but in practice it is.

First, I have yet to see the specification that totally captures the perspective of the user. Rather, it is used only as an input to the specification. As changes happen, ambiguities are resolved, or the feature is scoped (reduced), the needs of the user are forgotten.

Second, it is impossible to test everything. A completely thorough specification would be nearly impossible to test in a reasonable amount of time. There would be too many combinations. Some would have to be prioritized, but without the user perspective, which ones?

Reality isn’t quite this bleak. The user is not totally forgotten. They come up often in the conversation, but they are not systematically present. They are easily forgotten for periods of time. The matching of fitness to function may come up for one question or even one feature, but not for another.

Test has lost its way. We started as the advocate for the user and have become the advocate for the specification. We too often blindly represent the specification over the needs of real people. Going back to our roots doesn’t solve the problem.

Even in the best case scenario, where the specification accounts for the needs of the user and the testers always keep the user forefront in their mind, the system breaks down. Who is this mythical user? It isn’t the tester. We colloquially call this the "98052" problem. 98052 is the zip code for Redmond, where Microsoft is located. The people that live there aren’t representative of most of the other zip codes in the country or the world.

Sometimes we create aggregated users called personas to help us think about other users. This works up to a point, but "Maggie" is not a real user. She doesn’t have real needs. I can’t ask her anything and I can’t put her in a user study. Instead Maggie represents tens of thousands of users, all with slightly different needs.

Going back to our roots with manual testing also brings back the scale problem. We really can’t go home. Home doesn’t exist any more. Is there a way forward that puts the users, real users, back at the center, but scales to the needs of a modern software process? I will tackle one possible solution in my next post.

Comments (8)

  1. Joel says:

    I've had countless conversations like the example cited. Thing is, If you get to the point where test is responsible for the software experience not totally sucking, you've already lost the battle because it implies that QA are the only people involved who actually care.

  2. Agreed, you can't test quality into the product, you have to build it in from the start.  However, some things will be missed at that stage and it's up to test to ensure that the customer is represented at the end of the pipeline.  Test should also be involved during the design phase.  If they are only at the end, you have already lost.

  3. Jenda says:

    So that's the reason for the four textbox insanity for entering IP addresses that you see and inevitably hate  on many places of Windows. The worst UI design decision until the introduction of the-UI-formerly-known-as-metro where the user was missing even in the first step. The only worse UI for entering IP addresses would be four dropdowns, I can't think of anything worse than the-UI-the-Germans-understandably-did-not-want-to-be-associated-with on desktop PCs. And the fact is that Maggies were screaming.

  4. cheong00 says:

    From what I experienced, most of the time PM and developers can't resist bugs (on correctness), they often resist changes for "user experience improvement" because unless the said change is a trivial one, it often means late shipment.

    So yes, if you don't have it from start, you'd lost.

  5. mario says:

    I think most of the problem is that programmers don't work with designers. This is not really a code problem but a design problem.

  6. Tutor_eBiz-Developer says:

    The whole thing is BROKE, all the way across the board, for ALL parties. You have customers who "don't get software development" and you have developers who "don't get the product life cycle".  But I'm not really worried about this aspect, because those who "get it" eventually boil to the top in terms of management.

    One thing that worries me is that the WHOLE (yes, entire, from the bottom up) Microsoft approach needs to be revamped. 14 years ago I could use a stupid little MS utility called Homer and emulate 20K users doing browser based actions. I couldn't even tell you what I'd do with VS2013 Ultimate. MS examples for TDD are not even respectable (unless adding 2 numbers together is all your program does). The MS examples on TDD are so poor that developers don't know how to write code which lends itself to testing.

    (If you think I'm that far our in left-field, take your MS Contoso application and show us how to do the things I just mentioned above and post it. Bet you can't, not even a month from now will you folks be able to explain any process which is any kind of screen action recording and playback. Not to over-rant, but I've been trying to retrofit a cloud-based application with TDD now for the last 10 days. All I see is miles and miles of Entity Framework spaghetti SQL…)


  7. @Mario, That is part of the problem, but I don't think designers have a magic ability to understand what users want.  A lot of designers were involved in the Windows 8 UI and clearly that did not resonate the way we all hoped it would.  I think it has to move beyond looking to experts to understand users and move toward looking to users to understand users.

  8. Kristian J. says:

    Interesting comments… I think that it's more of a culture problem than a SDET process problem. If you look at how folks on teams, who actually, use Scrum operate you would not see this (take Xbox for examle). Unfortunately, some groups / organization were left to operate like you describe to their own detrimate… Cheers.