Testing In The Wild

Recently I read Edward Hutchins's Cognition In The Wild. Edward researches human cognition. Rather than setting individuals to a purpose-built task in a research lab he observed navigation teams on U.S. Navy ships. He was interested in how groups think and in how the environment affects the way a person thinks. He believes that human cognition does not take place only within a person's brain; rather, that the tools and other items a person uses are an integral part of their thinking process.

Navigation is an excellent example of this. The process of plotting a ship's position requires multiple steps, starting with taking a bearing on multiple landmarks and ending with a triangle describing the ship's location plotted on a chart. Multiple tools are involved in this process, each of which embodies various computations of various complexities. Taking a bearing, for example, uses a device called a pelorus to sight the landmark and read the corresponding compass point off the pelorus's gyrocompass repeater. The gyrocompass is itself a tool which adjusts the ship's relative bearing for magnetic and other variances. Each subsequent step makes use of its own set of custom tools.

Many of these tools are the latest in a long line of ever more complex tools which embody ever more of the calculations. The development of these tools goes back centuries. All of them share at least one quality: the person using them need not know any of the theory behind them in order to use them.

When the ship is on the open sea the navigation process is typically done by a single person. In tighter quarters, however, a team of navigators works together. This team is composed of people with varying amounts of experience in navigation, from the pelorus operator, who tends to have the least experience, up through the fix plotter. This aligns with the growing complexity of the operations involved: the pelorus operator need only know how to sight a landmark, whereas plotting a fix requires more knowledge. This alignment means that each member of the team knows exactly how the data from the preceding steps was created, which both helps them notice when the data is suspect and also helps them identify what the problem might be.

The organization of the team directly supports the team's cognition in multiple ways. For one, a brand new sailor can be dropped into the team, given a short training session on sighting landmarks, and be immediately productive. As they carry out their duties they will overhear other members of the team talking and start to pick out additional parts of the process. For another, the organization makes clear the order in which data is gathered and transformed, and errors in this process tend to be noticed quickly. For example, in the normal course of things the pelorus operator will never talk directly to the fix plotter; if they do, something is likely amiss. In Edward's words, "The task world is constructed in such a way that the socially and conversationally appropriate thing to do given the tools at hand is also the computationally correct thing to do. That is, one can be functioning well before one knows what one is doing, and one can discover what one is doing in the course of doing it."

Reading this I suddenly realized one reason why scripted testing is so popular in certain realms (e.g., management types, especially those who don't know much about testing): scripted testing allows a person who knows nothing about testing to be immediately productive. All they have to do is execute the scripts and report their results and lo - testing is complete! Ignoring for the moment all discussion regarding the usefulness of scripted testing, testers in such environments often are not surrounded by more experienced testers and so are unable to pick up knowledge about testing by osmosis the way the pelorus operators pick up knowledge about navigation. If they were, executing scripted tests might be a useful way for newbie testers to learn the ropes.

One might argue that testing is much more complex and complicated than navigation. I am not sure this is true. If navigation appears to be straightforward and simple it is because the gory details have been largely encapsulated in various tools. Test is barely starting to create such tools. Static analysis tools like lint and FxCop spring to mind; a tester need not understand how they work or what they do to analyze the reports they create. Load testing tools fill the bill as well; a tester need not understand the machinations they use to simulate thousands and millions of people hitting a website in order to subject their hardware and software to such scenarios. Mnemonics and patterns fit here too, as crystallizations of partial solutions to frequently encountered problems.

I find it hard to conceive that testing might ever become as rote as the process for plotting a fix. What would it look like though, I wonder, if we ever do develop testing to the point where navigation is today? If we have as rich a toolset which encapsulates most of the difficult bits? If we find ways to run teams which make it difficult for defects to be injected in the first place, and easy to be found if they are? If software development is understood well enough and is reliable enough and is predictable enough to truly be called engineering? While we may never reach that point - and in fact we may not want to - moving closer than we are now seems to me a big pile of goodness.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.