We Need A Better Way To Test
Testing started simply. Developers would run their code after they wrote it to make sure it worked. When teams became larger and code more complex, it became apparent that developers could spend more time coding if they left much of the testing to someone else. People could specialize on developing or testing. Most testers in the early stages of the profession were manual testers. They played with the user interface and made sure the right things happened.
This works fine for the first release but after several releases, it becomes very expensive. Each release has to test not only the new features in the product but also all of the features put into every other version of the product. What took 5 testers for version 1 takes 20 testers for version 4. The situation just gets worse as the product ages. The solution is test automation. Take the work people do over and over again and let the computer do that work. There is a limit to the utility of this, but I've spoken of that elsewhere and it doesn't need to be repeated here. With sufficiently skilled testers, most products can be tested in an automated fashion. Once a test is automated, the cost of running it every week or even every day becomes negligible.
As computer programs became more complex over time, the old testing paradigm didn't scale and a new paradigm--automated testing--had to be found. There is, I think, a new paradigm shift coming. Most test automation today is the work of skilled artisans. Programmers examine the interfaces of the product they are testing and craft code to exercise it in interesting and meaningful ways. Depending on the type of code being worked on, a workforce of 1:1 testers to devs can usually keep up. This was true at one point. Today it is only somewhat true and tomorrow it will be even less true. Some day, it will be false. What has changed? Developers are leveraging better programming models such as object-oriented code, larger code libraries, greater code re-use, and more efficient languages to get more done with less code. Unfortunately, this merely increases the surface area for testers to have to cover. Imagine, if you will, a circle. When a developer is able to create 1 unit of code (r=1), the perimeter which a tester must cover is only 3.14. When the developer uses tools to increase his work and the radius stretches to 2, the tester must now cover a perimeter of 12.56. The area needing to be tested increases much faster than the productivity increase. Using the same programming models as the developers will not allow test to keep up. In the circle example, a 2x boost in tester performance would only cover 1/2 of the circle.
Is test doomed? Is there any way to keep up or are we destined to be outpaced by development and to need larger and larger teams of test developers just to keep pace. The solution to the problem has the same roots as the solution to manual testing problem. That is, it is time to leverage the computer to do more work on behalf of the tester. It will soon be too expensive to hand-craft test cases for each function call and the set of parameters it entails. Writing code one test case at a time just doesn't scale--even with newer tools. In the near future, it will be important to leverage the computer to write test cases by itself. Can this be done? Work is already beginning, but it is just in its infancy. The tools and practices that will make this a widespread practice likely do not exist today. Certainly not in a readily consumed form.
This coming paradigm shift makes testing a very interesting place to be working today. On the one hand, it can be easy for testers to become overwhelmed with the amount of work asked of them. On the other hand, the solutions to the problem of how to leverage the computer to test itself are just now being created. Being in on the ground floor of a new paradigm means the ability to have a tremendous impact on how things will be done for years to come.
Update: There are a lot of people responding to this post who are unfamiliar with my other writing. Without some context, it may seem that I'm saying that test automation is the solution to all testing problems and that if we're smart we can automate all of the generation. That's not what I'm saying. What I advocate in this post is only a powerful tool to be used along with all of the others in our toolbox. If you want some context for my views, check out: