The Science of Software Testing


I was going through some old files today and came across a paper I had written February 7, 2002 shortly after becoming the director of test training at Microsoft. While a bit dated, it reflects many thoughts I maintain to this day on the role and challenges of software testing.


------------------------------------


Software testing was once an afterthought in the commercial software communities. Novice product managers and even developers looked upon software testing as an unfocused discipline that almost anyone could perform. Some popular books on the subject of software testing even included words such as art and craft in their titles, which may have led some people in the software industry to conclude that testing for quality was not a true software engineering discipline. To the contrary, effective testing is an software engineering discipline and a critical component of the sofltware development lifecycle to help improve software quality and reliability through defect identification, error prevention, observation and reporting. Essentially testing provides information for rational and analitical risk analysis.


The creativity in testing


Although some may consider testing software to be an art, its only similarity is that it involves a great deal of creativity. It could be argued that testing a large-scale computer program requires more creativity than designing that program. Glenford J. Myers backs this theory in his book The Art of Software Testing. "Testing is an extremely creative and intellectually challenging task." In fact, many people who could be designing and developing software admit that they find more aesthetic satisfaction in the creativity of testing.


A philosophical view of testing


Since the beginning of time, humans have participated in various forms of testing by questioning the designs of tools and processes. This constant re-evaluation led to improved quality and significant gains in effectiveness and efficiency. But testing based purely on trial and error is laborious and time consuming. In 1962, the philosopher Sir Karl Popper approached knowledge growth from a more scientific process. In his book Conjectures and Refutations, Popper hypothesized that science progresses by reducing falsity. Essentially, progress is achieved by formulating theories (conjecture - reasoning that involves the formation of conclusions from incomplete evidence) into testable hypotheses (refutations - any evidence that helps to establish the falsity of something).


Testing as a science


Computer software is similar to a scientific hypothesis; both are inherently fallible. The basic framework of the software debugging process is analogous to the trial and error practice that advances scientific hypotheses. Ccomputer software is simply technical conjecture. Test engineers attempt to refute the assumption of flawless software through rigorous tests mostly designed to prove that defects exist (falsification). The falsification process sharply contrasts with data-driven approaches such as induction and deduction, which attempt to justify validity based on the repetition of facts. Justificatory approaches mostly attempt to support claims through confirmation. Confirmation primarily endeavors to validate the error-free uncertainty by using specific data that demonstrates proper functionality. This approach clearly only proves that software functions correctly under certain conditions; it does not prove that the software is error free.


Establishing confidence in software


The best approach to establish confidence in software reliability is not only to demonstrate that a program works within the confines of the product specification, but also to expose potential defects in the design or functionality of the software. However, any conjecture can withstand countless hours of exhaustive testing and still contain errors. Refutation cannot guarantee flawless software because there are simply too many hypotheses to test in a reasonable amount of time. William Hetzel explains the futility of exhaustive testing and the limitations of refutation in The Complete Guide to Software Testing. He writes, "There are more test cases in a program with 70 branches than there are teaspoons of water in the Pacific Ocean." When a tester successfully refutes the software (finds a bug), he reports the defect. The developer then (hopefully) resolves the issue and assumes that the software works correctly and the refutation process repeats. Although testing can never prove that software is without errors, detecting errors highlights problems, which ultimately results in higher software reliability and quality, or at least provides a foundational perspective for risk analysis and mitigation.


What testers really do


When we ask test engineers to describe their job, most respond, "Breaking software." Perhaps this is why some people mistakenly view the role of the tester as a destructive process. However, testers do not really "break" software. Testers simply expose errors that already exist by refuting or falsifying the assumption that the software is flawless through the observed results of structured tests, logical deduction, and experiments. But, finding errors is just one aspect of software testing. Test engineers add tremendous value in the development process by validating the software, detecting errors in the software, providing developers with information to prevent bugs, presenting information to management to evaluate risk, and by acting as the advocate for the customers.  

Comments (2)
  1. MichaelBolton says:

    BJ in September 2006:  "I won’t pretend to know what the solutions are to the problems we encounter in software testing, but I am pretty confident the solutions are not going to be found by studying <b>cognitive psychology, epistemology, or systems thinking</b>."

    BJ in January 2007:  "In 1962, the philosopher Sir Karl Popper approached knowledge growth from a more scientific process. In his book Conjectures and Refutations, Popper hypothesized that science progresses by reducing falsity. Essentially, progress is achieved by formulating theories (conjecture – reasoning that involves the formation of conclusions from incomplete evidence) into testable hypotheses (refutations – any evidence that helps to establish the falsity of something)."

    One down, two to go.

    —Michael B.

  2. I.M.Testy says:

    Hi Michael,

    Good to hear from you again. I started reading Popper while pursuing my Anthropology degree in university. I absolutely agree with his theory that science progresses by reducing falsity. In essence it is the fundamental principle of troubleshooting any system. (I rely on the philosophy of reducing falsity regularly while troubleshooting the engines on vintage cars.)

    As I also wrote in the 2006 post, I think understanding general systems thinking, epistemology, or cognitive psychology are important in expanding one’s ability for abstract thinking regardless of profession.

    I also suspect that folks already know how they learn, can think analytically, and also think abstractly. I expect that testers already have the capacity to think for themselves, and are able to learn, interpret and apply new concepts and skills of software testing and computer science that are applicable and immediately relevant for their chosen profession.

    But, I am not an expert in teaching people the origin, nature, and scope of knowledge from a philosophical viewpoint, nor am I an expert in teaching the psychological study of mental processes, so I won’t pretend to teach such subjects.

Comments are closed.

Skip to main content