Regression Testing Strategies


There is a lot written about regression testing, and yet there seems to be a lot of confusion about regression testing as well. Just to make sure we are all on the same page, by regression I am referring to the denotation of the word to indicate a relapse to a less perfect or developed state (American Heritage Dictionary). So, the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition. It is important to note the purpose of a regression test suite is not necessarily to expose new defects. The primary purpose of a regression test is to identify changes in behavior from a previously established baseline, which is supported by Beizer’s and Myers’ definitions of regression testing.


However, even on small projects the number of tests required to ensure new builds do not regress or change previous functionality can be quite numerous. So, regression testing demands a strategy in which we limit the number of tests to establish an effective baseline measurement. In IEEE 610 documentation it states regression testing is selective retesting. Thus, the key to an effective regression testing strategy is to design a test suite that provides a high degree of confidence without retesting everything. To limit the number of tests in the regression test suite, we must systematically reduce the number of possible tests. So, we must decide what tests are included in a regression test suite?


Deciding what tests to include in the regression test suite


The most effective regression test suites I have seen include two categories of tests. The first category of tests includes high priority tests for commonly expected functionality (e.g. the 20% of the product that 80% of the customers demand or rely on). The second category of tests includes any functional defects that are found and fixed. Found and fixed functional defects are included because fixed defects do occasionally regress, and if a business decision was made previously to fix a defect then we probably want it fixed before we release the product.


Prioritized feature area/functionality buckets


The tests in the regression test suite should also be partitioned into functional areas and each test in each functional partition or bucket should also be prioritized based on risk assessment criteria.  If the regression test suite is especially large or time is limited, and the regression suite is portioned into functional areas (and those areas are mapped to the project files or modules contain that specific functionality and any dependencies) the regression test pass can execute a limited subset of tests from the regression test suite that strategically target the modules that have changed (and tests for dependent modules as well). Simple directory comparison tools (such as Diff2Dirs), and tools to identify dependencies between modules (such as Depends) are useful in identifying which modules change between builds and to map out dependencies between the modules in each build.


Automate, Automate, Automate


Also, since the regression test suite will ideally be ran on each new build, this is one suite of tests that should be 99.999% automated. Similar to the BVT/BAT test suite the purpose of the regression test suite is not necessarily to expose defects; a regression test suite provides baseline measurement of functionality. Therefore, since these are tests that will be ran several times during the software development lifecycle and are not necessarily designed to expose new defects the ROI for automation is very high. In fact, any test that cannot be automated is suspect for inclusion in the regression test suite.


These are a few ideas to develop a highly successful automation strategy. What other tactics have you found to be successful?

Comments (15)

  1. Shrini says:

    >>> So, the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition.

    This objective of regression testing is *nearly* "IMPOSSIBLE" to achieve to its totality.

    (I added *nearly* because in contexts, owing to business and market conditions, one might claim that they have achieved it)

    How do you qualify "non-functioning state" or "error-condition". what if there are multiple (infinite) ways to define these?

    What is your Test oracle here? Previous version of application and corresponding regression suite.

    The effectiveness of your regression testing thus depends upon "how much you know about" - these" non functioning states and error conditions. Right?

    Checking that the new build caused previously functionality to regress to a non-functioning state or error condition - is nothing short of saying "Test to make sure that everything that was *working* previously, is working now, too.

    When people say "By doing regression testing, we make sure that new code has not broken any existing functionality" or any variation of this statement - what they actually mean or do is to make sure that "Tests (as part of their regression suite) that were passing earlier, are passing now too." ( I owe this statement of wisdom to Michael Bolton)

    Please note that "old tests passing" is *way* different from "old code working now, too". This is a big GAP I observe when people talk about one thing (test passing) and in reality meaning the other ("old code working now)

    Shrini

  2. I.M.Testy says:

    Shrini,

    Please read the post carefully, because I think you are assuming a regression test suite should retest everything; and that is not what is written. Could a regression test suite be large? Absolutely. But, I hope your statement that "regression testing is *nearly* IMPOSSIBLE" doesn't imply a defeatist attitude and mean that we shouldn't create a smart regression test strategy to establishes a baseline measure of specific expectations as part of our overall testing strategy.

    In response to your rhetorical questions, IMHO a tester is responsibile to understand the product's attributes and capabilities, and thus should know what a non-functioning state or error condition is. (Please refer to my post on the role of testing, if you don't know my position on this.)

    I view Michael's "wisdom" as simply a play on words. So, just to make it clear when I discuss regression testing I am not talking about "testing my tests." I design a regression test suite composed of specific tests capable of tactically analyzing possible regressions (or other potential changes) in the product from the pre-established baseline. (Of course, I am assuming you know how to establish baseline measurements.)

    Hopefully in future replies you will present useful information or at least refute statements with specific and accurate facts rather than simply playing with words, making simple assumptions, or asking rhetorical questions.

  3. Shrini says:

    BJ,

    I think you have totally misinterepretted me.

    >>> because I think you are assuming a regression test suite should retest everything;

    No I am not. I *only* took your note on primary objective of regression and analysed to find out the feasibility of fulfilling such objective.

    >>>In response to your rhetorical questions, IMHO a tester is responsibile to understand the product's attributes and capabilities, and thus should know what a non-functioning state or error condition is.

    IMHO, we should distinguish between "responsibility" (some that is given to someone) vs. "Ability" (that comes from inside). My view here is that "understand the product's attributes and capabilities, and knowing non-functioning state or error condition" - is in itself a big puzzle that a tester is always occupied with. By using the term "Responsiblity"  - you seem to indicate that it is something very basic and finite. To me this is never ending and infinite.

    If we knew "complete product features and ALL error conditions" - there would not be any bugs slipped out of testing.

    >>I view Michael's "wisdom" as simply a play on words.

    Playing with words is an *important* part of being able to communicate clearly in *all* circumstances. As context driven and critical thinking tester, I am always encouraged to question and use various combinations of words to communicate. I see nothing wrong with this.

    >>>Hopefully in future replies you will present useful information or at least refute statements with specific and accurate facts rather than simply playing with words, making simple assumptions, or asking rhetorical questions.

    It is up to you to accept or reject my comments to your blog post. I stronlgy belive that being rhetorical is a BIG part of me being a tester and I take your view of "me being rhetorical as a compliment

  4. I.M.Testy says:

    Yes, it is quite possible that I misinterpreted your rebuttals to my post suggesting possible regression testing strategies because you have not stated your position with definitive clarity or substantiated your position with logic, reason, or fact.

    The difference between you and I is that I take a position and attempt to put forth clear, unambiguous, and concise arguments to explain or defend my position. (My position may be incorrect in which case I will admit I am wrong or not completely aware of the facts and modify accordingly.) Conversely, vague rhetorical statements such as “regression testing is nearly *IMPOSSIBLE* to achieve to its totality” or analyzing “to find out the feasibility of fulfilling such objective” are without meaningful substance. It’s sort of like telling a 5 year old child to do something and they reply “No!”, and when you ask why they simply reply, “Because I said so!”

    Responsibility is the capacity of rational thought or action, the ability to discharge obligations, characterized by good judgment and sound thinking, involves accountability, and yes, is given to individuals who are reliable and dependable. Thus, responsibility is usually given to individuals who are accountable for something within their ability.

    That is why I stated, a “tester is responsible to understand the product's attributes and capabilities, and thus should know what a non-functioning state or error condition is.” If they don’t have that ability, then I will not rely on them for rational thoughts regarding the product’s attributes or capabilities, or give them the responsibility for making sound judgments regarding the product.

    As a test manager I was responsible and accountable to my managers to provide them with information that I could defend based on hard data and facts. To accomplish that, I relied on the talented people on my teams whom I trusted and could depend upon. If someone lacked the ability we attempted to train them. If a person was incapable of performing necessary obligations they were often reassigned roles that matched their abilities.

    Also, your statement that “if we knew complete product features and ALL error conditions – there would not be any bugs slipped out of testing” is simply foolish and erroneous. (Also, if you are going to quote someone, quote them correctly.)

    I have taught a variety of subjects to a variety of people for many years. It is truly a gift to be able to rephrase a statement, or use synonymous terminology, or even apply a metaphor to clearly communicate a complex idea or fact under a variety of circumstances. (For example, it is quite difficult for some people to fully understand JBS Haldane’s concept without understanding Boyle’s and Dalton’s laws, while some people simply accept the theory at face value.)

    This is quite different than “playing with words” in which a person twists or manipulates the meaning of words to imply some alternate philosophy or definition because they are attempting to disguise their message. Yes people who play with words are masterful with the language also, but the intent of playing with words is not to inform or educate, it is simply to mislead or to obfuscate the the problem with extraneous information.

  5. Shrini says:

    BJ,

    My position is:

    Regression testing objective when stated as "To validate that the application has not regressed to an error condition or non functioning State" becomes impractical and impossible to achieve.

    I would like to restate the objective of Regression testing to

    "Execute a set of Tests (Regression test) to validate that Application passes the set"

    Main Key words here are "Regression suite" and "pass" - When I say this build has passed regression test - I mean it is passed the test cases in the suite.

    >>>> Conversely, vague rhetorical statements such as “regression testing is nearly *IMPOSSIBLE* to achieve to its totality” or analyzing “to find out the feasibility of fulfilling such objective” are without meaningful substance.

    Please note that I have never said "Regression Testing is nearly *IMPOSSIBLE* to achieve to its totality" - what all I said is - the way you have stated regression testing makes it impossible to achieve. I would say that Regression testing when defined as a "process of executing a set of test cases and checking if they pass" is POSSIBLE. Then the focus of the tester is to design and continusly to improve that Regression suite.

    >>>Responsibility is the capacity of rational thought or action, the ability to discharge obligations, characterized by good judgment and sound thinking, involves accountability, and yes, is given to individuals who are reliable and dependable. Thus, responsibility is usually given to individuals who are accountable for something within their ability.

    Your argument is similar to "Police are responsible to maintain Law and order situation - They are responsible and accountable for any crime that happens in their area". There are certain things that are beyond good and able human intentions.

    By setting up testers that "you shall be responsible for ensuring this ...." and giving them a target of confirming that current build has not regressed to a non-functional/error condition --- a sure way of setting them for failure.  It is something similar to "count the number of stars you see in the sky". For the application of sound judgment, sound thinking with accountability - one needs a rational goal.

    >>>Also, your statement that “if we knew complete product features and ALL error conditions – there would not be any bugs slipped out of testing” is simply foolish and erroneous.

    Read carefully - my statement emphasizes the fact that it is NOT possible to know complete product feature and error conditions considering that software is complex and one can never claim that they know enough. If you believe in 100% Testing and Zero defect software product (your assertion that my comments are "foolish" - indicate that you are do believe in them)

    - feel free and continue to preaching that "it is possible to perform regression testing in your way - validate that application has not regressed to a non-functioning/error condition.

    I am not sure if I wrongly quoted some one my replies to this post. As far as Michael is concerned - I just showed him this whole post and checked if wrongly quoted him. He replied that my reply is correct.

    In my attempt to "play with words" - I am not trying to hide any information or message or trying to obfuscate the problem with extraneous information. I am trying to expose a dimension of your argument. In other words - I am viewing your argument from "feasibility angle" (lateral thinking) and say that regression testing in way that you mention is not feasible"

  6. I.M.Testy says:

    Hi Shrini,

    I thnk we may both be saying similar things with different words. I stated "the primary objective of regression testing is to determine whether or not modifications or additions in each new build of a product cause previous functionality to regress to a non-functioning state or error condition." What I did not explicitly state is that if the regression test suite passes then our defined baseline measurement has not changed and the "application passes the set". This doesn't mean there haven't been changes in the code, this doesn't mean there are still not a lot of defects in the system. It simply means that the criteria we used to establish a baseline measurement for identifying regression of specific capabilities or attributes of the system has not changed. (Perhaps we are getting hung up around what baseline measurements are.)

    You stated "Execute a set of Tests (Regression test) to validate that Application passes the set." I am going to assume here that you have well crafted stochastic tests that prove or disprove the same hypothesis each time they are executed (which is how we establish baseline measurements) and the ideal is the application under test passes the defined test suite. However, if a test in the regression suite fails, then the tests have in fact identified a change from the established baseline that requires investigation.

    If you disagree and think we are still at odds then please provide specific arguments with factual data as to why "regression testing in the way that you [I] mention is not feasible." You might also consider adding constuctive strategies how you would develop a regression testing suite. That approach would add more value to this discussion.  

    Also, my argument of responsibility is in no way similar to "Police are responsible to maintain Law and order situation - They are responsible and accountable for any crime that happens in their area." Unfortunately, your lateral thinking took a left turn if you think police are responsible and accountable for crime. I agree police are responsible to maintain law and order within their abilities. It is the individual who is responsible and held accountable for commiting a crime. If police see a crime and do not act, then they are held accountable for failure to uphold the law. (Perhaps you can have Michael coach you on using analogies and rebuttals.)

    My suggestions provide strategic thought on how to develop an effective regression test suite. The ideas may not be perfect, and they may not apply in all situations, but they do offer solid, specific advice based on applied practice. So, yes, I will continue to adovcate strategic thinking about how to constructively develop more effective regression testing strategies with some specific examples and suggestions. (Thank you for your permission to do so.) . In return, you should feel free to continue your "lateral thinking" and "feasibility angle" approach.

  7. Shrini says:

    BJ,

    Great Reply -- Thanks. Let me get back to you with Strategies for building a good REgression Suite...

    Shrini

  8. kvinayaks says:

    - "regression testing demands a strategy in which we limit the number of tests to establish an effective baseline measurement."

    Can we consider BVTs a part of regression testing startegy?

    v-vinaku@microsoft.com

  9. I.M.Testy says:

    Hi Vinayak,

    This is a great question! The short answer is I don't consider the BVT as part of the regression test suite. I do consider the BVT as a separate (and most likely the first) baseline measurement established after each new build.

    But, in my experience the regression test suite (assuming it to be 99.999% automated) is often ran (almost) immediately following the BVT, and generally using the same lab machines as used for the BVT, (or more if necessary to distribute the load of the regression test suite.) I should also add here, tests in the BVT suite were not included in the regression suite (that would simply be redundant).

    When I designed the BVT test suite for the Windows 95 international versions produced in Redmond I had a constraint of 30 minutes to validate the integrity of 4 language versions of each new weekly build of the operating system (This is before the single world-wide binary devleopment models often used today when each language version was recompiled with #ifdef's - meaning there was functional differences in each language version.) At the pinicle the intl. BVTs were distributed across 12 machines in my office. (I still remember the warmth and constant whir of the fans when curling up to catch some sleep under the bench.)

    As I developed the BVT suite from mostly manual tests to 99% automated my manager would occasionally ask me the status of the BVT and I would reply with vague, non-commital answers such as "well, it's pretty good, but...blah blah blah." To which he replied I must make a firm decision...no sitting on the fence post...it was my responsiblity to make a decision. My manager made it very clear that the purpose of the BVT was to establish 1 of 3 possible outcomes.

    1) If the build failed pre-determined critical tests it was rejected and kicked back to development. Now this was a decision not to be taken lightly, because this meant that not only were the dev's going to be working around the clock to get the weekly build out, it probably also meant most of the team would have to work the weekend to make up for lost time.

    2) The second outcome could be what we referred to as Release for Test Only. This meant the build had some problems detected by the BVT, but some of the problems could be "worked around" and the build was stable enough to regress fixed defects, and continue testing large areas of the product. Some minor areas may not have been testable, but they were not usually critical areas. For example, in one build the menu links in the Start menu failed to launch the applets (Wordpad, Notepad, etc.) but a work-around to the problem was to launch the applet via the Run dialog or command prompt, or by double clicking the executable, so the build was released to the test team only.

    3) The third option was to determine whether or not the build was Released for self-host (this was later called dog-fooding). This typically meant the new build passed the BVT (at least above 90%) and was deemed stable enough for everyone on the team (including managers) to format their main machines and install the latest build to conduct their day to day work. Trust me, when I got this wrong I was in the dog house for the entire week.

     I should take this and make a new post about BVTs because I have more to add here. But, I hope you get the gist of the BVT and the distinction between the BVT and the regression suite.

     By the way Vinayak, I like your blog...the last few posts are indeed interesting.

  10. kvinayaks says:

    (Asking since I don't see any mention of impact analysis.)

    I am taking it that the two categories of tests to include in the regression test suite are selected after impact analysis. Or are you saying that ALL "high priority tests for commonly expected functionality" and ALL "functional defects that are found and fixed" would be part of regression test suite always?

  11. I.M.Testy says:

    Hi Vinayak,

    Please clarify what you mean by impact analysis. In my experience impact analysis is used to assess or predict behavioral changes as a result of a modification to a "system." (For example, if I rename a variable or declare a different data type for a variable I must also assess how that change will impact the rest of the code.)

    I did not say or imply that ALL high priority tests should be included in the regression test suite. The size of the regression test suite must be managed properly for the suite to be effective. So, it is important to carefully evaluate each test that is incorporated into the regression test suite.

    I did say that "any" functional bug that is found and fixed should be included in the regression test suite based on the assumption that if the team made a business decision to fix it once, they probably don't want that defect to reappear. However, are role as testers involves making smart decisions all the time, so perhaps we should also analyze these tests from an ROI perspective as well.

    For example, some functional defects are not catastrophic (no crash, no data loss, etc) and the probability of a user encountering said defect may be extremely low, but we fix that defect. Should that defect be put into the regression test suite? Probably not, because the cost of automating it would probably exceed any reasonable return. I might want to execute that test at some later milestone (just in case) but, I might not want to run it every regression pass. So, again, it is up to the tester to make smart decisions about what tests to include in the regression test suite.

  12. MichaelBolton says:

    >The difference between you and I is that I take a position and attempt to put forth clear, unambiguous, and concise arguments to explain or defend my position.

    The inference that Shrini does not do this strikes me as fairly odious.

    ---Michael B.

  13. MichaelBolton says:

    Pardon me; I meant implication, rather than inference.

    ---Michael B.

  14. I.M.Testy says:

    Well, everyone is entitled to their own opinion. Personally, I find rhetorical questions such as "How do you qualify "non-functioning state" or "error-condition", or "what if there are multiple (infinite) ways to define these?" and statements without a basis of fact such as "If we knew "complete product features and ALL error conditions" - there would not be any bugs slipped out of testing" preposterous.

    But, I think Shrini has some good points to make and encourage him to refute my writings with strong arguments based on factual evidence and specific examples or suggestions.

  15. ferd berple says:

    compare the tables before and after the change.  any rows/columns that have unexpectedly changed were either an error before the change, or an error after the change, or both.  

    If it was an error before the change, then you are missing a test case.  If it was an error after the chagne, then you need to check if you have a test case.

    in any case, must faster to use table before/after tests to establish regression than trying to build exhaustive regression test.

Skip to main content