Tuesday, July 10, 2007

Regression Testing and Insanity ....

"Insanity is doing the same thing, over and over again, but expecting different results"
Albert Einstein (attributed to Ben Franklin too)

Does this sound similar to what we do in the name of "regression testing”? In regression testing (more so in that regression testing whose test execution is automated - note I am not using the term "automated regression testing") , we believe that by doing repetitions of some things that we did previously - we can find new results.

Thanks to inherent complexity in software and involvement of humans in development, testing and usage of the same - this regression insanity appears to be *working*

This one of those Francis Bacon's idols - Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

Read following to know more about Francis Bacon and how his idols are relevant in testing

Francis Bacon’s New Organon (James Bach)
Bacon and boundary testing (Mike Kelly)
The four Idols of Francis Bacon
The Four Idols of Sir Francis Bacon by Ben Chambers and Zeb Dahl.


More on this later...

Shrini

11 comments:

Ben Simo said...

Sometimes doing the same thing over and over and expecting a different result is not insane. ... thus, the pinata heuristic (I heard this from James Bach at CAST):

Keep beating it until the candy comes out.



Ben Simo
QualityFrog.com

Gals said...

Doing same thing over and over again on the "same thing" might not reveal different results but doing the same thing on different things does really do.. and coming to the "regression tesing" the intent of tests here is not to show that features fail.. but to show that some features work the same way as they were working earlier... Is there something incorrect about this? ... while.. It is a completely different thread to discuss on automated / manual regression testing.. pros / cons

Shrini Kulkarni said...

Hi gals --

thanks for commenting

>> coming to the "regression tesing" the intent of tests here is not to show that features fail.. but to show that some features work the same way as they were working earlier...

My only concern is about the word "working" which can have multiple meanings to multiple people. Try reading above paragraph by replacing "features work the same way as they were working earlier" by "Tests that were passing earlier are passing now TOO" - that would read fine right?

Read Matthew Heusser's blog post on Automation ( dwells on regression testing) and my comments - identifying at least three types of Regression ...

http://xndev.blogspot.com/2007/07/test-automation-iv.html

and

http://xndev.blogspot.com/2007/07/test-automation-iii.html

I prefer to use "Tests that pass" in the definition of regression testing in place of "features that work" - as former is more objective and less likely to be challenged.

I owe this peice of wisdom to Michael Bolton ...

Shrini Kulkarni said...
This comment has been removed by the author.
Shrini Kulkarni said...

Hi Ben,

" ... a different result is not insane. ... thus, the pinata heuristic"

Great .. a new meaning to celebrated insanity ...Did you say " ...not insane THUS pinata heuristic"

or
you wanted to say " ... not insane - BUT you have a whole new meaning/interpretation for it - Pinata heuristic"

Using BUT in place of THUS can make a significant change in the meaning of sentence right?

Practicing precision talking/writing...!!!

Thanks for bring that here ...

I am big fan of James' work ....
Post more about Pinata Heuristic

Keep coming...

Ben Simo said...

Sometimes doing the same thing over and over and expecting a different result is not insane.

That's just what I intended to say. Each time we repeatedly do something we may be changing the state of whatever we are doing it to.

Each time we smack a pinata with a stick, we are doing the same thing over and over and we are expecting a different result. We expect that the pinata will eventually break and drop the candy it hidden inside.

Each time we beat the pinata, we are changing its state -- even if it is not noticeable. We may beat on it 20 times and no candy drops. We may beat on it 50 times and it still doesn't break. We still expect it to break because we are hoping that our beating is weakening the pinata. If we stop before it breaks, we won't get the candy.

In testing, true repeatability is nearly impossible to achieve. This is because each action we take on a system changes its state -- even if we don't notice the change. If we keep beating on it, we may eventually do whatever it takes to reveal the "candy" -- the bugs.

I see this in my automated model-based testing (generation and execution from an explicit model). The automation helps me beat at an application longer than a human tester is willing to work at it. However, I do admit that in some cases, an observant human tester may be able to better observe the results of each "smack" at the application and find the "candy" faster than a robot "blindfolded" by an explicit model.

People that remove the blindfold are able to observe and make decisions about how to beat the pinata to find its weakness.

Robots are able to beat for long periods of time without tiring, but they are not able to learn and adjust like a thinking tester. A robot under the control of a thinking human being can be a useful tool.


Ben Simo
QualityFrog.com

Anonymous said...

Hi Shrini,

But, regression testing (repeating tests which have previously been executed; and [hopefully] successfully passed) is not doing the same thing over and over and expecting a different result.

Regression testing is doing the same thing (or some equivalent variation) over and over and expecting the same result!

I don't necessarily want a test in my regression test suite to fail because that means there has been a fundamental change in the software behavior.

- Bj -

Shrini Kulkarni said...

BJ --

When I said “Regression testing is doing same thing over and over and expecting different thing” - I meant following -

Doing same thing - Repeating some old tests, hoping that the tests are repeated in real sense (Note that it is impossible to repeat a test in its entirety)

Expecting Different things - Hoping that those tests will reveal some new information, bugs due to intended code or platform change.

I have seen people religiously executing old tests expecting them to (new) catch bugs. They might in some cases but then, that happens partly by fluke or partly because tester (by mistake) deviated from script and stumbled on bug.

>> Regression testing is doing the same thing (or some equivalent variation) over and over and expecting the same result!

Which is that result?
Tests passing? That would be realistic.
Assurance that features that were working earlier are still working fine? - This might not be objectively confirmed as "worked" is a highly subjective term

Shrini

Ben Simo said...

Shrini,

BJ is correct in that the purpose of regression testing is to get the same results as past executions. Thus the name "regression": it regresses a previously traveled path. The failure comes when people expect regression tests to find new bugs that are not on the scripted path. Whether scripted or automated, our expectation should be that regression testing passes every time.

The value in regression testing is its ability to tell us if code changes impact something it was not expected to impact.

Regression testing is just one of the many things that are in a tester's toolbox. It should not be the only tool we use.

Ben Simo
QualityFrog.com

Mandar said...

Shrini,
Digressing a bit from the main topic (regression and testing) but catching on to the why "do the same thing again and again" thing...

If you see a James Bond film , then all Bond films have a strict formula. In each adventure, he saves the world, defeats crazy villains, and gets to play with cool gadgets and beautiful women.

Moral of the story: If you are successful at something, and you know why it works then do keep repeating it.

So don’t change your winning formula for the sake of changing it. So if you know that your tests are potent bug killas then why not keep running them again and again...? Unless you hit the pesticide paradox and then you change the actor who plays James Bond and hope for same magic again.

Sofia Hunt said...

Very informative post and it was quite helpful to me. I also wrote something on similar lines on regression testing - http://bit.ly/1LwT5Jp