C-DLICE’ing in software Testing

Let me take credit for making this pneumonic up: C-DLICE. I was listening to Michael Bolton's video interview on youtube. He said testing is more than verification, validation and confirmation. It is about Challenging claims, Discovery, Investigation, Learning and Exploring. Any skilled tester would do one or more of these activities as part of testing. By explicitly chaining them in a pneumonic, a tester can focus on a specific aspect of the interaction with the test subject.

Let me expand the pneumonic –

C – Confirmation. Other than traditional words like Verification and Validation (whatever may be the meanings of these terms) most people on this planet think that sole aim of doing testing is "confirmation". It is seen confirmation of claims made about the product, conformation of what developers "felt" that they created in response to requirement specifications that received and interpreted to their best of abilities. The confirmation about some specific user expectations (assumed to be routed through specifications into the software product). In basic form confirmation is somewhat like "Click this button, such and such thing should happen – Does it happen?" While confirmation is important aspect of testing, any testing that focuses on confirmation will become boring, brain dead and poor way to think about testing. Notions like anyone can do testing, process plays important role in testing, testing without test cases and requirements is not possible – are creation of confirmation oriented testing. I will not dwell upon challenges in confirmation oriented testing. And there is a big deal called "reference" against which you confirm – specifications. If your reference is wrong, ambiguous and incomplete – so will be your confirmation. That is the weakness of confirmatory testing.

Though my pneumonic is more about DLICE, I will still keep this "C" in there to remind us that confirmation may be as important as other letters in the pneumonic.

D- Discovery. While we test tester, we discover information about software, certain behaviors. It is like discovering an unknown island. As product grows bigger in size (in terms of codebase), discovery becomes important. No user would use the software as per the user manual. Discovering way in which software could be used and misused is important aspect in testing. Discovering is about finding information about unknown areas. For a growing software application, every time there is more to test than before – more to cover than before. Under such circumstances, you constantly discover the application, its variations, behaviors and so on.

L- Learning- This is a freaky one. A significant part of testing is implicitly spent on learning about everything that software under test. Be it business domain, technology domain, community of users using the software, cultural and social set up in the organization producing the software, we learn all time. Learn about how the software is constructed, deployed, distributed and so on. Often, I have seen people downplaying "learning" aspect of testing as they would like to position themselves as "experts".

I – Investigation As testers we investigate claims about the product. How people perceive the product? Investigate inconsistencies. Investigate bugs, Investigate impact of a new technology, software change on the over image of the software. Investigation is about focused information gathering, analysis on certain events. Examining the evidence etc. Investigation starts off as open ended.

C- Challenge (used as verb) – As testers we need to constantly challenge the assumptions, beliefs around how people think about software. What each stakeholder thinks about the capabilities of the software? Challenge the premises and so on. Challenging would require the design of tests, experiments etc to expose the weakness of an aspect of software.

E – Explore – Somewhat similar to Discovery, Exploration enables any information gathering exercise. Explore market conditions. Exploring is about taking a tour. Exploration helps in modeling the problem space. Exploration is more open ended than investigation.

Notice that each letter is has some overlap with others. You can learn while discovering or challenging a claim or exploring a feature. You can investigate something by exploring it or discovering it. You can challenge something by investigating it or learning about it and so on. One way to think about DLICE is – Discover like Magellan, Columbus, Learn like a learning a new language, Investigate like Sherlock Homes, Challenge like Lawyers, Explore like exploring moon's surface or deep African jungles.

Few practical themes to apply dlice'ing

1. When there is a new thing most people around you know little about - something that you do not understand well , then – Discover, Explore and Learn
   
2. When there is something that several others know but you do not – Learn through exploration, discovering
   
3. When there is"suspense" or "mystery" about a thing – Investigate – a defect, a strange behavior etc.
   
4. When there is some that is "well known" to you (you are pretty sure) about some claim – Challenge it and prove your point ( backed up by prior discovery, exploration, investigation and learning)
   

So, next time you feel bored doing testing, try switching your focus … try doing some investigation, discover new ways of using software or explore an area of software and so on. You would find that testing is always interesting but you were told about only one dimension of it (confirm, find bugs, check it passes tests) so you felt low or bored about doing testing that way ….

HAPPY C-DLICE'ing

Shrini

Context Driven Testing gets a boost – to grow stronger…

It is a fantastic "new year" gift (and also week end feast to many like me who is spending better part of Christmas/new year in front of laptop) to all testers… Dr. Cem Kaner (along with James Bach) has posted about context driven testing with some new definitions and articulation. I belong to context driven testing community with many others. Context driven testing community has proclaimed its philosophy, guiding principles since its inception. As years went by, various groups of people (quite a few "unidentified" ones) started pushing false propaganda about our community. This I believe might have driven Cem and James to "rework" the original principles and overall articulation about context driven community.

I have personally been part of many discussions, where I heard people just attacking the theme of context driven testing by saying "Everything in this software world happens with the context. Only a fool would work without context … so talking big and calling oneself as context driven tester is no big deal. Context is forever one … every one applies the context to best of their ability and knowledge". The new articulation of "Context aware" - attempts to describe such people. If you think about a practice first, then tailor/modify that practice to the context – you would be a "context aware".

Another interesting point this post makes is that some people in agile software development community have found so much common with context driven testing that they were claiming agile and context driven testing as one and the same. With so much focus on people and their interactions – agile and context driven testing can be said to have common roots – "people focus" (as against process/standards focus). I would say Cem's articulation could have been much stronger when it comes to the differences between agile and context driven testing. Insistence of 100% automated unit tests, compulsory standup meetings, TDD is a must and all sorts of "standard" stuff is clearly deviation from agile manifesto (choice of people over Processes) as articulated by James Bach in his STAR West 2008. I see context driven testing community distancing from agile community on these areas …

Also look the new, cracked definition of context driven testing – "Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing." I am sure most of my friends who are not in the CDT (context driven community) will say … oh yes … that is a common sense. We work for stakeholders, customize our testing approaches to context … what is big deal? Why are you making common sense a big thing?" My response to such people would first "so called common sense in this case is not so common … so there is a big deal here. Secondly, thinking about practices first then context is "context aware" not "context driven".

Critical Focus on "best practice" has been rather mild I would say (appears in only 4 times in the post). Cem, apparently left the proponents of best practices, lightly by saying "Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context…. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven."

Context imperial testing is somewhat similar to what I normally refer as "goal displacement" (a term I learnt from James Bach). Instead of designing and adapting the practices to project /organization/group context, Context imperialists would RATHER change the project/group/organization itself to suit the "best practices that they are aware of. A recent example … someone said, we cannot afford to do testing this way as it will not allow us to collect the metrics that we need, so let us change the process so that we can collect and use the metrics". It is sad state of affairs that, people will get away with such "context imperialism".

Context driven testing community's view point on "detailed specifications, detailed test script documentation etc" has always been perceived as "means to promote exploratory testing". Many people, whom interacted, would immediately say "Oh!!! You mean to say do exploratory/adhoc testing then" in response to my statement that "In context driven community, we do not insist on detailed specification … In fact some of us think that is a crime to refuse to test without specifications". In reality, the crux of the matter is, in CDT, testers need to cope up with whatever information that they get and start from there to gather information that they need. In those situations where time, information, people with knowledge - are rare commodity, context driven testing is clear winner. It prepares its tester for such eventual realities.

Finally the assertion that "There are no context-driven techniques" – should put all those statements/viewpoints to rest such as "Exploratory testing is a context driven testing technique". Neither ET, nor context driven testing is a testing technique in itself…

Overall, this post is a great milestone in the history of Context driven testing … should be mentioned in context driven website.

Shrini

MS outlook as Alarm Clock: Is this a bug?

[Background in the beginning … those who are interested in reading about bug I am talking about, skip first few paragraphs and go straight to Bug]

There is a saying that "Software users, most of the time, do not use the software as perceived by the designers or analyst". I happen to deploy Microsoft outlook as an alarm clock for me to help with a wake-up call. I am away from home, not carrying a cell phone, ipod is not a good tool for this purpose, buying an alarm clock for short duration would be a waste … hence zeroed in on Outlook meeting reminder feature as a software solution to help me to give wake up call.

I set up a recurring meeting of 0 minutes at say 7:00 AM everyday in the morning with reminder zero minutes. I thought this would work. I needed another hack … a sound alarm long enough for me to wake up. Most of small sound files shipped with windows/MS office were not helping me with this purpose. I had Yahoo messenger on my machine with an audio file that plays for few seconds. I decided to use that as reminder sound file. Even there was a problem. I thought 2-3 secs of Yahoo sound file might not be long enough to wake me up. So I needed to play this one file repeatedly say 7-8 times and record the whole thing so that I can get a sound file that plays for 8-10 seconds.

I went live with this setup, thinking that my next day's worries of waking up at specified time were over. It seems that UAT was not proper … first morning the alarm did not wake me up at all… when I investigated, I discovered that previous night before sleeping, I had muted the audio of the laptop … so far so good. I was waiting for next day to see if it works … Bingo … it did … exactly the way I wanted. No big deal I thought. However, I was happy to discover a low cost (zero indeed) technology to a problem in hand … software alarm clock.

I am sure there might be better ways … one could have written a small program in VB or Perl or Python to do this … with even snooze feature …. But, I think my solution worked for me … only to fail next day …. What ..? Yes, next day, the important day where I had some important meetings to attend… Outlook alarm failed … Is that a bug … Looks like I accidently stumbled on this bug …

[BUG] I think few hours before the scheduled appointment – alarm to kick off, a pop-up message about (Junk Email alert) came up prompting the user to take action. This is a modal window. Hence before dismissing this dialog box, outlook cannot proceed with any other pop-up windows like meeting reminders. So, my wake-up alarm did not get activated as there was a modal window waiting to be acted upon.

Is this a bug? May be or may not be …. Some might argue that this is not a bug as outlook is not supposed to be used this way. A developer may say that Junk email feature was required to be implemented as modal dialog box, it was assumed that people will act upon it … as they do for any other modal dialog box. Others may say it is too small problem to worry about …Some may even point out a work around for me … check on "Please do not show this dialog again"… so that my alarms will work without any problem. The Fix might be simple … (superficially) just make it as non modal window…. Or there can be other implications. I do not know... What if I had missed the flight back to India due to this software problem? What if I was delayed to all important meeting due to this software problem leading to huge financial loss?

What do you think?

Shrini

Metrics or a liquid in a container?

I was responding to a thread on "productivity in software - when the term is relevant" on test republic and posted following paragraph about quantification and metrics, which I thought would benefit larger audience ....

People (especially managers) when trying to make things that are not (easily, at least without changing the form of task or thing)quantifiable, quantifiable - end up in changing the core of the thing.... that is goal displacement. I once heard a manager asking us to change the way we do testing because the method we were using was not quantifiable (having more human elements etc). We then changed the approach to testing to meet the quantification needs of testing rather than original needs of "information" and "evaluation". Following this, the manager was happy, metrics were available, everything was quantifiable .. but testing suffered. When that happened, manager managed to shift the blame to something else (totally unrelated) and gotten away. He could do that and get away with it because metrics enabled him to do so ... metrics being numbers have no story or no descriptive form on their own. Given a metric, you can tell any story that you want to tell and manipulate the world.

Software Metrics are like gases or liquids, they do not have their shape or form, they take the shape of the container in which they are placed. Be careful about this when making or dealing with things in software that are "quantifiable" ...

Shrini

Quantum Theory and Software Testing - Any connection?

Atomic physics, Atomic structure and history of how all those brilliant scientists discovered (still discovering) atomic structure, electrons (dual nature of wave and particle) etc always fascinated me. More so now, than in my college days. Often I dream of going back to college days and debate with my teachers about stuff like atomic structure. It is tester in me, who curiously looks for finding some answers to questions that keep popping up in my mind.

I happen to pick up a book on quantum theory titled “In search of Schrodinger’s Cat” by John Gribbin. What attracted me is a quote attributed to famous scientist Niels Bohr, that appeared on the back cover of the book.

“Anyone who is not shocked by quantum theory has not understood it.”

I immediately applied it to current state of practice and perception about “software testing” and I would say

“Anyone who is not shocked by popular practice and theory of software testing has not understood it (especially the human element of it”.

I will be shocked when people make claims (what I call as "popular perceptions") about software testing such as “all testing should be documented, test cases are important for performing testing, test cases must be traceable to requirements, you cannot test without specifications, tester’s role is to find bugs, testing assures quality of the product, testing needs to be more process oriented than person dependent etc. I can say that those who are not shocked by such claims have not understood about software testing that relies on human elements (thinking, questioning, observations etc).

Quantum theory is fascinating so is software testing …. I am looking for more connections between these two … Can you help me? Or I am just dreaming?

Shrini

Can Software ever get IT right?

Matt Heusser wrote this beautiful piece of writing about software development practices – quoting another famous Blogger Joel Spolsky.

"... which has a programming method in which programmers code stories based on notes written by designers that are based on requirements documents created by analysts that are assessments of what the customer actually wants. It's practically designed to get everything wrong, to insure that, no matter how ignorant the analysts and architects are on an issue, they'll find someone who knows even less to write the actual code ..."

It is interesting that with so many "loose" ends and human elements (thinking, question, modeling and analyzing), many still fancy the changes of "zero defect software", compare software to manufacturing, glorify processes to fix the problems in software that is INHERRINTLY designed to GO WRONG. If you look at the chain of analyst ->Designer->developer ->tester -> Customer, each one works with less or totally different set of information than all others.

Notice what Matt has to say – to make sure that they get the final software wrong – they will find someone who knows probably the least to write actual code !!!!

How can this (software) EVER GO RIGHT? Can this?

Read entire blog post here …

Shrini

Perils of Quantification – what harm metrics can do for you?

This is a hurriedly written post (just to make sure that I do not lose the thought – "fieldstone" in Jerry Weinberg's terminology) – I plan to use this as a place holder for expanding ideas on this topic … Please bear with me for a while with this "being cooked" idea.

I stumbled on something that Michael Bolton said about metrics – in response a Google group discussion thread. Michael mentions "What you want to beware of, in particular, is turning rich information (stories about bugs, problems, risks, value) into impoverished data (numbers).

I think that is a great way (rather an interesting way) to think about "software metrics". To me, software metrics are great way to "squeeze", heavenly simplify" and "horribly trivialize" rich information about bugs, test ideas, problems, risks, value about software. While they provide a simplistic view of rich and often qualitative/subjective data/information – there is huge danger of "oversimplifying and information loss".

Many people argue with me saying "quantification" – associating something that we try to understand in term of numbers – is essential for science and engineering. Some even quote "you cannot improve anything that you cannot measure". I feel that the "urge" for measuring, notion of be quantitative /objective is simply "over emphasized". Let us consider the perils (ill effects) of quantification. Some entities lend themselves for quantification – say counting. Counting people, counting vehicles on road, counting fruits on a tree, marks a student scores in an exam. Many entities that are related to humans especially are difficult to quantify – tend lose lots of information when quantification is attempted. This is very true with software.

Consider following quantified information – what do you think? What do you lose when you quantify …

1. One tsunami
   
2. 1 billion Indians
   
3. 1.3 billion people in the world below the poverty line of 1$ /day
   
4. 8 million people affected with AIDS disease
   
5. Software Quality of Six sigma
   
6. In 2003 there were 6328000 car accidents in the US.
   

   Finally
   

   6300 bugs in Windows 2000 ..
   

Notice that each of these numbers have rich information about loss of life, health of people, quality of life and so on. By squeezing rich information into a number, we lose the information. Numbers can be manipulated, argued in any way you want, they hide information, you can be cheated by numbers. Numbers are single dimensional where as information they tend to represent are often multidimensional.

"As proven by modern accounting scandals, you can make the numbers say whatever you want" – Mike Kelly

To be continued …

Shrini

A conversation on Automation ROI Part 1 …

When automation is required, either by contract or due to technical constraints, ROI computation may not be helpful. Intangible factors may constitute the bulk of the return, and thus arithmetic computations won't indicate the real value of automation. Fortunately in these situations we often aren't faced with questions about the value of automation because it must be employed regardless.

-Doug Hoffman

Here goes a conversation with a colleague of mine who wanted me to help him with some ROI calculation for an automation project.

Colleague: Do you have a formula or framework for calculating ROI from automation?

Me: I might … first let me understand what you are looking for.

Colleague: It is simple man … Here is a client who is looking for investing in automation and she is interested in knowing the ROI so that she can take it to her boss with business case.

Me: That is good. What are the elements of ROI you are interested in knowing now?

Colleague: What do you mean?

Me: To me, ROI has three elements – a notion of investment (effort, money, time etc – all these can be interdependent in some way), a notion of "return" (called as benefits – some tangible, meaning quantified in terms of numbers, some intangible – soft benefits – qualitative measures) and finally a timeline usually in terms of direct measures like calendar months, or work/effort months OR indirect measures like number releases, number of test cycles, number of platforms covered etc. Which one is of interest to you…?

Colleague: All three … of course!!!

Me: Then you have some hard work to do gather information, data, expectations, some historical and some current.

Colleague: Well... I thought it is easy to find out ROI … I was told that there are many freely available ROI calculators especially catering to automation … are you aware of them?

Me: Yes, I have seen few of them … not so impressed … One problem that I have with most (or all) of these calculators is that a) they use a highly simplified model of testing that is totally out of context. Meaning you can just apply that to any project, any technology, any tool … you will have some numbers coming out … That is too good to believe 2) They equate automation to human testing literally 1:1 … In my opinion, automation is a different kind of testing – remove the human being (to the extent possible) and introduce the machine (automation script) – then think (dream, pray and wish) that program does EXACTLY like a human.

Colleague: This is too confusing …. Let me try to explain my problem in a different way. Customer is investing x dollars for automation, she wants to know when will she be able to recover the investment and when she will start reaping benefits (possibly without investing anything incrementally). How can we help her?

Me : OK … that is fair … Here we come again to same structure – x dollars (investment), when will she recover the investment (time lines) and when/what benefits she can expect, without possibly not incrementally investing (returns). Let us attack one by one … How your client wants to recover the investment?

Colleague: that is silly question … she wants to save manual testing effort by automating all or whatever is technically feasible. So, cycle time reduction is what she is looking at.

Me: So, the questions are – How much will be the cycle time reduction (assuming that that is possible and worth pursuing), by when, that reduction will be realized? What are the incremental benefits till that point of time? Right? Anything else I am missing?

Colleague: Good … I think now you have understood my problem … what next?

Me: Are all cycles of the same size? What happens to application under test for all these cycles (meaning does it under go change or not?) What is the current test cycle time (manual)? What all happens in a cycle? What are things under tester's control and what are not? When do you repeat cycle (assuming that you repeat cycles)?

Colleague: Oh!! My God … my head is spinning … I will have to get all the information and data... Are you sure these are required for ROI calculation? Anything else?

Me: Yes, at least in my opinion, to give a reasonable picture of ROI where R = cycle time reduction – I would need these. There are some more things that I would require to complete the equation … but let us get started with this ….

BTW, what makes your client believe that machine replicate what humans do? There are things machines are good at and there are things humans are good at. No matter what you do … I think in the context of test automation – machine cannot do what a sapient human tester can do (unless, human by design behave as though they are brain dead and emotionless).

Colleague: No …. No … not again … Do not try to confuse me … I will get you the details that you asked for… then let us fit an ROI formula. Please put you're the tester in you to sleep till then …

Me: (smiled) OK … Please bear in mind that "you cannot compare even one cycle of automated execution to same cycle of manual execution".

While my colleague is out to get the data that I asked for … what do you think? What have been your experiences of calculating ROI with automation … how did you deal with "improbable" yet simplistic model of treating automation execution equivalent to what human tester does – hence talking about cycle time reduction etc? What other returns (benefits) that automation provides, have been successful with your clients? How did you quantify them?

I work in IT services industry. Day in and out, I hear people asking me such things - while I attempt to explain them the hazards of the simplistic model of testing and automation they use in ROI, need for business case to push automation (that requires numbers and quantified measures) makes me to look for innovative ways to articulate what I want to say but in way that "business" people can agree on …

To be continued ….

Shrini

Extraas:

Here are 3 useful and well written papers on "Automation ROI"

1. ROI of Test Automation by Mike Kelly (2004)
   
2. Cost Benefit Analysis by Doug Hoffman (1999)
   
3. Bang for the Buck Test Automation by Elisabeth Hendrickson (2001)
   

Closing thought:

"… every time I hear "let's take a look at the ROI," or "it will increase your ROI," or "all we need to do is use the ROI calculator" some little part of me shrivels up and dies. It drives me insane. I refuse to believe that for products as complex and involved as automation and performance testing services (where you need to understand infrastructure, application architecture, business and use cases, deployment models, culture, risk tolerance, and the other aspects of the design, development, and testing taking place for a project) that you can so easily capture the ROI. If it were that easy you wouldn't be talking to me about it."

- Mike Kelly.

2 Notorious “E”’s of Testing

Efficiency is doing things right; effectiveness is doing the right things – Peter Drucker

Let me make a dig on these two notorious and the most abused terms - testing effectiveness and efficiency. Raj Kamal has a post that discusses this aspect here.

To me, effectiveness has a notion of "degree of serving the purpose". For example, we can say "this measure" taken to curb the inflation has been effective (means it appears to have served its purpose). This medicine is effective in slowing down the disease. So when talking about effectiveness with respect to testing - we should map the results to the mission of testing and say the techniques, approaches that you have deployed served their purpose or not. Remember, as testers we serve our stakeholders. Different stakeholders have different expectations from testing. Testers form their mission to suit those expectations.

So, in order to be effective in testing, we need to understand the possible stakeholders, their expectations and which one to focus on. That would lead to testing mission. Any testing that happens, should serve the mission. Along the way, testers employ different approaches, techniques, tools and methods. Few of these would be "effective" in serving the mission and hence serve the stakeholder the information that they are interested in knowing, few many not. Therefore, if you are thinking about articulating about effectiveness in testing, think about stakeholders first, then their expectations, then the testing missions, then approaches, tools and techniques, finally link all of them to the results that you produce. I am not sure if a simplistic metric of an equation that counts "reified" entities like bugs and doing some math (like taking cube root of sum of all bugs and so on). Bugs are not real things but they are the emotions and opinions of frustrated "someone" who matter in your project context. Can you quantify frustration?

Also remember, since there could be multiple stakeholders (hence multiple testing missions), your testing (approach, tools and techniques) cannot be effective for all missions. Accept this fact and you don't have to be guilty about it. This becomes very visible when there are contradicting expectations and hence contradicting missions. Learn to negotiate with the stakeholders, try to iron out conflicts and state in your test strategy which missions you are focusing on and why.

Now, let me come to the term "efficiency". You might have heard people saying – "this vehicle is fuel efficient", "this equipment is energy efficient", "this worker is efficient". To me, the term efficiency is related to the notion of degree of conversion of deployed input (human and machine capital) to desired outcomes. Let us take the example of an internal combustion engine and put the definition of efficiency into perspective - The ratio of useful work to energy expended. Like effectiveness, identifying the best way to convert useful energy for testing to useful results that our stakeholders value – is never a simple task. There is no one right way to things also. To serve multiple stakeholders and testing missions, we as testers need to employ a diverse set of techniques, tools and methods. Hence there can be multiple ways to define "efficiency" with respect to software testing.

While Peter Drucker provides a simple framework for thinking about these terms, I would say it is too simplistic and rudimentary model to apply it to software testing. We neither have one "right" way of things nor a specific set of "right things to do". There are many right ways to do things and there are many right things to do. Who defines the notion of "right"? Our stakeholders. Therefore, it is very important to align our work as testers, to what stakeholder expect. First step towards this is to identify our stakeholders. Have you done that for your project?

I would like to highlight another thing here. Since the notions of efficiency and effectiveness as applied to software testing, are multi-dimensional and cannot be reduced simple set of numbers. Avoid temptation to simplify these parameters into simple metrics defined in terms of entities like bug counts and test cases counts etc. Think broadly and deeply, consider multiple stakeholders and testing missions.

In short, effectiveness deals with "fitness of approach/tool/techniques to serving mission" and efficiency deals with "conversation rate of deployed capital (humans and machines) to intended output". In other words, effectiveness is about "how powerful is your way of doing things" and efficiency is about "how well you do things". Both of these parameters are important indicators of testing work and are multidimensional in nature.

Shrini

Exploratory Testing - the state of the art, Evening Talk

I am delivering a talk on "Exploratory Testing - The State of the art" at STeP-IN forum. This talk is happening at Intuit Campus at Bangalore on Nov 6th.

Find the announcement for this evening talk here.

I plan to cover mainly the advancements, tools, trends in last few years in the field of ET and shed light on controversies and myths associated with exploratory testing. There would discussions on SBTM, ET cheat sheets (Elizabeth Hendrikson), thoughts of Jonathan Kohl (analogy to music), Cem Kaner's thoughts on "ET after 23 years", Works of James Lindsay among others.

Here is a most popular myth... (can you beat this?)

"Exploratory testing is a technique"

Any suggestions, ideas are welcome...

See you there !!!

In the mean time, here are few posts that you can read about exploratory testing ...

Exploratory Testing Shock
18 Myths associated with ET
ET challenged

Shrini

Questioning Software - Bizzare?

Rex Black does not like James Bach’s definition of testing “Questioning a product in order to evaluate it”. I am not sure why. During test2008 conference, this issue was brought up during a discussion. Rex said (paraphrase) “Questioning a lifeless thing like software is bizarre. I cannot question my dog”. I attempted to catch-up with him later next day to see if I can know more about his views.  When I managed to get his few minutes, he took me to a pillar (painted with red) and  pointing his finger to the pillar, said “Are you red”? He continued  “I am asking the pillar. Am I getting an answer? “Questioning software is ridiculous and bizarre”.  I thought I would get a chance to react to what he said. Rex being a busy man and did not have time for “bizarre/meaningless” debates, excused himself and went away.

Is questioning software really bizarre? I don’t think so. When Rex walked up to that red pillar and asked “Are you red” – what was he doing? – questioning the pillar. Right? He did question the pillar. Through his eyes, he could figure out that it was a red pillar. What is the problem then? May be, he was referring to the inability of the “lifeless” pillar not to answer him back in some human language that he could understand. Well, that was answering part- not the questioning part.  Let us apply this to software, everything that we do as a part of testing can be thought of as a question that we ask (not necessarily in the same way as humans communicate) and  Software does answer (unlike the pillar) in a subtle way. Thinking about testing a questioning process is a strong and powerful way to organize the thought process about testing. As testers we must develop skills to question, skills to interpret the answers, skills to improvise questioning and skills to analyze the subtle answers given by the software.

What do you say Mr. Black?

[Update] More on questioning, meanings and various interpretations of "questioning" software by Michael Bolton is here.

Shrini

Soft Part of Software Requirements ...

Michael Bolton in response to a discussion on software requirements mentioned this ...

"... There are many requirements that are matters of opinion, aesthetics, value, usability, compatibility that can't be subjected to a formula, can't be anticipated in advance, and which change over time "

Really true ... what we typically ignore about software requirements is that --

1. Requirements evolver over time consuming, human interactions and communications

2. Formal languages and domain vocabularies have a place in eliciting requirements but we should not confuse them to provide completely unambiguous, clear, testable requirements.

3. Sometimes these formal languages and domain vocabularies are costly, time consuming or simply not feasible.

4. A software specification should cater not only to software program that is being developed and also for the usage and all human interaction related that the program.

What do you say?

Shrini

10 Commandments for Test Automation Outsourcing

Most of my test automation experience and learning comes from working with IT groups and GUI based automation using COTS tools. Outsourcing of testing is a top agenda on today’s IT manager and automation happens to be one of the popular and “most frequently discussed” item.

I am regularly asked to formulate, present and consult automation projects in IT space. Based on this experience, I am attempting to formulate commandments - about 10 of them. I believe these are few considerations that an IT manager who is thinking about outsourcing test automation work.

Automation is not an answer to your testing problems such as limited testing bandwidth, limited time to test, poor application quality etc. These are true testing problems – one way to address them is to set your “manual testing” right then think of Automation. If your manual testing is poor, throwing automation in that will only cause that “poor” testing be completed quickly.

Automation is White Elephant While automation has clear benefits only when it is treated carefully. Automation is not a turnkey, as your application undergoes changes; automation solution needs to be maintained. Plus there are recurring tool costs, training costs and test case/project management over heads. Is your vendor telling about this sufficiently? Is your vendor downplaying this aspect? Watch out …

Anything that is quickly creatable – is quickly perishable Do not believe claims of “script creation in minutes”, “Automated script generation” etc. Any enterprise level serious automation is software development and requires clear attention and methodical approach to design/create and maintain.

Your business users can not create and own Automation solutions Let your business users/SME’s and domain experts do what they are best at – business support. Do not believe claims of business users creating and maintaining automation solutions. It requires good amount of testing and automation knowledge.

Judge your vendor by the questions they ask about automation Suggesting an automation solution requires thorough study of context of project, release cycles, current state of project, future expected life of application, business expectations, state of automation practice etc. Vendors who propose a “standard”, “gold plated” automation solution without asking questions to probe the context are “simply” selling you something – be skeptical about such vendors.
Higher % of offshoring in automation higher investments to make While moving work to offshore will give cost benefits, when it comes to automation there are number of factors to be considered while deciding how much of automation can happen at offshore. Key thing is about creation of application environment at offshore. If a local application environment can be created at offshore with tool licenses – higher degree of automation work can happen at offshore. It is important to note that for GUI centric/COTS automation tool based automation, both automation tool and application should reside on same machine. Hence pay attention to your application environment and feasibility of creating local environment at vendors offshore location before deciding the amount of work that can move to offshore. Even when such local environment is available at offshore, certain activities like acceptance testing, demo to users, and certain types of test cases will need to be done only at onsite. Be sensitive to these factors. Make sure your vendor asks about this and suggests suitable alternatives.

Pay attention to dependencies and Quality of current test Artifacts A successful outsourced automation requires that project dependencies are well understood by all stakeholders. Access of application from offshore, access to testers/developers/SMEs on test cases, Access to reviewers and code acceptance people are few dependencies that one needs to track as part of project. If automation is on the basis of existing manual test cases- make sure that these are detailed enough and available in a form that can be sent across to vendor offshore team.

Decide Acceptance criteria Formulation of Acceptance criteria is not item that is often given least importance while planning outsourced automation projects. Identify an internal owner in your organization who will accept the code/solution delivered by the vendor. Make sure that this person gets engaged in the project from the beginning and formulates the acceptance criteria along with vendor technical team. Failure to identify automation acceptance person from your end and getting a formal agreement on acceptance criteria can leave you “High and Dry” and leaves an open space for the vendor to deliver any “working” automation code but not necessarily the one that stays for long time.

Avoid linking an automation project (and its deliverables) to application release dates one common mistake committed by IT groups outsourcing automation is to link the automation deliverables to immediate ensuing product/application release and cut down time/effort for manual testing. One IT manager said to his project team “We are having a major release for application XXX in January and we have automation solution coming from a vendor by December. Since Vendor has promised that 90% of manual testing will be automated – let us plan to allocate 2 weeks of testing instead of 10 weeks planned earlier. As promised by vendor we can deploy automation and cut down testing by more than 50%”. What is the problem here? What if Automation delivery slips for the reasons beyond control of everyone? What if development delays? What if due to poor test case quality and fast changing application, automation scripts give inconsistent results? This will result in conflict - “believe automation” or “get some good manual testing done”? Such conflicts can severely impact your releases and hence business plans.

Record and Playback (RP) Automation is for Kids This one point can never be overemphasized. Many IT managers still feel that record playback features of industry standard automation tools can help them to create automation quickly and thereafter their own resources should be able to record and create/maintain automation scripts. However, the experience has shown time and again that RP approach is not beyond learning about how automation tool works with application and can not be used for real time, sustainable automation. If vendor proposes this as a part of the solution – you should be alert and suspect the abilities of this vendor to deliver automation solution

Do you agree with these commandments? Any different experience?

[update]
Some additional tips :
What a Vendor should ask you
- questions about your manual testing practice
- Your objectives of Automation and expectations
- Readiness to take up automation in terms of test cases, application state and environment
- Ask about your expectations on ROI

What vendor should suggest you
- Plan for automation maintenance when supplier is gone.
- Automation may or may not reduce cycle time – that depends upon nature of tests, application technology stack, nature of tool etc
- Automation may or may not reduce the cost of testing.

What to look for in an automation proposal
- Acceptance criteria
- Automation design details – how tests will be structured
- Environment related assumptions
- Pre-requisites about tool licenses, test cases, test data, access to developers, testers and business users (for clarifications about test cases)

You can not automate testing – all of testing as testing is intrinsically a human thinking and investigation activity. What you in reality claim to automate is some portions of testing – namely “test execution” of some select test cases. Note that activities like test design, bug investigation and logging, Test results verification etc are still to be done by a human tester. Automation will take you some places but not all. Unfortunately, the places where it does not take you – are the ones where real problems lie – only a skilled human tester can help.

Shrini

Automation reduces cycle time Part III

I wrote about it earlier here and here

“If automation can not reduce cycle time, save testing cost and reduce manual testing errors, why do automation at all” said a colleague of mine in a discussion over mail chain yesterday.

After all, automation and test cycle time (whatever may be the meaning and definition) are so “inseparable” – the theme often evokes deep emotional, business, technological thoughts. While I agree with my colleague on the last item he mentioned “Reduce manual testing errors”, for others I would say they are true under a highly restrictive and idealistic conditions. This is similar to the practice in automobile engineering of specifying mileage of an automobile under “standard road/test conditions”. Such claims are made by automobile manufacturers to push their brand as the fuel efficient vehicle. There also goes a disclaimer that says “Actual mileage may vary depending upon the prevailing conditions on the road, driving habits etc”.

This is very true with respect to test automation too. While in ideal conditions and specific types of testing, automation can help in reducing “test execution” (not testing) time, in reality actual benefits from automation may vary depending upon the project context, state of development and testing practice and others.

I see a future where Automation tool vendors are forced to add a disclaimer to protect them from getting sued by someone who just purchased the tool expecting return as claimed in sales and marketing materials of the automation software.

As I mentioned here, testing cycle time is a complex variable that depends upon several parameters and many of these parameters are out off control of automation and even testing. Establishing a straight linear relation between automation and testing cycle time, according to me is a “horrible” and “unrealistic” simplification.

What is (are) the (fundamental) problem(s) that in testing, the automation is expected to solve?

I am afraid, there are very few testers or test mangers out there that begin their exploration into automation with this question. For many, the notion of automation appears to be well understood I am still struggling to answer this question in a context free sense. How can there be a universal solution to a vaguely or incompletely specified problem?

Dear reader, what do you think are the problems that automation is expected solve?

Shorten test cycle time? Reduce manual errors? Help to kill boredom and fatigue due to repeatability? Save dollars of testing cost? Bring consistent testing results? To supplement human testing capabilities?

To be continued …

Software - A game of questions and answers

"The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question."
— Peter Drucker

"Testing is a questioning process in order to evaluate software" - James Bach

"Computers are useless. They can only give you answers." - Pablo Picasso

"One who asks a question is a fool for five minutes; one who does not ask a question remains a fool forever." - A Chinese proverb

Other day I was discussing with one of my colleague … somehow our discussion went about interpreting in simple terms the whole “game” of software development and testing. Here is what and how ended up in agreeing on the “simple” model to describe software and software lifecycle …(not SDLC but SLC)


Software development is about coming up with answers (and demonstrating those answers with an example) to the questions raised by testers, end users and other stakeholders.

Software testers ask questions about claims and capabilities of what software is supposed to do, take the questions to developers and project mangers and ask for answers.

Project manager or project sponsors scan these questions and pick up those that they think are worth “answering”, prioritize them and pass on the developers for providing answers and ways to demonstrate the answers. Before releasing the software to testers, developer do some question-answer session with buddy developers and leads (peer testing, unit testing and code reviews)

Developers then get on mission to analyze questions and develop/construct answers in the form of capabilities in the software and “release” to testers to check to see the answers are “satisfactory”. When developers do not get answers or feel that it takes relatively long time to find the answers – they turn to project manager with their analysis as why answer can not be made available immediately. Project manager then takes the decision of “deferring” those “unanswered” question to be taken up in future releases.
At times Testers, act on behalf of end users and other

Testers verify those answers and check to see they are OK … some times there will be follow-up questions or new questions (regression bugs/issues) and they are routed to developers via the project manager. This cycle repeats until there are new questions to be answered by the developers.

So … as long as there are questions to be answered about software … there will be the need of developers (who will provide answers) and there will be need of Project managers (to prioritize and check which questions need to be answered) and hence a software development project …

Guess what … it is software testers who drive the whole thing by asking relevant and important questions about software – about it’s claims and capabilities …

So … as important trait of a tester is to practice asking “good” questions …

Shrini

Software - A machine or an organism or ?

“The first obvious difference between machines and organisms is the fact that machines are constructed, whereas organisms grow. …

Whereas the activities of a machine are determined by its structure, the relation is reversed in organisms - organic structure is determined by processes”

This is how Fritjof Capra opens up chapter 8 of his celebrated book “Turning point”
http://www.mountainman.com.au/capra_1.html

How do we understand software …? As a machine or an organism? Does software grow? How do we understand software by it structure or by observing (!!!) it behavior? What are useful models of software that help us to understand?

What kind of thing is this software? One thing is sure … it is not just code …

A general System …!!!! It is code and about various connected systems … can you think of all those systems connected with software system or systems?

Shrini

Are all best practices "worthless"? Testing Best Practices

Other day I was quoting following from Jerry’s new book on testing to one of my colleague who is a “best practice” proponent.

…..The risks in these two situations are vastly different, so do you think I recommended the same testing process I used for finding a personal web-writing application? Do you imagine I recommended that my client install random freeware pacemakers into live patients until he found something he liked, or didn't dislike? Why not?

I took above sentences as reference and told him.. “Can you use software testing strategy that one uses for web application writing to that of an embedded software in a heart pace maker? Hence best practices are such a junk thing ...”

To that he was silent for a while answered --- I agree with your point that test strategy or approach used for web application cannot be applied for embedded software in pace maker … How about picking the practice from a same field/domain – will that not save the time, energy and effort for my client ? Let us say I develop a list of practices for a given field (embedded software used in human bodies) and keep “selling” them as best practices (jump start kit) for those clients who deal with such software? What is your opinion? Would you still say … best practices (in a context) are junk?

I did not have a good answer for him …. Then we discussed about “universal best practices” (I am not sure if such phrase exists as all best practices are universal in nature by default and context less??) such as “walking is good for health”,”Test considering end user scenarios”, “Do unit testing” “Do code review”, “Aspirin is good for heart”, “Drunken driving leads to accidents”, “Do meditation to calm your mind” etc. I told him about at least 3 contexts for each of these best practices where following best practices can lead to harmful effects.

After listening to me … he said … Shrini … you appear to be "making up" all these contexts to prove your point …I want you to answer my question – Are all generic best practices recommendations are worthless or fake? When customers want something readymade that will help them to jumpstart the work, they would like to see if I, as a consultant, can bring some “best practices” from my previous similar experiments. Is that expectation unreasonable?

I am thinking ... I don’t have a good answer for him … do you? I hope Jerry would have some answer …

Are there any "universal best practices" or by default all best practices are universal and context free? Will a best practice cease to remain as bet practice once it comes with a context?

[update] Quoting from Jerry's book again - "As humans - we are not perfect thinkers, we are affected by emotions and we are not clones. We are Imperfect, irrational, value driven,diverse humans - hence we test software and test our
testing AND hence test "best practices" that sales and marketing folks associate with software testing.

Shrini

Exploratory Testing SHOCK ....

A colleague of mine other day expressed his struggle to make exploratory testing work for his time (scalability and making it as a best practice !!!) . He said "Exploratory testing is HIGHLY person DEPENDENT - that is the biggest problem for me ... Do you have any process document for doing best exploratory testing. I will have that included in our testing process framework. BTW that will help us in earning some kudos from our CMMI level 5 assessment team."

I said --" ... that is true ...why exploratory testing ... any good sapient testing is "person - human" dependent. A good testing requires thinking and inquisitive human mind. Are you planning to get testing done by machines, robots - so that your person dependency goes away? If yes .. kindly ask the CMMI team to order few robots for a Testing POC (proof of concept) "

He could not answer for a while ... then responded with low voice "I know you will say something like this only ... I have an answer… Automation!!!! I have raised a request for buying 10 licenses of this # 1 tool in the test tools market .. Howzzzzzzzzzzzzzzzt ?"

Now it is my turn to faint ....

Side Effects of Metrics/Statistics

Jamie Dobson writes this piece of "reality" with respect to statistics/metrics and numbers.

“... that human beings will always work toward their defined success criteria."

True and very revealing for all metric enthusiasts. Just let people know what they will be measured on - they will modify their work pattern and output to suite positively on the measurement criteria. For example, a test team is measured on number of bugs the team logs, you will see more and more bugs and if a test team is measured on number of test cases they execute, you will see testers executing increasing number of test cases.

One thing that happens is what I refer as "goal displacement" - Goal of doing “required” work getting displaced by doing *that* work in *that* way as described/interpreted by measurement criteria. Can you see a problem here?

When working with social/cultural setup involving human beings, introduction of "monitoring/measuring", typically causes a "shift" in a overall behavior of the group towards "what is being measured" instead of "what is required". We tend to believe that people behave the same with or without a measurement system in place – we are wrong

This is the side effect that I am referring when a metrics program is introduced in a software project setup....

Are you aware of this? What steps can be taken to address the side effects?

Shrini

Software Testing certifications Part II

Dr Cem Kaner posted a note to Software testing yahoo group on the topic “software testing certifications”. I thought, it would really make lots of sense and value to a discussion about software testing certifications to share those views here.

Dr Kaner quotes following in his note.
http://www.channelinsider.com/c/a/Careers/VARs-IT-Certs-More-About-Marketing-Less-About-Skills/

Continuing my thoughts on certifications, here is something that I would like to add … on the basis of Dr Kaner’s notes.

1. Certifications have value as “marketing” aid and most confuse them to be as means of getting knowledge or experience or learning.
2. IT organizations and service providers use their “certified staff” as “proof” of their well trained staff to their clients.
3. Certifications do have place in hiring. Whether you like it or not, organizations still use certifications as main filtering mechanisms in hiring just like a college engineering degree.
4. Certifications matter for those who are in the initial stages of their career especially those who are looking to get their foot in testing field. Most of the time, certifications get them a call to the interview.
5. Certifications can get you an interview call, might even get you a job but there after it is your skill and work that “keeps” you on job. Do not mistake certification for life time warranty for the job.
6. One very common argument in favor of certification is that “certification help in knowing the testing vocabulary” – This is true to some extent. But while going for certification with this objective, keep in mind that - there are no universally accepted authorities that define and mandate testing terminologies and terms and practices vary across the board.
7. Certification enthusiasts claim that certifications are means of learning and gaining knowledge in the subject. WRONG … there are better ways of studying and learning than going for certification
8. With the help of internet, thanks for Google and other search engines, today the information is everywhere, just look around you can learn lot and gain knowledge by effectively searching the web, reading blogs, writing blog and engaging in conversation with other in the community.

Now about let me talk about those certifications that hold value in today’s world

• CISSP (Certified Information Systems Security Professional] To earn a CISSP, candidates must have five years of experience and endorsement from any professional certified by (ISC)2, the organization that awards CISSP certifications.
• CCNA (Cisco Certified Internetwork Expert]) – this one especially my favorite as it requires the candidate to demonstrate the knowledge as part of the exam in a lab environment – e.g fixing a faulty router.

To summarize, certifications tend to be of value for some (hiring managers and new entrants) and there are some examples of good certifications that test the skill of the candidate. Do not confuse certifications to “learning” and “knowledge” – most of the current software testing certifications are to be used as “marketing” tools.

What if Automation finds bugs ....? Good thing or bad thing?

Ryan in response to my post on "cycle time reduction and automation", mentions that "It is an accurate statement that automation will not improve cycle time if it finds bugs, where bugs would have otherwise gone undetected. However, I believe the claims automation vendors make, is based on the fact that manual testing is also going to uncover those bugs."

So, when automation finds bugs, your cycle time increases. When people claim cycle time reduction, they "overlook" this fact. why? Those bugs will be found by manual testing too. This may or may not be the case. Bugs discovered by human testing and Automation tend to be of different types.

Now, let us track that trail of what happens a bug is discovered -
In automation - situation could be bit tricky especially when the automation tests, logs are bigger. An error/bug reported by an automation bug needs to be checked to see if it is a bug in automation code or a bug in application or bug in data setup or some timing or synchronization related problem (in GUI automation scenario). Let us say you have 5-7 pages of log file - you will have to scan/read through the log file an locate the bug. You might have to do execute failed automated test manually (and corresponding data setup etc).
In manual testing, human tester can easily trace and follow the bug trail and document the bug. At a high level, bug investigation and isolation tasks tend to become relatively low.

Hence, when automation discovers a bug - things get really problematic.

If one were to cut down cycle time by automation or otherwise, they HAVE to make sure either "no bugs are discovered" or "any discovered bugs are IGNORED" or "bugs that are discovered, if fixed, not tested again and other regression testing is done ....

Can automation control or influence any of above events - prevents bugs being discovered or igonore the bugs if accidently discovered or mandate that bugs fixes will not be subsequently tested?

For the sake of argument, let us suppose that both human test cycle and Automation find same number of bugs ... and take out "bugs" portion of test cycle, how automation can save test cycle time? On what parameters this cycle time reduction by Automation depends ?

Type of test - nature of interactions between "test execution agent" (human or an automated script) and nature of verifications (during and post execution).

• GUI Forms with lots of data input fields - can result in quick form fill tests when automated (zero think time).
• Tests that require longer processing time can not gain from automation as automation can not speed up the processing time.
• Tests that require visual inspection - window titles, error messages, color, tool tip and other GUI centric elements - are better tested manunally as programmatic tests would mean lots of investment. Human testes are quicker and cheaper in such cases.
• Result verification that requires detailed analysis, text file processing, database check etc are good candidates for gaining cycle time.

Thus, there are parameters that are beyond the reach of automation ... hence the notion of cycle time reduction has to be really, really taken with "caution".

Shrini

A catalogue of Test Automation Benefits ...

Other day, some one asked me “in your opinion, what are the real benefits of automation”. That triggered ideas for me to write this post… Let me attempt to consolidate and list all the benefits people claim around “automation”

Real: In my opinion, these are “real” and achievable benefits.

- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency
- Quick – No think time while running tests. (Computer program does not think – So not useful in those cases where you need to think as you execute. How does automation program help you to gain speed when thinking is required?)
- Supplements human ability to spot software bugs
- Helps to run big number of data combinations – can test robustness
- Helps in multi platform combinations (OS/browser/database and other application setups)
- Repeat the testing (test execution) done for configuration A in Configuration B also.
- Hence increased Test coverage
- Following non test execution tasks
- Generate and manage special set of test data
- Large Volume Test comparisons
- Automated workflows and alerts
- Environment setup

Fake: These benefits are on transition point – where the focus slowly starts drifting from “real” to “imaginary” and hence “fake”. These are false promises that are realistically not possible.

- Improvement in application quality (automation can not improve application quality – even testing can not … only developers and business analysts can)
- Improvement in Test process (Test process is pre-requisite for automation)
- Non technical people can do automation – no programming language required
Improved test planning (not sure how)
- By executing a set of test cases for a specified number of times – ROI from automation can be realized. Say in 10 executions –automation test pays for itself after that is “cost saving” all along


Conditional: These are the benefits that are realizable and could be reasonable but under a very strict set of conditions – “once in a while” cases. When these benefits are stated, mostly the conditions that must be fulfilled to realize these benefits, are not stated. This makes these benefits look as though there are real and universally applicable

- Simply – saves time.
- Test effort reduction
- Improved Time to market
- Improved test productivity
- 24x7 testing possible
- Knowledge retention


Totally outrageous: There are some really outrageous and are more or less like a typical sales pitch. People, who make and believe such claims, do not seem to understand human side of testing, testing itself and automation. They are just believe that machines are better than human testers.

- An automation test execution is equal to an hour of skilled human testing
- Automation can replicate human interactions
- Reduced dependency on human testers
- Solves problem of resource crunch and less time available for testing
- Reduced defect rate – fewer defects


Anything I missed?

What is a bug ... A new meaning ...

Let me give a try to this one-liner ...(short post)

"A software bug is a reflection of the mind of a confused human user"

Analyse this statement .....

[Some updates]
Above one-liner proposed by me, seem to have generated interest in some ... Let me clarify further ....
This one liner of mine is a beginning of an effort to understand human thinking process while he/she sees a bug. A human goes through a series of emotions while dealing with a software bug. A dominant emotion among these is "confusion" - state of perplexity, chaos, uncertainty.
Let us say you are running a test, observing what is happening and doing a quick comparison with what were expecting ... Suddenly something "unexpected" happens, your mind starts remodelling all that is happening, you were expecting "x" to happen, where as you are seeing "a", "b" and "c" are happening - that is contradicting your model .... your heart beat goes up, blood pressure goes up ... lots of physiological changes happen ... your brain tries to stabilize, create a new model on the fly, comprehend a things on the ground and after a while, you calm down and begin to put up an explanation of what is happening and why? A bug report comes out at the end ....

So Bhargavi --- the life cycle of a bug begins with a "confused" mind of a tester. A confusion is a mental state where you are not able to comprehend and reason to the situations and problems that you are subjected to (imagine a traffic cop at traffic jam on road, a kid on first day in a new school etc). It is the confusion that triggers the thinking in the mind of tester, tester follows the trail of the thinking and builds up a new model and finally when is mind is settled, comes up with a explanation. So when a bugs gets nailed down, mind becomes calm.

Even an obvious bug - when you look at it deeply is a result of thinking process triggered by "confusion" what is expected and what is observed. As a normal human reaction to a state of confusion, you reason out the things and make them clear - then bug becomes obvious ...
If an obvious bug is at level "0", confused state is at "-1" level. Also the "sense" of confidence is always followed by a subtle at times quick state of confusion. When try to open your car door in wrong way, open your cupboard keys in wrong position, when you are trying to open a door by pulling that opens only on "push" mode ... our mind goes through a quick state of confusion ... you quickly notice the situation and gain your calm.

It is deep cognitive process .... Psychologists will be able to explain better ... "Psychology of a BUG" ?

Shrini

Goal of Testing and a quotable quote

Michael Bolton in response to this post from Steve Rowe mentioned this gem …

“Do our automated tests take into account the notion that different people might value different things, and that one of the tester's primary goals is to recognize different constituencies and the ways in which their values might be threatened? “

Many people in our community think that as testers our goal is:

1. To find bugs
2. To prove that the application "works" as per the specifications.
3. Run a bunch of tests and report the results,
4. Develop some automation to speed up the execution of tests

But to “analyze”, “investigate” the value systems/perceptions of different stakeholders of a software product and explore various possibilities where stakeholder’s value is threatened. This involves among other things – finding some bugs, running some tests, writing some automation etc. Note that End user or customer of the software is an important stakeholder.

This is statement from Michael is an extension or probably logical conclusion of the famous statement that is often associated with testing “Be customer’s advocate” (or “think like customer”). Michael seems to suggest that it is not only customer whom we should consider, as testers we think about all stakeholders and explore what each of these stake holders value.

A stakeholder is a person is who is affected by the success or failure of a project or the Actions and inactions of a product or the effects of service – Cem Kaner

In one of the comments for the same post, I found another worth quoting statement from Ben Walther …

“There will always be inputs into your system that violate the assumptions made in building it. A computer generated test will not be able to violate such assumptions.”

Very true … that is why testing is so challenging and exciting

Shrini