Thinking Tester

Cause and Effect - Non Linear Systems

2011-11-12T22:28:00.001+05:30

Here are three examples where cause and effect do not appear to corroborate. Take a look.

Build a flyover on a busy road hoping that traffic will ease – A personal experience[Traffic will actually increase with flyover]
Dip a thermometer in boiling water. What happens to temperature reading? – Adopted from Gerald Weinberg’s Book “Introduction to General Systems Thinking” [Thermometer will show a lower temperature reading due to difference in thermal expansion between mercury and enclosing glass tube]
Making Cars more safer will cause drivers to become more aggressive and rash – “Peltzman Effect” Adopted from freakonomics post "What happens to your head”

In each of these cases – the effect or what is expected is not what happens – but the opposite. There can be explanations – lesson for testers : think holistically, develop systems thinking mind.

Why this happens? I think we often approach in terms of or use analytical/reductionist thinking - just bread/divide a thing into its constituents (atoms) and study them. This linear thinking of single cause-effect (taken one at a time usually) can help understand some aspects an object/phenomenon. With non linear systems such as societies, political/cultural systems, business systems etc - this simple cause-effect thinking simple does not hold good.So - think in terms of systems and their interactions.

Shrini

Do statistics lie?

2011-08-11T19:55:00.002+05:30

There are no whole truths; all truths are half-truths. It is trying to treat them as whole truths that play the devil" – Alfred North Whitehead

I received an email this morning with this topic of statistics and the content of mail and overall theme of the mail boldly proclaimed “statistics do not lie”. When we speak of statistics in software world, we typically refer to metrics and various numbers representing effort, number of defects, test cases, cost etc. So, instead of talking about statistics – lets talk about “Do numbers lie?”

We have two items here – numbers and lie or truth.

Let us start with numbers. What is a number after all? The need of having numbers probably is related (or caused by) need for counting. In Egypt, from about 3000 BC, records survive in which 1 is represented by a vertical line and 10 is shown as ^.
According this historical account, Pegan priests need to calculate the frequency of natural phenomenon. One of the best known examples of this period is the Stonehenge stone circle in Britain, built by the Druids as a kind of celestial observatory in 1,800 BC. For cave men of pre-historic age, counting facilitated sharing food items - probably. The pictures on the caves and archeological finds give us an idea that people in those days counted by drawing lines to indicate “how many”. Counting fewer items, let us say 6 fruits could be fitted in this idea of counting with fingers or counting by drawing lines. But how would you count number of fruits in a tree or number of people in a village? Discovery of place value system allowed counting items in 100’s and 1000’s. Hind-Arabic numbers 0…9 were known since probably 300 BC. Before Hindu-Arabic numbers, people used “roman symbols”. Interestingly enough, Greeks and Romans did not know the idea of “place value”. And human civilization evolved.

Gonitsora – an initiative from few students of Tezpur University India, carried an article on history of counting that said –

“The first motivation for people to create number was the human desire to the manyness of a set of objects. In other words, to know how many duck’s eggs are to be divided amongst family members or even how many days until the tribe reaches the next watering hole, how many days wills it be until the days grow longer and the nights shorter, how many arrow heads do one trade for canoe? Knowing how to determine the manyness of a collection of objects must surely have been a great aid in all areas of human endeavor.”

When we say “9” what does that indicate in a purest and objective sense? Nothing. OK, let us say 9 cars? What does that mean? Extending it, “9 cars parked in front of a house” what does that mean? “9 cars parked in front of house of a celebrity in London”. You might say “might not be any significant or interesting” as it is common for a celebrity in London have that many cars. Now if I say “9 cars parked in front of a house of politician in Delhi” – Something surprising or some planning happening on a political discourse. Now if I say “9 cars parked in front of a poor in a remote village in Somalia” – what does that mean? You would really get interested to know what might be happening in that house, who came in those cars, where did the come from? Did this poor steal those cars and all sorts of questions.

Pause for a while – and think did or do the number 9 revealed any truth in each of these situations? Is the number 9 capable of telling any truth or for that matter lie at all? A number meaning full and relevant by set of object/objects, people, ideas, events that the number points to. Thus it might be totally meaningless and absurd to say “number don’t lie”, as a matter of fact numbers of incapable of telling truth or reality independent of context, observers and recipient of the information.

Let us talk about truth. What is truth – a question that is at the base of all philosophy, science and every root of what we call “knowledge”. For the purpose of this post, let me use this definition (very tentative, provisional) - “truth is a qualifier that we can attach to a piece of information about which a group people do not disagree by and large”. Back to software world – give me an example of truth. One might say “In this month there were 9 sev 1 incidents in production”. Is this a truth? You may point to live incident tracking system and show a list of incidents reported in this week and say “look here is the truth there are 9 live incidents”. Let me apply my “provisional” definition of truth here. Let us call 10 people – few programmers, few business analysts, a project manager, a business unit head, a customer and a sales manager. Let us put these people in 10 separate rooms and show them the list of 9 live incidents and ask them “is this information true?” Let us record each response. What do you think would be those responses? Will all of these agree on the notion of truth of 9 live incidents? I guess – many would say “Yes, I know there are 9 live incidents this week But ……” What follows after but is each person’s view point or story of how they view (defend, attack, frown, shout, feel sad etc) those incidents. How do you extract “truth” from this beautiful, “god-like”, impartial number “9” quantifying live incidents”?

You would soon have a consultant selling a version of “cost of quality” and attach some dollar figures to these 9 incidents and sell a multi year “transformational” deal to reduce the cost associated with these incidents. Should you believe him?

Often when executives say “I need statistics, numbers” – it seems to me that they are really (should be) interested in the stories behind those numbers, they are (should be) least bothered about numbers themselves. Numbers are masks for stories, events and emotions that they represent.

Numbers, statistics – are incapable of telling anything in absence of context, stories, people and their motivations. For now, I can say the issue of whether statistics tell lie (or truth) is settled – they don’t tell anything.

An exercise: When I was preparing this post, a colleague of mine, Joy Chakraborty challenged me and said “company financial results” are objective truths about company’s performance (he did acknowledge Satyam Saga and other irregularities about how company financial results could be manipulated). He simply asked how the numbers in the statement - “Goldman Sach’s reported net earnings for Q2 2011 – of 1.1 billion USD - 77% up over previous year’s same quarter” are not objective truths? What do you say?

Is Pythagoras theorem true? How about Einstein’s General theory of relativity?

When Testing Fails ...

2011-06-27T18:40:00.002+05:30

Carl Segan once famously said “Science is a self correcting process – an aperture to view what is right”. Carl Zimmer in an article on Indian express says “… science fixes its mistakes more slowly, more fitfully and with more difficulty than Sagan’s words would suggest. Science runs forward better than it does backward.” According to Zimmer, checking the data, context and results of a published scientific work is not of much interest to journals that take pride in “first” publishing ground breaking new research. For scientists scrambling for grants and tenures – checking published work is not attractive and often an exercise not worth its effort. As a result of this, Zimmer says, original work/papers often stand with little or no investigation or scrutiny by peers. This surely, is bad for science. Zimmer, suggests the community to have focus on “replication” and setting aside time, money and journal space to save the image of science as self correcting pursuit of knowledge in Carl Segan’s words.

Well … does this have anything to do with software testing? I recall James Bach once saying “it is easy for bad testing to go unnoticed”. Like science and scientific community’s social fabric – software testing world (software world in general) has built-up layers of wrapping and packaging. It is easy to find some reasons in one or more of these layers in the ensemble of social systems leading to few missed bugs that cost few days of outage of say- a popular e-commerce website. Any query or investigation on the effect of outage or episode of missed bugs would set-up a nice blame game looping all the way to testers, developers, business analysts, test data, environment, lack of technical/business knowledge and host of other things. Like it happens in published science works – hunting down the culprit or group of culprits would be time consuming job. In any case it is regressive job and takes valuable resources from many “progressive” tasks. Right? In Zimmer’s terms – spending time on production bugs is similar to running backwards. Does testing work well when running backwards? Do stakeholders like it when testing is running backwards? Not sure if published research wrong or publishing a new perspective on the basis of an existing work can be as productive as hunting down a missed bug and trying locate the reasons for its birth in the first place.

While process enthusiasts, Quality assurance people, Sick-Sigma (oops… six sigma) people might protest and insist on full blown root cause analysis of all bugs reported in production. SEI/CMM folks might make it mandatory to document all lessons learnt from missed bugs and refuse to sign off the project closure unless this is done. In spite of this – in a fast paced life of software where cycles between conception and real-working-software, are shrinking, ever increasing number of platforms (don’t forget mobile platform) – who has time to look at root cause analysis reports and all those missed bugs?

I remember a business sponsor once saying – “I can’t wait for testing to complete, once the development and put the stuff into production. If something breaks, we have enough budget provisioned to take care of any production failures.” Here, failure to put (buggy) software in production is more disastrous than waiting for “perfect” software to develop. Back to layers of social system in software testing – it appears to easy to hide bad testing – unless you screw-up very badly, the chances are that your stakeholders will never notice what good testing can get them.

I often wondered looking at some testing that happens in some organizations – how they are managing to stay afloat with such bad testing? The reason is probably – when testing fails, it is difficult to attribute it to testing and call it as such. It requires courage. How many testing leaders are willing to admit their testing sucks without hiding behind fancy looking metrics, root cause analyze reports and charts? That is the effect of “software testing” as “social process”

Sure Ways to Reduce Test Cycle Time through Automation

2011-06-07T20:05:00.010+05:30

World is simple, we complicate the world for the sake of it – says a friend. There is a simple concept called "automation" and another one "testing cycle time". Why these terms can not be simply understood without much fuss and inquiry? he often argued with me. This friend is a manager and works for IT services company. A constant pouncing on me by this topic of test-cycle-time-reduction-through-automation and related hype, frustration and feeling of achievement, made me to think deep about basic laws or golden principles that govern this phenomenon of cycle-time-reduction.For the benefit of my blog readers, here, I make them public. Read them, understand them, implement them and be blessed. A caution here any criticism and cynicism about these laws will have harmful consequences to the beholder. These are golden principles and axiomatic about test automation!!!!

First principle - “About Testing”: Strongly believe that software testing is a deterministic, highly repeatable and structured process – somewhat akin to a step-by-step procedure to produce say a burger or car. You have to believe that given a fixed scope of testing, it always takes a finite and fixed time (effort) to complete testing. Needless to say, you have to have absolute faith in testing processes and standards. Your faith in the power of processes and standards making testing predictable and repeatable is an important success factor. It is also necessary to abandon any misconceptions you might have about relationship (or dependency) between testing and automation. You should treat testing truly as a mechanical, step-by-step and repeatable process to ensure Quality. Positive ideas about metrics to improve testing and consistency are - a sure bonus. You should resist and fiercely oppose any attempts to link automation and testing, claiming that automation can and should work independently.

Second principle – “Definition and Meaning of testing cycle”: Never challenge or probe definitions and meanings of word “Testing cycle”. Keep it (testing cycle time) loosely defined so that you can flip in any direction when confronted by a skeptic challenging your claim of cycle time reduction. The more the vagueness of the term “testing cycle and cycle time”, higher are the chances of meeting goal of reduction of cycle time through automation. Any variables and factors that make “testing cycle” somewhat unclear – should be ignored and not discussed. It is important to have faith in the fact that “testing cycle time” is universally known term and does not need to be redefined in any context. You would be looked upon as genius if you simply talk about “cycle time reduction” and omit the unnecessary qualifier “testing” or “test” (as in “test cycle time”).

Third principle - “Playing to the gallery”: Use words like “business needs”, “business outcome”, “business processes”, “success through business alignment”, “Time to market” etc., liberally during all communications related automation and testing cycles. Confront any opposition by skeptics about automation and its connection to cycle time by “calling authority” – say “Business/Market needs it”. Strongly believe in statements like “Automation will help releasing products and services faster – and hence will improve customer satisfaction and company bottom-line”. Your success in achieving cycle time reduction depends upon how often and how strongly you make reference to “business” and “business needs”. These powerful words that do all the magic required. You need right rhetoric to spread the message. Another important keyword here is “market”. Make sure you thoroughly mix up and use the terms “Testing cycle time” and “Time to market” interchangeably. The more you talk about “Time to market” and how automation can directly help it – more you look authentic and convincing.

Forth principle - “Motivations and Incentives”: Believe economists when they say “incentives” drive change and motivate people. How can this not work here? The last but not least of the measure that one needs to be aware of is to provide incentives for people to reduce cycle time through automation. Define performance goals of individuals involved in the automation in terms of cycle time they reduce through automation. Penalize those that question the idea and fail to meet the performance goals. The automation initiatives tend to be successful in their stated goals of cycle time reduction when they are integrated with performance goals of individuals involved in the game – especially automation team. Also make sure (when automation is done by a decentralized team) to define and impose the performance goals ONLY on automation team and suppress any attempts to include manual testing team. After all automation and testing are not related in anyway and it is the responsibility of automation team to bring about cycle time reduction. Right?

If you find these principles and ideas rather strange and are intrigued – one of the several possibilities is that you might be working in a software product organization as opposed to IT and IT services organization. The folks in software product organizations that build software – unfortunately approach automation some what “differently” and cycle time might not be a very familiar term to you.

If you are not from software Product Company and still confused about the ideas in post and you think they are inconsistent or incoherent with each other – do comment. That is a good sign that you are thinking about the topic.

Let me know if there are other ideas and principles that you might have successfully used to reap benefits of cycle time reduction through (and only through) automation – I would be more than happy to incorporate them with due credit.

Shrini

All wannabe software testers out there …

2011-05-25T23:37:00.002+05:30

An anonymous comment posted on my blog read “hi i am non IT background i want to do software testing course in india. if some give me some of the institution in india who teach very well that help u when u get job for software testing and ISTQB test exam.”

Nothing new here, I receive mails regularly asking for suggestion on how to get on a fast track for software testing and get a job. This particular post has three dangerous things that I see – which a new entrant in software testing should be aware of and avoid them.

“Software Testing course” – Let me tell you from my experience – there no such course in world that can make you a tester worth a job overnight. Any course that claims to do is a complete hoax and fraud. Shorter the course duration and taller the claims made by it – deadlier it is. Folks – be aware of such courses that claim you to get a testers job – please don’t fall into the trap. Another dangerous mix or variation here is – claim of “teaching automation or one or more world leading automation tool”. If you want to be software tester - no matter whether you are from IT or non IT background – don’t waste your money and/or time on such courses.

“ISTQB” (replace this with any popular testing certification” – while much has been written by my colleagues and many real life (not so pleasant) experiences out there – I would want to touch upon one thing. ISTQB or for that matter any testing certification WILL NOT teach you how to DO testing and how to gain expertise at it. That is their limitation. Please understand it. Considering certifications as businesses making money – teaching testing and assessing testing skill of a candidate in real time is not their cup of tea. Real certifications and exams that do subjective assessment of testing skill – are not scalable and hence certification people can’t make fast money.

ISTQB and other certifications have done one thing well – marketing. In India, many organizations and recruiters insist one or other testing certification as a must for an entry level testing job. In some others, attaining certifications is a criterion for promotion. Sad state of affairs – though. Sad – because it sets a wrong precedent. It creates wrong expectations among entrants and companies that hire these newbies. It creates a wrong image about software testing in general and at times it trivializes the craft. I have people telling me how easy was to pass a certification exam when they had no prior background or practice about testing. At the most you can get to know some terms used in testing and their fixed meaning as used by a specific group of people. Worst - in some cases they are taught as though those terms have universal meaning and acceptance.

Getting Job – this is third danger in the wannabe’s should avoid. While in some cases you might be lucky get a job using some testing course or certification (hiring process for software testing in India – lot needs to improve – but that is a different topic) – you will not survive long unless you practice real testing – get your hands dirty, feel and learn to think like tester. I can relate software testing skill to that of a musician. If you want to be an expert guitarist – what would you do? Take a 2 week course and a certification (theory) exam and claim a professional guitarist job? As taking a 2 week guitar crash course will not make you a competent – taking a software testing course will not prepare you for a professional career. Similarly learning few words and vocabulary related to music and guitar – will not help you to give performances – though knowing common words and their meanings can be helpful. This is not much different with respect to software testing – getting an ISTQB can help you to know words like “regression testing” and “severity of a software bug” – but those are mere words. Role of ISTQB ends there and real work of practicing testing starts.

In a nutshell – if you are reading this post, is someone who is trying to make it to software testing, want to get an IT job in software testing – here is what you need to keep in mind

Don’t do these things:

Look for or ask for software testing institute that gives a software testing course (shorter the better. One that guarantees job – the gem)
Taking a certification – especially when you have no clue on what is software testing.
Look for job on the basis of 1 and 2 above.
Don’t take a crash course on automation or automation tools in order to get a software testing job.
Take shortcuts

Do these things (few of things that personally helped me in my career) – Try them

Have a time horizon of 1-2 years to the minimum and a complete dedication to learn and practice software testing
Get a mentor – there are many willing to mentor if you show real passion and dedication. Read software testing blogs and engage in conversation. Use social media to your advantage – blogs, twitter, facebook, linkedIn. Make your presence felt as hungry, passionate new comer in software testing – let world notice you. Build reputation.
Practice testing - A platform I recommend is weekendtesting.

Any questions?

Shrini

How IT deals with Test Automation ...

2011-04-12T23:24:00.002+05:30

Here is a short post on how test automation is dealt in IT/IT services organizations.

For example : A typical test cycle:

Before Automation -- (AS IS statistics - documented)

Number of Manual test cases = 300

Time taken to execute these test cases = 10 Person days

Tester's productivity = 30 Test cases per day (derived data)

After Automation -- (Assume 150 of these test cases are automated)

How do you think test cycle would look like?

Number of test cases Automated = 150

Number of test cases requiring manual effort = 150

Time taken for 300 test cases = Time for Automated execution + Time for Manual test execution

= (how much time will 150 automated test cases will take to complete) + 5 days (with 30 test cases per day productivity)

= 1 + 5 (assuming that automated tests run at 1/5th of time of manual execution)

= 6 days

Hence a business case will be developed to justify automation investment

The business case would say - with automation, we can save 4 person days of testing effort per cycle. Depending upon how many cycles of testing are lined up - we can break even the automation investment.

Oh... What a great feeling ... This is what I see all day in and out..

Let me tell you one thing .. executives, managers love this stuff ... so neat - objectively stated in numbers, justifying in terms of monetary terms why we need automation.

But some discomfort and terrible feeling in me ... what you people think - we are missing here?

Programmers make Excellent Testers - Arguments and Counter Arguments

2011-03-13T01:12:00.004+05:30

Janet Gregory’s post on Programmers as testers prompted me to do this post. She mentions in the post that “Programmers make excellent testers”. So I asked myself can I think of few reasons to support and few others reasons to refute the statement. Here I go …

Programmers make excellent testers because:

Programmers understand their code and their fellow programmer’s code /very/ well. Hence knowledge of the code helps them to test /it/ better. Creator knows the best about his/her creation having created it.
Programmers understand /better/ the technicalities of the platform (anything other than “software application under test”) on which their code runs.
Programmers can /write/ /better/ automation code that tests their product code. Writing automation is part of testing right? Or test driven development?
Programmers being closest to the code – can find and fix the bug in smallest time possible – hence can do efficient testing. First opportunity of finding bug in a code is with programmers. If there is a problem in the code – they are the ones that should know about it FIRST.
… I can’t think 5^th one … probably need some help...

Now, let me cross over to other side.

Programmers make not-so-good testers because:

Programmers are /usually/ blind to their own mistakes. Their testing is limited by cognitive bias (confirmation bias)
Programmers /typically/ good at “construction” work – getting spec to working code – not a key tester skill. Programmer testing is more like a cook tasting food made by him/her before serving.
Programmers testing is /typically/ happy path – where would they get time to do “out of the box” as testing, unusual paths, usability, security and performance related tests? (Unless explicitly called out as a part of specification). Programmers /often/ do not see big picture.
Programmers, all through their professional life, work to improve their coding skills – so testing is a part time work for them (which programmer would work to improve testing skills unless he/she decide to become full-time testers)
It is hard for programmers to /think/ like users (many types of them) – their mission of Spec2Code limits them to think in terms of code

A bonus: Typically programmers hate or avoid testing (other than writing automation –which is again a coding work) as far as they can. Many would say “testing is not cup of tea” but I need to do it as we all in a team are responsible for quality. Programmers can’t make excellent testers simply it is not their job (in all).

In sport like Cricket – there are specialist batsmen, bowlers and also all-rounders (who can do both equally well). That is not true for testing.

Someone might say Janet’s context is /Agile/ development model – well, how does that change my “for” and “against” arguments here? How far does that matter?

Update : Pete Houghton (Twitter @pete_houghton) mentioned that debugging is a form of testing - programmers do it well. To me, debugging is an act of hunting down a reported or known problem and fixing it. Thus debugging follows testing - not a form of the latter. By going through the process of debugging repeatedly, programmers gain understanding of how they (programmers) make mistakes and how to avoid them. That also helps them to think about interesting ideas about testing. That is one reason that helps them to be better testers.

BTW, working in customer support - gives probably the best experience for testers, through exposure to wide variety of usage patterns (many of them - out of scope of specifications). Unfortunately - end users do not use applications as per the specification or user guide.

Shrini

Book Review: Selenium 1.0 Testing Tools - Part 1

2011-03-10T19:28:00.002+05:30

In a first of its kind request, few months back someone requested me to do a book review – that too a book on “Selenium”. I wanted to learn Selenium since Long time, to compare and contrast traditional GUI based automation that I primarily worked on.

That is how I picked up this book by David Burns – a senior software engineer in test at Mozilla Corporation (Not Mozilla Foundation) I am not an expert in Selenium – this review of mine as from the eyes of a learner. I have attempted to convey how someone like me trying to learn Selenium would find this book. Those who are with Agile movement and likes might find my comments rather unusual – bear with me. I am from a different world …

Where do I start from? Before I started reading the book – I did some reading on the web. I was looking for a good definition for Selenium. Adam Goucher defined it as “Selenium is, at its core, a set of tools which let you control a browser. What you do with that control is completely up to you. Some frequent fliers use it to reserve aisle seats on their flights, but the majority use it as part of their testing process. After all, the end user of your product is not going to be experiencing the product through their browser as well, not via some wire-line unit testing framework.” I think, Adam’s definition is simply the best one to get started.

I would strongly recommend anyone starting with Selenium to read its history and especially support from Google.

As it happens with all most all books on “selenium” – even this book claims that Selenium tests the application and not selenium is a test automation framework. To me “testing” and “automation” are two different but complementary things and I think Selenium as an Automation tool rather than a “testing” framework.

Here is an account of my experience with the book – chapter by chapter.

- The book starts off with IDE installation and Selenium- “Hello World” kind of test (record-playback). The chapter covers few additional items to sample tests such as adding comments, debugging tests, working with multiple windows. AJAX applications make a sudden appearance in the chapter - not sure why. I feel the section on AJAX is not well connected with rest of the chapter. The section on “Saving Tests” is very brief – could have been expanded to cover exporting or converting IDE tests in other languages (Java) and automation frameworks. A brief introduction of TDD or ATDD or kind of automation that Selenium supports would have been a good way to start a book. Explanation of what is “Selenium” lacks depth and can confuse beginner and experience a like.

- The chapter on “locators” helps the reader to learn about locators. Coverage on DOM, XPath, CSS is pretty good. I would have liked the introduction be deeper. Finding the elements that make up a webpage is the work of these locators. You can locate elements by ID, name, DOM (through java script), XPath and CSS selectors. If you read the introduction of the chapter and summary – I am still confused about this question – “When we can get the page elements through record and playback function – why one needs to bother about locators – so many of them”. I think I know the answer. But I would have expected the explanation to be part of the chapter… “Why one needs to learn about Locators”. If you don’t ask this question – you can very easily follow the chapter and do hands on exercises without any problem and learn about IDE and some tricks

- As it is the case with other chapters so far – the introduction of chapter “Pattern Matching” lacked the depth and connection. It does not seem introduce the chapter by attempting to answer “Why pattern matching? Where? How this is applicable to automation in selenium”. The chapter, as in others mentions “In this chapter we shall do …..” I would have expected the book to cover – why part of these activities. I would have liked to see “how these activities fit in overall landscape”. Usage of pattern matching in general has been explained well. Use of “exact”, “glob” and other nuances of regular expressions, is covered in detail.

I will cover in next posts remain chapters of the book … Do Check back

A Paradox of Reification – A Tool or a Fallacy?

2011-03-07T18:37:00.000+05:30

Reification – Making some idea into a thing [Wikipedia]

Reification is a process (or is an essential element) of modeling – a process of reducing a problem involving multiple variables into single or few variables (For example say “drinking some brand of health drink makes you grow high or makes super intelligent”). If you see a complex (say meaning something that involves many interacting variables) problem made simple (reduced to a single variable problem), be sure to find “reification” hiding around. I would say it is a form of “reductionism”

Strangely enough, the life of Sales/marketing professionals, I think, involves nothing but reification. They can’t sell anything if they become aware of reification and start thinking about model vs. reality. A salesman selling insurance policy would say “Buy this policy – your health is secured – health is reified in terms of money you will get to pay for illness as and when it happens.

Politicians are another group of people who live by reification (I am not sure how many of them are aware of this term, though they might be aware of its effects). A politician is constantly required to create simple models of social life and impose them on his target electorate so that she can win votes. How else you think one can make promises to bring social equality, eradication of poverty by distributing rice at 2 Rupee per Kilogram or provide 70% or so reservations to certain class of people? It is reification @ work. Can any politician dare to think about (giving up) reification fallacy?

See the paradox in the act of reification. As a manager you need to support (and possibly create many new ones) reifications like “Continuous improvement”, “Customer focus (yes this is a reification), “SMART goals”, “Consistent Processes”, “Best practices”, “Metrics”, “objectivity”, “Predictability”, “knowledge transfer”, “business value”, “customer satisfaction”, “transformation” and so on. But when you become the target (rather a victim) of such reifications – you suddenly become aware of the problems of reification – while you may not know or identify by this name or label. How many of us and how often we disagree on appraisal ration ratings given by our managers (yes – your appraisal ratings are biggest and most influential reifications in our professional/corporate life).

Try asking a director, CTO, Head of Testing of a software organization – what is testing? He is most likely to talk about best practices, consistent process, business transformation, customer focus, continuous improvement etc. This person needs to reify ideas and abstractions so that he can do his job. He would not let complexities at the deep-down disturb him and simple picture is what he needs to work with.

Now, note that reification is not simplification. It is a process of abusing “simplification”, resistance to think beyond the world of abstractions. Thus, depending upon which side of the fence you find yourself in most of time, reification becomes a tool or a problem or fallacy.

In short the paradox is – you can use reification to work (mostly managers and people who need work with simple abstractions) as a tool. When it bites back to you as a victim (as in appraisals) – you would hate it and fight it as a type of “logical fallacy”.

Why wouldn’t Sales/Marketing folks think about reification as a fallacy? Do they?

In Pursuit of Quality, let’s Call Quality as something else

2011-02-23T19:41:00.000+05:30

The theme of this year’s Euro STAR conference is “In Pursuit of Quality”. A great theme but an ambiguous I would say. I wonder what could have been the meaning/interpretation of the phrase “Quality” here. One might say “your own definition of quality” – in a group, it is a problematic one with the missing context.

It is interesting to note many in software profession acknowledge that the phrase “quality” has many meanings – different people at different instants of time paint/experience quality differently. Many admit that objectification of “quality” should be avoided. Yet, it is a rampant practice to portray “quality” as having some universal objective meaning that everyone agrees. Popular uses of the phrases like “Quality Software”, “Cost of Quality”, “Quality Engineering” etc., are examples of such practice. For many, “quality” is just a bucket where everything related to expectations of the users can be lumped together. It is one term that evokes a great array of human emotions from extreme pleasure to awe, from frustration to sense of achievement.

Let us imagine “quality” in terms of how Quantum Mechanics describes quantum states of subatomic particles. This “quality” is a like a subatomic particle in quantum superposition state and represents many meanings (or states) at any given context (or Einstein’s space-time). When one makes a swipe at this term and uses it with respect to a context saying “this is quality stuff” or “this quality is unaffordable” or “quality is free” – the wave function collapses and a particular meaning of quality shows up.

As Michael Bolton explains while posing a testing challenge about agile teams -

”For example, in a particular context, if we want “quality” to mean “bug prevention,” let’s say precisely that. Then let’s recognize the ways in which certain approaches towards preventing bugs might represent a threat to someone’s current interests or to their personal safety rules. If, in a particular context, we want quality to mean “problems solved for customers”, let’s say precisely that. Then let’s recognize that there are many approaches to solving problems, and that some problems might be solved by writing less software, not more. If we want “quality” to mean “many features in a product”, let’s say precisely that. Then let’s recognize how “many features” can satisfy some people—while adding complexity, development time, and expense to a product, thereby confusing and annoying other people. In other words, let’s use the word “quality” in a more careful way, which starts with deciding whether we want to use the word at all.”

That is to say if you want to say something to indicate quality – say it (without using the term quality). In this way, you can prevent misuse of the term for the better. If you want to pursue quality - say what you exactly mean so that people can understand you without confusion … Probably people will stop using phrases like “Cost of quality” or “Quality plan” or “Quality Assurance” or stop using quality with another set of abstractions like “Creating business value through quality software”

To repeat Michael “Quality isn’t a thing, but rather a complex set of relationships between products, people, and systems. We should be calling quality as something else that states what it is in a context rather than calling it with one word always.

I have a suggestion. Instead of using QUALITY – let us use words made of up characters in the word “QUALITY. For example - you might say LITYQUA of this software is poor to indicate usability or you might say TUAQLI of this software needs to be worked on to indicate security or cost. Once thing is sure to happen when you do this – people will say – what? When you explain them the meaning or interpretation of attribute that you are trying to represent is using the “jumbled” up word – they might relate it to that one thing and only that thing. Thus misusing and lumping “all” in to Quality, stops – I hope.
Try saying some other term when you want to say “quality” – difficult?

Two versions of being practical...

2011-01-30T16:28:00.002+05:30

My boss tells me always that my ideas about testing are impractical and esoteric. I find it hard to understand why he thinks so. I believe my testing and automation ideas are as practical as one can get because I can demonstrate the practicality of the ideas by "doing" testing or automation. I suppose being practical - should be mean "demonstrable". I can show how my ideas of testing can be and are practical. Boss is not impressed.

Probably, my boss has another version of practicality. It goes something like this. Regardless of what subject matter or technicality of testing as it is practiced in real time - there is a social, political, business side to it. Many things that testing practitioner believes to be true are not possible in real world due to political and non technical aspects around testing. Under such situations - being practical means changing your testing philosophy and approach to suite the context of the testing world. Harsh word for such adaptability could be "compromise". If you are surprised or questioned instances like stakeholder seeking quick benefits of automation (something that reduces cost of testing), counting bugs, counting requirements, using bug metrics to measure testers, expecting testers to find all bugs, doing 100% testing - you probably are practical in another sense.

Living with many irrational notions about testing and trying to do what is possible, taking the world as it comes without challenging it or fighting the problems is the version of practicality - probably my boss believes.

Both of us could be right or wrong. How do I know?

This reminds of me of a Jerry Weinberg's saying (para phrase) - "Instead of calling something as irrational or illogical - consider it as rational or logical from another perspective or set of values".

You would like to call "testing" as an engineering pursuit? Wrong... it is becoming (it was from the beginning but now it is showing up more clearly) more social. Testing works in social systems and is affected by cultural, emotional, economical issues.

Time to start studying economics, social science to reason behavior of people in testing context. It is not rational at all times....

Shrini

What type of tester are you?

2011-01-13T22:47:00.000+05:30

I was assisting homework of my 7 yr old daughter in her science assignment. The assignment was to collect pictures of living and non-living things and make a collage. I thought about how one could approach this work and have fun while doing it. Being a tester, it is hard not to see connection in a work like this. Here is what I thought….

Let us say two testers “A” and “B” are given this homework assignment of creating collage of living and non living things.

Tester A: Living and non living things? Just collect few pictures of animals, plants, humans - paste them as living things and collect pictures of cars, buildings, mountains, bridges and paste them as non-living things. Home work done. Test pass and on to next thing…

Tester B: While thinking about traditional ways of looking at living and non living things, this tester also questions his own understanding and meaning of “living” and “non living”. He looks around, reads, discusses with colleagues about these things. He, then ends up with a broader list and accompanying narration.

What is the difference between these ways of approaching a task at hand? Tester A is more focused on “completing” task with minimal mental engagement about the settings of the task. He takes “widely accepted” meanings and understanding of the things and optimizes the details of the task. Tester B, starts with questioning the premise of the task, meaning and understanding of the things required to complete the task. He seeks a broader understanding of task elements and attempts to question biases, prejudices of his own and others around. Though on the face of it, it might appear that Tester B will take longer time to complete the task (homework), through practice, he might able to finish the task nearly at the same time while understanding of broader aspects of the task and clarifying biases and assumptions.

Today’s testing industry promotes type A testing mentality. Type A testers eventually will become “commodity” (with price tag 20 Dollars per hour – can execute 30 test cases per hour etc). I am not sure that is a good thing or not. If you are a manager or someone who manages testing as business – you would probably have (or would like to have) more of Type A testers as such people are plenty. Simply, take truck load of freshers from college, run them through 15 days of training on testing vocabulary, make them memorize some domain stuff (banking, insurance, telecom etc) , teach QTP – there you go an army of (Type A) testers ready for deployment (hence billing).

If you are a tester looking at long term fulfilling career – you would want to take path B – practice to do things differently, challenging status-quo, thinking broadly about the problem and its context etc. To become “Type B” tester you need practice – practice of doing testing (weekend testing is an excellent platform or training ground). I am afraid there is no shortcut here – no 15 day testing courses to get there.

There is a catch here… Type B testers are mostly rebels and free thinkers and often want break status quo, create new paths, explore new territories. Much of our industry is not very nice with such folks. Very few can tolerate “ever questioning”, skeptical and rebellious tester in the team. In a control freak, military kind of set up where key focus of the job is to “follow instructions” or “getting job done” – Type B testers get frustrated. Some get converted into Type A or leave the organization. Sad situation but true in many cases. I hope that one day either rebel testers will learn to live harmoniously in a hostile, confirmatory environment or industry recognizes the value of these “problematic” folks and make space for them.

“Testers are like lighthouses at sea shores, headlights of an automobile” says James Bach (paraphrase). Good testers, apart from completing the task given to them, engage in parallel about “meta” things for better and deeper learning. That is a good thing (can I say for testers not for folks who manage testing as a business?).

Did someone say “Type C” tester?

What does that mean?

2011-01-04T23:49:00.005+05:30

The other day I visited a science exhibition at my daughter’s school. As I was passing through an array of models and demonstrations, I could see every student was enthusiastically trying to draw every visitor's attention to his/her piece of work. Everything sounded good except one thing - the students were seen literally reading some sort of script to explain their work. Some smart guys memorized the stuff well so they did not require any hand written (mostly) note to assist them delivering the script but many did keep one by their side. It was hard to find someone who “knows” what he/she is trying to explain rather than forcing the stuff through using pale dialogue delivery.

There at the corner, a bright spectacles boy waved his hand calling me to see his work. I walked up to him. He was demonstrating “law of conservation of momentum” through a model that roughly looked like the one above.

After a customary greeting, he read through his memorized statements about law and looked at me with hopeful eyes asking if I understood anything out of it.

I asked him “what does this law tell you”.

He promptly, this time with more energy and desperately attempting to sound confident, repeated the law (the statements) like a tape recorder.

What do you understand from the term “momentum”, I asked him.

He was totally unprepared for the question (missing in his notes - missing test case?)

He quickly recovered gently tapping his spectacles said “It is simple sir - it is mass times the velocity”

I know that is a mathematical formula for momentum.

Student finally found something that I agree with him and said “Yes. you are right sir”.

At that instant, I remember my hero Richard P Feynman - the (most) curious character and asked him “But what does that mean? Mass times velocity?”

This time, student was not expecting that question at all.

I moved to another demonstration where they were showing double helix model of DNA and the girl explaining the model, gave a 5 minute non stop explanation of what DNA is and what genes are.

Again, some question “What does that mean” to the explanation “genes carry genetic code so that characteristics are passed on from species to species” got everything to a grinding halt.

I became pretty notorious from that day in among my daughter's friends about asking such silly questions about things that are taken for granted for the students. My daughter later told me that her friends were impressed with my “latest” knowledge of science and my questions.

Later, over the week end, one the students (harassed by me in the exhibition) paid a visit to my home apparently (I thought) to take revenge on me. He, incidentally was my daughters classmate and asked me “what do I do for living”. I said to him “I am a software tester”. While he did not understand that very much, he wanted to know how learn things. I explained him about few things that I learned of late as a student of software testing (than when I really a student to learn science). He appeared to have understood some of my approaches.

When he was about to leave, I gave this book “ Surely You're joking Mr Feynman” and told him - "learn science like this guy". I was happy that I introduced Feynman to a student hoping the “Feynmanism” spreads in the school.

I wish a very happy and prosperous New year to you all

Shrini

Fish baking story ...

2010-09-17T20:09:00.000+05:30

This was a story that came to my mail box.

A little girl was watching her mother prepare a fish for dinner. Her mother cut the head and tail off the fish and then placed it into a baking pan. The little girl asked her mother why she cut the Shead and tail off the fish. Her mother thought for a while and then said, "I've always done it that way - that's how grandma did it." The girl was not satisfied with the answer, and went to visit her grandma to find out why she cut the head and tail off the fish before baking it. Grandma thought for a while and replied, "I don't know. My mother always did it that way." So the little girl and the grandma went to visit great grandma to find ask if she knew the answer. Great grandma thought for a while and said, “Because my Grandma told once that the baking pan was too small to fit in the whole fish”.

In most cases we fail to challenge belief systems and assumptions.

This is true about many ritual centric processes that we are asked to do in testing today. If you dig deeper, you would be surprised like this little girl. There are no real reasons to do certain things. Typical answers might be “my boss wants it that way”, “this is how our clients want” or “this is how I have done it always” or “this is in that book” or “this is how the consultant who we hired told us” or “this is how it always appear to work”.

One trademark quality of a skilled tester is skepticism and ability to think beyond rituals – ask “why I am doing what I am doing”.

Are you asking this question frequently in your job as tester?

Shrini

How can Mathematics be the Language of Nature?

2010-08-05T23:30:00.002+05:30

I am reading this wonderful book about Physics (and history and philosophy of Physics) "Tao of Physics" by my all-time favorite Physicist and author Dr. Fritjof Capra, a contemporary of Werner Heisenberg. Incidentally, Dr. Capra's other famous book "Turning point" has really introduced me to the concept "systems thinking" while I struggled to grasp the subject by reading another influential legend and Guru of software (testing) Jerry Weinberg. James Bach and Michael Bolton introduced me to the subject of "systems thinking" as science of complexity.

Well... coming to the point, this post is not about system thinking but about a paragraph from the book "Tao of Physics". Here goes the text ...

"... The words of our language are thus not clearly defined... Science, on the other hand aims for clear definitions and unambiguous connections. Therefore, it abstracts the language further by limiting the meaning of its words and by standardizing its structure in accordance with the rules of logic. The ultimate abstraction takes place in mathematics where words are replaced by symbols... In this way scientists can condense information into one equation, into one single line of symbols. The view that mathematics is nothing but extremely abstracted and compressed language does not go unchallenged...mathematics not just a language to describe nature but is inherent in nature itself. Originator of this belief was Pythagoras, who made famous statement - "All things are numbers" …

This puzzles me ... considering(or conceding) that mathematics is an abstract language used by scientists to represent theories, models and scientific thought in a precise and unambiguous way, how can such an abstract language be suitable to describe nature (even in parts) ?

Nature is live, rich and multi dimensional where as mathematics is at other end of the spectrum - abstract, precise and single dimensional. Did Pythagoras and for that matter Galileo make mistake by saying "book of nature is written in the language of mathematics". I am certain that mathematics does not be to be interpreted where as nature and its manifestations need to be. Am I making sense?

Nature of Problem(s), we testers (should) investigate (or solve)

2010-07-08T12:10:00.006+05:30

I am puzzled about the nature of problems that software testers, today are juggling with or should be losing their sleep over - more than the problems themselves. As humans, we love simplicity (not sure about about other species - not sure if there is any way to check if they love simplicity too). In every walk of life, we seek and embrace simplicity. Many experts tell the stories about how they have solved mysteries of the world through and give us the tools, techniques to conquer/dominate the world around us. We now have quantum computers even though there is barely anyone who has fully understood Quantum theory. We can make machines, apply our partial knowledge but make a mistake of claiming glory of scientific knowledge saying “this theory is true as I have experimentally verified it and there are practical applications”. The mistake that we make is about “completeness of knowledge” – making a machine based on a theory does not amount to complete understanding of the theory.

Alfred North Whitehead, the famous English mathematician and Philosopher warned us “Seek simplicity and learn to distrust it”. Simplicity is so tempting that it fools our minds and thinking and makes us glued to it. It appears to me that we have not learnt to distrust simplicity at all – worst many even does not know that we should “doubt” it. There is “banana” principle, attributed to Jerry Weinberg. For many things like simplicity, we have only learnt how to start but have horrible lost the trick “when to stop”.

Many in software world believe that software testers are expected to answer questions that, simply lead to “pass” or “fail”, “works” or “does not work”. These questions are like asking “is 4+4 =8? Is this an isosceles triangle? Is the temperature of this body 86 degree centigrade? What is National Stock market Index today?” Engineering, mathematical and scientific questions right?

In our software profession, we love to pose “engineering/scientific” questions that lead to binary answers. After all we are software ENGINEERS and software is ENGINEERED. Hence I started this post with simplicity and our obsession with simple models of complex things. What is the true nature of questions/problems that software testers investigate and solve? Let me make a bold conjecture here. I think they are similar to the questions that we pose in the realm of social sciences. For example, questions like “Is homosexuality linked to or mainly due to some mental illness?” or “Should burqa be banned in public places” or “Is there a God”. These questions are asked in a broad social setup and are linked various material practices of the people in the society, about their culture, history etc. Much in a similar way, software operates in a social context and people are mainly concerned about how software alters their day-today work. The nature of questions/problems requires non-binary, qualitative answers and solutions. The questions about quality are based on emotional, subjective feelings (there is no binary 0 or 1 about software quality).

As Michael Bolton says “decisions about quality are always emotional and political”. How can there be binary mathematics in emotions and politics? That is the point these software engineers and process people miss and insist on quantified definitions of quality and other abstract entities associated with Software. It appears to me that they are constantly conditioned by the business environment to believe so. Coming back to simplicity, software engineers (process/CMMI/Six Sigma) people definitely think that the problems that software testers/engineers solve (investigate- actually) are purely engineering/mathematical in nature. Hence they believe in absolute answers, causality, objectivity and certainty.

I think they are wrong !!!! When do we learn when to stop saying “banananananana..?” While you are reading this and interested in debating about “nature of testing problems”, many budding software testers are happily busy in churning out hundreds and thousands of test cases and blissfully shouting “pass” or “Fail”. Well, you might say do they have options … I am sure they do.

Shrini

A movement called weekend testing – What it can do to you and your testing career?

2010-03-07T20:33:00.003+05:30

An extremely contiguous phenomenon is sweeping in some circles, communities of software testing; it is called Weekend testing (twitter @weekendtesting). Presumably, born out of few young minds (Ajay Balamurugadas, Parimala Shankariah, Sharath Byregowda, Manoj Nair ) and influenced by context driven software testing philosophy. In a short span of time, it has traveled from Bangalore (the home of Weekend testing) to Europe and the US. So it is globalized movement. As testimony of its popularity and the value, who's-who in global testing community have acknowledged it

It is hard for anyone who is a passionate/skilled software tester not to be associated with it. Once you are in, you keep going and loop in others. Such is the power of the concept and the way weekend testing works. I would probably equate (not mathematical sense) weekend testing to open source software development. What drives these movements? Passion, Desire to learn, Demonstrate skills in the real time "On Demand". The last point is something that is more apparent in weekend testing movement.

What makes weekend testing - a movement or a revolution in its infancy?

The objective:
Practice testing (or software testing) in real time with like minded (I wish if some dis-like minded folks join it – we need fight confirmation bias always as testers. Being vigilant about fallacies of "you will see what you want to see or you will fall in love with that which confirms what you know").

This objective is very powerful and has enormous potential to grow big. Keywords here are "practice" and "real time". Now you can see testers saying "I was practicing testing" just like a musician doing is "Riaz" or the sportsman sweating it out on the ground. A manager would call few of his low performing testers and say "Go and get some practice of testing in week end testing sessions". While his testers are training, the manager can check their progress by viewing the session transcripts and related blogs posted elsewhere. Transparency to the core!!!

The structure

Here is a generic format of how a weekend testing session is arranged – more detailed description can be found here

A group of testers sign-on to Skype or Google talk and get started with their testing at a specified time as announced on the weekend testing website. One of the testers assumes the role of facilitator giving the mission for the session and moderates the whole thing. Mission of the session is itself tested first – meaning it is scrutinized for possible ambiguities and a refined one is adopted. Testing starts bugs, issues are logged in an open source bug database and followed by a debrief session. The session transcript is posted with results for everyone to see and learn further. It is almost free for those that have computers and internet connection. Weekend testing thrives on usage of open source/free tools. It is more or less self-driven – all that was required to set up an initial structure for testers to get started and posting the results and transcripts of the sessions.

It is a terrific place to learn With so many willing, skilled and available testers and coaches around what you are learn is nearly unlimited. With a wide variety of products to test, variety of testing missions to try out – there is something new to learn in every session. With people across the globe participating, you get access to an interesting cross section of skills, cultural legacies to learn from. You can improve your communication, writing skills in a friendly yet mildly critical environment. People here tend to me cooperative and more interested in seeing everyone succeed. It is the genuine interest in other's growth and improvement that makes weekend testing a unique platform for greenhorns to gaze around.

It promotes "community" certification/acknowledge - not given by any "standards" like bodies. Needless to say the people associated with the movement are known for their "critical thinking" and would go to any extent to be "open for scrutiny/challenge any time anywhere". That makes this movement strong and really unbeatable.

It makes testing demonstrable in its deepest details possible. For long, people influenced by context driven testing have struggled to explain the power of exploratory testing. They have been trying to argue for skill in testing as opposed to so called best practices and process driven testing. And there "test cases" the ubiquitous building blocks of testing and the questions that were often asked "how can you do testing without test cases, How can you repeat testing? How will client accept that we tested?" Now we have a proof in a public movement and "open for scrutiny anyone anytime" – You can literally see testing happening in front of your eyes. What more evidence you need?

The People

The founding members of this movement are, at the core, very dedicated, passionate software testers whose sole distinguishing qualification is their appetite for learning and become skilled software testers. That drives them to do things that they do. They are constantly building their reputation through demonstrating their skill and learning constantly. So, hanging around with these people can be very dangerous. They will make you like them. The weekend community is very aptly mentored, coached and supported by people like Michael Bolton (no… not the singer or the "officespace" character), James Bach, Cem Kaner, Pradeep Soundararajan, Matt Heusser, Jerry Weinberg, just to name a few.

We (notice, I am using this "we" after a long hesitation as I have not contributed enough to include myself in this movement) are thankful to these leading lights for encouraging the movement and advertizing it.

The Future of weekend testing – The way I see it.

Few thoughts …

There will be many chapters of weekend testing all over the world. There will be weekend testing conferences to the scale of STAR conferences in the US.
Big universities teaching software engineering will recognize this movement and will have representatives from them
Commercial product companies will engage weekend testing folks to test their products. With the support of crowd sourcing companies like utest, week end testing will gain authenticity and grow.
Weekend testing will then get into weekdays as people want more of it – we then have to look for another name to call and identify ourselves … But that is an interesting problem to solve

I am looking forward to participate and contribute this movement …

Update (10th March 2010)

Parimala Shankariah has brought my notice that Santhosh Tuppad, yet another promising testing star in the indian software testing horizon, who was recently featured on utest - has been a very contributing member of this movement. Thanks Parimala for point this out ....

Santosh ... nice to know about your contributions.. Keep going... Look forward to participate in sessions with you.

Shrini

Man and the machine ....

2010-02-22T13:41:00.003+05:30

Can a software be produced and maintained using a machines in a factory … like Google does?

Notice, with their machinery and factory setup, they have been able to cut down the testing cycle to 4 minutes... Sounds unbelievable .. I think it is ..

Google produces Innovation machines and IT vendors set up factories… So back to Factories and start oiling machines. Probably we should change IT designations to map roles on assembly line shop floor … - foreman, supervisor (this is there already in some sense), operator, inventory manager, assembler…. As few expect, IT work will eventually be a factory job. For one, my wife and kids would be happy. I can then work in shifts and come back home at sharp, say 5:30.

It is interesting to watch what Google is doing ….though their innovation factory recently had an outage…. With “Buzz” having “bugs” related to privacy...

You know what … as a tester, bugs make me happy… that is an eccentric , one sided view …especially with those who boast “machines and factories” of software. After all we are very far from the day when truly software will be produced in machines, requirements will be read from humans through bar-code type readers, we would be able recall software as cars (instead of sending a patch) and there would software garages.

Dreaming …!!! Someone woke me up... It is a brand new Monday….

Shrini

I’m blocked… but why?

2010-02-15T00:53:00.002+05:30

"There is no such thing as writer's block for writers whose standards are low enough." American Poet William Stafford

I am blocked from doing any interesting writing for over 2 month or so. And this post is all about the condition of being blocked in writing.

As paradoxical as it sounds, I have not been able to write anything to my satisfaction so that I can publish. In the world of writing, they say it is a condition called "writer's block". It's a terrible feeling for me as block keeps haunting on every attempt to complete a half finished work or a new one. Here I am sitting with over a dozen of unfinished posts, like a kid confused about which toy to pick and play. It is funny that not long ago, I was chatting with Parimala Shankaraiah, a (or the?) curious tester about this condition called "writer's block". I suggested to her that when blocked, try writing about it. That is what I am exactly doing now. To illustrate the influence of WB on me, I am planning to keep this post small and focused on the topic and close it as soon as possible.

For non starters, WB is a temporary mental condition of extreme self censorship leading to rejection of semi finished parts of writing as this reference suggests. Keyword here is self imposed censorship. Those getting stuck in WB, can take conform in William Stafford's statement consoling themselves with pat on their backs for their high standards of writing.

Why I am blocked on writing?

While I am not short of ideas to write, when I pick up something to write after writing few paragraphs, I start getting a strange feeling that I am repeating or the plot of writing is hypnotically drawn into a predetermined destiny. That is where I stop writing. Not sure what is happening.

WB, like inattentional blindness appears to occur when you are ultra/super focused on a topic so that you literally stop thinking. I am not sure, my state of WB has anything to do my focus (or lack of it) on writing. As defocusing tactic, I switched to twitter and tried my hand there sufficiently enough to get an attention from none other than Jon Bach. That still leaves me high and dry and I am back to WB. Another thing that I suggested to self – stop writing but speak (do pod casts) or draw/paint (cartoons etc). I need to try these channels to see if they help. I think what happens in WB is one (of several) channels of expression of thought gets blocked while keeping others intact.

One thing is sure; I am under pressure of not being able to write even though I have many "good" ideas to write. Some say WB leads to lack of motivation or happens due to lack of motivation. I am pretty sure that is not the case with me. Anxiety, fear of being ignored or failure certainly is back of my mind.

Aiming for High standards? High degree of censorship? Yes, that appears to be the case. I reject lots of writing (pile of trashed pieces of writing) as they don't qualify my own criteria to be good enough to be posted.

I don't agree to generic prescriptions like Lack of preparation, reading, and picking up boring topic leads to WB. My experience of WB is that I am blocked on those topics I deeply interested in and dying to writing something significant, lack of preparation/reading might be an issue in some cases but the in case of the ideas that are pretty original, I often avoid reading to get my flow of my "unbiased" ideas.

Well, now I am suddenly realized that I am supposed to suffering from WB…let me abruptly end this post and start mulling over the cures for WB.

Oh!!! What a relief. This post comes to an end. Have I overcome writers block? Over to you…dear reader!!!

Note: I am surprised to note that there are other types of writer's blocks such as this and this. Here the word block is intended as "collection" or "store" of useful writer tools. Same name different connotation.

Why GUI Test Automation is Popular and Tempting? Part I - Business context

2009-12-22T00:14:00.002+05:30

It is rare to someone on testing in IT world not be aware of "QTP, WinRunner, Rational Robot etc ". For the most in IT world, the word automation is synonymous with GUI automation. Perhaps some even might ask - "Are there any other forms of automation other GUI automation". Lots of how IT world sees automation has to do with how they see testing itself. You might have heard this "celebrated" statement - "Regression testing is more suitable to do automation (read GUI) so that we don't have to have our IT staff do that unnecessary testing manually".

To my surprise, having been associated with IT for many years now, people here still believe in perfect requirement, perfect test case and hence "doing it right first time". In contrast to this, software product world (likes of Microsoft, Google) treats test automation and hence testing in a total different way. These two probably operate from two extreme ends of automation spectrum. While discussion how automation is perceived and realized in IT and software product world can be a separate post in itself, I propose to elaborate here my views on the popularity on GUI Automation in IT.

Testing is necessary evil and should get done quickly

This is the reason #1 that drives GUI automation in IT. Business spends on getting IT application features developed and hence often fails to see a reason why they should spend on "testing" at all. Business feels that IT should get their acts together, get aligned to business (hence IT-business alignment is often a well understood but poorly executed theme) and use the money towards delivering business value. I even have heard some business folks saying "We pay for the application features not for their testing". Hence funding testing becomes IT's problem. That is why they look for solutions to reduce testing (or spend on it). As simple as it sounds, IT pays for testing (not the business)

Testers (read business users or analysts) are hard to find and they hate regression testing

Traditionally, in IT business analysts (aka BA's) are hot commodities. They do most of what we call as "testing". With outsourcing wave sweeping IT, some of BA roles are taken by either BA's from outsourcing vendors or some junior tester working under supervision of BA through highly scripted test procedures. The problem is that these BA resources are few and are in great demand. But as software releases happen, changes come in – testing (regression testing) has to happen and BA's hate regression testing (or all of testing). So, an IT manager needs to find a way out to carry out this regression testing somehow.

How to get testing done fast and cheaper?

If you want to do something fast, give it to machine or a computer program - goes a typical approach of an IT manager. So automation could be the answer. If you position automation as an aid to reduce the cycle time to market chances of you getting funding is very high. Surprisingly this claim is so powerful that when made often goes unchallenged. The magic word "reduction of cycle time" (that too without any qualification) is tempting for many that it will make them to completely overlook bottlenecks in development (programming), bug fixing, investigation and any other non-testing activities. Thus speaking in terms of business language is so important in proliferation and apparent success of GUI automation in IT.

With GUI automation, you can easily make a business case for automation.

To me, the terms like test cases, regression testing, test cycle time etc are the ones that business stakeholders appear to understand very well. As compared to unit tests and other forms of non GUI automation, you can make your case for investment in GUI automation to business stakeholders. Let us say you have 5000 test cases for a regression testing to cover that takes 5 person days to complete, automate 50% of them you can straight way knock off 50% of your testing cycle time reduced. This claim due to its simple math and logic makes a compelling case for GUI automation or simply "automation". What else can be more appealing than investing once in automation (for the sake of making the business case and further forever – down play the maintenance costs) and saving on a percentage proportional amount of automation achieved for 100's of cycles in the future?

Tool vendors help you by talking in business language

Enter tool vendors – with the fancy looking charts, scary quotes like "more than 60% of project cost goes to testing, what are you doing to reduce it" and likes Gartner's and Forrester's to support the claims with numbers, life for an IT manager in charge of testing has never been so easy. Just buy a tool, engage an outsourcing vendor to do GUI automation – all your testing worries are over in one go. Dreaming never ends here. Take a look at any brochures, websites, commercial literatures – they are full of business terms like ROI, Cost of testing, business impact, and Time to market and so on - making a perfect connect with those that matter in making decisions about spending.

Now try doing this (or these) with non GUI tests such as API tests or xUnit framework based unit tests – what will business say? What will IT managers say – "well, developers got to do that testing anyway"? So there is no apparent business case to make so sell automation. Writing unit tests will not speedup your testing cycle, will not reduce time to market, might reduce some portion of that "boring" regression testing – but. So you have no financially oriented incentive to do any automation that is non GUI. Since GUI automation paradigm as so much hardened into dogma – very few challenge it.

If you are the one working in a software product company – you would say "do less of GUI automation, they are very brittle and costly to maintain". You would be surprise to know/hear – the opposite works (or at least appear to work) in IT world.

It is "power of talking in business language, very few challengers for the claims made about automation and excellent support by tool vendors" that makes GUI automation popular and tempting to attempt. Even if you fail – you can blame it on almost anyone or anything around – test cases, tool, automation folks, application changes, changing requirements, test data, operating system patches and so on.

To be continued in Part II

Eurostar 2009 – A Trip Report

2009-12-12T00:58:00.001+05:30

I am just back from EURO STAR at Stockholm. As a ritual, and like an obedient delegate and speaker, I am narrating the experience. It was my first visit to Sweden, home for Alfred Nobel. The conference started with a bang – welcome speech by Dot Graham, the program chair for 2009 edition of biggest European software testing event. After her being first program chair at 17 years ago, she still appear to breathe same energy of someone who is attending her first testing conference.

Lee Copeland, the veteran of testing conferences in the US, who followed Dot as a key note, was at his usual calm and composed best. He spoke on next few big things in testing- a good list to watch out for. His talk also contained slides on good books for the testers to read. Probably it has become a habit for him to promote his book on "Test design" in each of this talk … (I have heard 3-4 of them). He might call it as "Shameless plug" or "No… I didn't mean it include it here", I think he should stop doing it as the book sells on its own merit. Thereafter there were parallel tracks and everyone had a choice to go to the talk that they wanted. This time I like many others, had the power of twitter to spread information about conference, with the hash tag #esconfs, I was one of those many active twitters using tweeting about the talks, and happenings at the conference. Check out all tweets related to this conference here.

As it happens in every conference, much of the action was happening outside the track sessions, people meeting, exchanging cards, and fiercely debating on topics that they hold to their hearts. There I was, roaming around trying to see which group to join. Eurostar Test Lab was in interesting addition to this year's conference, run by James Lindsey and Bart Knaack - was a big distraction (constantly pulling passionate testers towards it and away from the track sessions). I spent few hours of testing and spent some good time with James Lindsey one of my favorite exploratory testing proponent. I logged few bugs too. It was a good experience. James reminded me to explore and find out when I had a question about a feature of the application that I am testing. Here is a report from Michael Bolton on the lab

Michael Bolton was an attraction, true to his image of great speaker and testing wizard. I rarely saw him alone, always surrounded by few people, Michael was just keeping them engaged with his teaser questions and people kept coming to him. His talk - "Burning issues of Day" was a presentation inspired by collection of white board statements of conference attendees, presented in an immaculate MB style. He took mock at several testing folklores and argued convincingly against standardize and over structured software testing practices. One thing that was really remarkable in that talk was Michael's ease and confidence - with which he entered the podium and started the presentation. He was at his total ease. Many wannabe speakers should watch him starting a speech and finishing one. To me, Michael gave an example of "what it means to be being oneself". Good learning there.

It was great to meet/see likes of Jonathan Kohl, James Lindsey, Fiona Charles and Ray Arell. In the conference made few new friends – Johan Johansson, Tobias Fors, Daniel and others. That evening we got together with and headed for a dinner and followed by a meet up at a friend Pabalo's house. That was a chance for me and others to have a go at musical instruments. I settled at Drums, Michael and Pabalo took guitars. The concert went until 12-1 Am. I decided to call it a day as I need to catch up with some sleep before my conference talk next day.

It was a big day for me. Getting to speak at Eurostar has been my dream. My talk was about metrics and how they can be dangerous if used inappropriately. I wasn't nervous and was sure that I will speak my heart. I requested Ray Arell and Fiona to help me with few photos. Michael's and few others presence at my talk, helped. I managed to finish my 15 odd slides at one dot – 39 minutes. Few questions and it ended with applause. I was greatly relived doing my duty at the conference. I plan to write a separate post about that later.

I met this guy Daniel in the conference– he did not carry his business card hence do not know his coordinates. Daniel and I argued for hours about the nature of the software – physical or abstraction. Daniel insisted that software has a 3D physical existence. He even said that he had confirmed this with 2-3 physicists he knew. Software when reduced to its "atoms" or basic units is some sequences of 1's and 0's that are stored on a magnetic material. When we power the computer or a computer like devices, electric current passes through transistors and other electronic components and bring the software to life. Daniel argued that hardware components that held the sequence were real and physical. We could never take the argument to the conclusion. But it was a good argument.

Third day of the conference started with keynote on agile adoption at Intel and experience sharing by Ray Arell. It was a good session with lots of advice on getting agile right. I had to do lots of running between office and the conference venue (yes, I had some official meetings set-up on that day). I track-chaired a session by Aslak Hellesoy on "Cucumber". The session was good and well received. The name cucumber look puzzling to me as why "Cucumber" – Aslak clarified during the presentation that it was the name suggested to him by his fiancé/girlfriend. It is become fashion to name the tools/frameworks companies by totally unrelated names. Steve jobs success with Apple continues to inspire many to name their creations with the names that you typically would never make a connection. In the evening there usual conference rituals like panel discussions, prizes, vote of thanks and announcement of next year's track chair – John Fodeh. Zeger van Hese's paper wins the best paper award, Naomi Karten wins the best tutorial award and Michael Bolton wins best bug of conference. Michael wrote about it here.

In all, it was an excellent experience of being at Eurostar 2009 and at Stockholm. I will carry many memories from it for years to come… Thanks Eurostar!!!!

Don't forget to check out the photo gallery

Check out few reports about this conference

Female Funtestic Fanatic

Rikard Edgren's Test Eye

Star Tester issue 45

Note: Narrating a story or an event like a conference is tough, I think. I attempted to do it see if you like my story.

Defining the problem : How business works

2009-10-15T01:05:00.003+05:30

Lee Jack, a software testing consultant/IT services Sales Manager is tensed about his proposed meeting with Mark Johnson, a Senior IT managers for a large IT group. Mark, after a cold call, is looking forward to sell testing services to Lee’s group and hence to his entire IT group. Here is how the conversation goes.

Lee Jack: What is your problem statement?
Mark Johnson: Not sure. We recently had a cold call – you say you provide Testing services what you can offer me?

LJ: You need to focus on Independent testing
MJ: We have been managing with developers doing testing for last 6 years – have no problems whatsoever. Why do I need to bother now?

LJ: May be your developer testing is not proper or inadequate or it is costly?
MJ: How can you say that?

LJ: Tell us about production defects and quality of your applications in general…
MJ: I told you already. We have no production problems that we worry about. Our site is every now and then goes down. We wait for 30-40 mins and start over again everything works fine. Why bother?

LJ: Do you say your applications are of high quality?
MJ: I would say our applications are of “good enough quality”. I get what I pay for and I have the quality for the application that my users care for. So far so good.

LJ: What are your “business plans” going forward? New growth etc? What you seem to be saying “good enough may not be good in the future?
MJ: May be. I don’t think that far ….

MJ; Have you thought about cost of testing? Do you have any mandate from your boss about reducing IT budget and hence reduced budget for testing?
MJ: Oh!! Yes. We have taken care of the larger part of cost problems. We have nearly replaced all our commercial software stack by open source equivalents. Same is true even for tools. So we can say we run our entire shop on open source tools. That is our achievement.

LJ: Tell us about your development /testing model ?
MJ: We are pretty advanced from that angle. We follow scrum based agile development model – outsourced. Developers do all required testing and development and everything is fine. I am hearing you talking about QA again and again without me asking about it. We do agile model and as such there is no distinction between development and QA. As such we do not think cost is our concern.

LJ: I did not say “QA”, I said “testing” – I suppose you know the difference.
MJ: Fine … here we use these terms interchangeably. Well, does that matter in agile?

LJ: How can we help you?
MJ: You have been saying that you give QA , sorry testing services – Do you provide white box testing services?

LJ: Our Testers are trained well in both black box and white box techniques – that should not be a problem.
MJ: Let me make it very clear – in the name of QA or Testing, I don’t want some so called tester coming and doing “negative” testing and showing 100’s bogus UI level bugs. That is not what I want.
LJ: Our testers are trained in exploratory testing and can flush out hidden bugs
MJ: I said I do not want negative testing – UI level keyboard banging. I want white box testers.

LJ: What according to you is white box testing?
MJ: I am surprised that you are asking this silly question. It is the testing done with the knowledge of product internals. Something that developers do. Don’t you have such testers .. I mean developers?

LJ: Well, the reason I asked this question is – there are no universally accepted meanings of the term white box testing and I wanted to make sure I understand your version. OK Why can’t your current developers do that?
MJ: Our developers are already stressed out. We have very well defined agile processes. Our developers do very smart and JUnit based automated tests are our key to success.

LJ: OK … What can we do for you? It seems that you have everything that you need.
MJ: How about your folks doing an assessment and tell us what our maturity is? How well we are placed with respect to industry standards?

LJ: That appears like a different problem …. Anyway, are you sure You want an assessment?
MJ: Yes. I would like our processes to be best as per industry standards.

LJ: What are your expectations out of this assessment?
MJ: We need to add few white box testers to our existing team as we are weak in that area. We are sure we do not need QA resources that do negative or exploratory testers. We are also sure we do not need testers. We need developers who can do white box testing.

LJ: Oh!!!! That is what you want? I can give you developers who can do white box testing? Why you need assessment?
MJ: You see, we are looking for a consultant to make case for us to get budget for these resources. We really do not worry about testing, QA, White/black box testing etc. Can you get a consultant?

LJ: Great … I will have a top notch consultant at your office tomorrow.
MJ: Thank you. Make sure, you get someone who understands our problem very well. I don’t want to go through all these questions again.

LJ: So, let me recap. What is your problem?
MJ: Damn it …. I need a business case for increasing my team’s headcount and I need white box testers. Are you clear?

LJ: Yes … Sure …
MJ: Thank you very much.

What do you think … what is the problem here?

Update: I did a mistake of swapping the names of MJ and LJ. This is corrected. See how it reads now.

Shrini

Necessary Tester skills ....

2009-10-06T13:34:00.002+05:30

Curiosity? Keen and close observation? Skepticism? Creativity? Arguing/debating? Imaginative? Analytical and Holistic? Good at Probing/Investigation?

None – would say many IT testers, I meet on regular basis

Today’s IT testers have forgotten these terms I suppose. When looked at macroscopic and group level – todays Test managers, delivery heads appear to have lost sight on these core skills for testers. When I ask any typical IT testers what they are doing to enhance the skills, typical answers I get are - learning domain, usage of specific tool, or a programming language, or some process/methodologies stuff. That is not wrong but having any of these skills without the basic skills required for testers – we would be developing the armies of robots who are very good at following orders and over a period of time, stop thinking on their own.

NY Police (a profession similar to testers) are taking course in “observing and describing” – What our testers should be doing to sharpen their testing skills? How they are putting their cognitive skills in their day today work?

http://www.smithsonianmag.com/arts-culture/Teaching-Cops-to-See.html

Shrini

A process is what you actually do. A process document describes what someone would like you to do, ideally. They rarely coincide exactly, and sometimes don't overlap at all - Jerry Weinberg

Who decides what is a bug and what should be fixed?

2009-09-11T16:04:00.005+05:30

I was submitting a proposal for a paper for STC 2009 conference. After completing all fields – about 20 of them , I accidently hit clear button (by practice – “OK” or “submit” button appears first in such forms) and all that I entered is “gone” – worst there is no way to recover.

Is this a bug?

Another catch – for the date field – no format is specified, neither there is a calendar control. How do I find out the required format? Try one and get to know about the format.

Is this a bug?

If you were a tester who would strictly go by “test cases” generated out of “specifications” – you are likely to miss such bugs – remember there is something called “Requirement based testing”. Alternatively, you might argue that these are not bugs or bugs of low severity. Who has the final authority to say which is a bug and which one should be fixed and when?

Shrini

Your user acceptance testing is fixed – Is that a problem?

2009-08-19T19:22:00.003+05:30

“This troublesome user acceptance testing got to be easier. It is really a pain” said of my colleague quoting an IT manager responsible for release management and user acceptance testing. Major problem according the IT manager was getting the “users” to do their job – testing a defect fix, feature enhancement or a new release. Users are always busy with their “day job” and for most of them doing testing is a low priority activity. I have heard many similar stories regarding this important aspect of “end user” responsibility.

In traditional IT organizations, user acceptance testing is a formal stage in the testing life cycle where business users will test the proposed changes to production software applications. IT organization that delivers the software to meet the needs of business users, require a (formal) approval of proposed changes that come in terms of defect fixes, enhancements and new releases. Since it is business users that ask for changes for their existing and fund the development/testing work – they will have the final say on “acceptance” of proposed changes to production applications.

Primary purpose of user acceptance testing is check (cursory- final) proposed changes to the production software by those users that asked for the changes so that any last minutes changes and surprises are avoided. User acceptance test is the last gate before software goes Live so that it provides last opportunity for all involved (both IT and business) to make corrections if required. Depending upon the nature of business supported, nature of software application, nature of changes proposed UAT may last for few days to few weeks. Software that fails in UAT is typically assumed to be insufficiently tested and is variably returned to IT for fixes and more testing.

The problem of UAT
Business users who hold the responsibility of approving software updates to production systems, typically are not testers and as such do not come with testing skills. Depending upon the nature of software updates, IT team will ask business users to carry out testing to check the features of new system. Most of the user acceptance testing tends to be repetitive and that is what drives “real users” crazy. For development team (IT), their work is not complete until the piece of code is user tested and accepted. Hence they literally chase user group to perform UAT with users complaining about the time crunch and their “important” business work getting impacted. This creates a situation where both parties (IT and business) want UAT to be somehow completed so that each can carry out their business as usual. Hence UAT often gets “fixed” with the consent of both parties having stakes in the activity.

Forms of fixing UAT
Between Business and IT, UAT can be fixed in number of ways – some of these are reasonably business driven models given the constraints.
1. UAT will be performed by the members from IT team where business only reviews the results of the test. If the results are OK then UAT is ought to have been completed
2. UAT will be performed by a third party, such as someone from support staff. Business will review the results and accept the proposed changes if results are OK
3. Business will provide a prescribed set of test scripts that can either be used by anyone IT staff or support staff. Results will be verified by the Business.
4. UAT will be done like a demo to the users where IT staff will execute some pre-approved test scenarios related to proposed changes.
5. IT team will train the business in new features proposed to be introduced. Business users after training, test (use) the proposed software and accept the software

Why fixing user acceptance testing is bad thing?

Why bother to UAT after all? For IT, more often than not, it is a formality to be completed before they push the code to production.

In my opinion, it is the spirit and purpose of UAT that gets compromised. Typically, in spite of all best efforts, the depth and frequency of interactions between IT and business throughout the project remains low. When business users do not participate with full spirit in UAT, lots of things go unnoticed into production. This might results users (non participating ones especially) getting surprised when they see the product.

What has been your experience? Do you bother if you see your UAT is fixed?

Shrini

Tom DeMarco’s confession

2009-07-26T06:17:00.001+05:30

Confession is probably a harsh to describe what Tom DeMarco, the creator of celebrated punch line of software managers – "You can't control that you can't measure", wrote recently. But, if you read Tom's recent article "Software Engineering – An idea whose time has come and Gone?" in IEEE software, you would probably say something similar to this. In this short 2 Page article, you will find Tom DeMarco in reflective and retrospective mood.

In the book "Controlling Software Projects: Management, Measurement, and Estimation (Prentice Hall/Yourdon Press, 1982) ", Tom talked about "Controlling" and "measuring" in Software Engineering. Nearly 40 years later, he now appears to admit that he pushed the notion of "control" and "measurement" too much.

This "confession" has apparently caught the attention of many. Jeff Atwood writes an obituary to "software engineering". More than his post, the comments for the post of Jeff Atwood are interesting to read. How come, suddenly so many are accepting now that "software" is human centric, people oriented, "engineering" is not the right term to use and so on? Michael Bolton's recent article (three kinds of measurements and two ways of using them) on stickyminds appears have been triggered by Tom's "confession". Matt Heusser writes about metrics here, here and here.

Managing vs controlling

In his own admission, Tom seems to distinguish between controlling and managing. He gives the example of "upbringing of a teenager". As any family therapist would recommend, you manage your teenage kid rather than controlling them. Like in most human endeavors (including software programming and testing), you can manage quite lot things than controlling them. I think we can manage (and also a control a bit) WITHOUT measuring ANYTHING at all. I like Matt Heusser's example of hair cut. Many of us control and manage our hair (style) without measuring.

Manage people and Control Money and timelines

This is Tom's recipe for managing the project without controlling it. While breaking project into human elements and non human elements (such as code, schedule, money) is a welcome change, I am not sure how can you do it – managing people and control money and timelines. To me, these two sets of things are NOT mutually exclusive so that you can treat each in a different way. Controlling time and money impacts people and managing people impacts timelines and money.

Some software is really engineered!!!

Dave Markle makes an interesting point in Jeff Atwood's post - "IMO you can't say that programming languages themselves aren't engineered based on solid computer science. You can't say that something like LINQ hasn't been engineered. Whenever you use a FSM in your software, you are applying computer science, which makes you a software engineer"

Programming languages are engineered, operating systems are engineered and so are software algorithms. It probably the user mass/ size of the group decides engineering vs. crafted.

Dave uses the Paint analogy nicely to drive home the point - the paint artist uses is engineered (developed and mass produced using the principles of chemistry and physics) where as the "art" produced by the artist is NOT. This stirs the nest of debate of what is engineering and what is craftsmanship. Are Engineering and Craftsmanship are mutually exclusive? I am afraid NOT.

Tom, Damage has been done and is still happening. Many still many abuse your punch-line to push loads of documentation, process, approvals, and meetings and of course endless charts/graphs of metrics – all in the name of "control". Probably the time has come to step back and be sensible on "measurements" in software.

Shrini

To tweet or to blog ...?

2009-06-16T02:36:00.005+05:30

Due to me crazy traveling and work, I have not been able to write as frequently as I would have liked, for the blog. At least half a dozen potential blog posts are waiting to see the light of the day from my side. It is just not working out.

I have been on twitter (for starters - you can take this as a quick blogging or microblogging) quite active in recent days. Happy to see many following me now. It is suiting me for now as I need not feel guilty of not being able to discharge my duties as a blogzen (no!!! this word has not yet been used by someone previously)of software testing blogosphere.

So till, I get to full time blogging - please follow my thoughts on twitter. I have added twitter feed on this blog to facilitate for my readers to catch up with what I am working on ....

Thanks for being my blog reader ....

Shrini

10 ways to make automation difficult or ineffective

2009-05-14T12:19:00.003+05:30

Here goes another 10 items list for automation. If you are in IT or IT services space and manage/deliver automation solutions -- make sure you stay away from these as these items have high likely hood of making automation inefficient/ineffective and difficult.

This list is an extension of a topic and this list (of 10 items again) for test automation outsourcing

10. Wild Desire to automate 100%

9.Attempting to automate existing test cases without scrutinizing them for “suitability” to automate

8. Mapping test case to script 1:1 linear model – falling prey to deceptive traceability and gold plated reporting.

7.Not building automation solution bottom-up , unidentifiable building block of the solution.

6. Trying only one type of automation or attacking only one layer of the application – Farther you go from code, messier it gets.

5. Focusing only test execution related tasks

4. Treating automation as scripting – ignoring “generally accepted good software development practices for hygiene.

3. Failure to involve developers from the beginning – Not attempting to testability or automatability of the application.

2. Jumping to automation to speed up testing or save cost before fixing testing problems – inadequate, inefficient and broken.

1. Failure to arrive (formulate) at the right mix of human testing and automated test execution.

0. Using Automation as solution to testing problems.

I reiterate that these are applicable mostly to COTS driven, GUI functional Testing automation that is typical in IT/IT services environments. WI might have to rewrite some of these for xUnit type formalized unit testing (that is also automation and some call it even as "testing").

Shrini

Is this a bug?

2009-05-13T15:37:00.003+05:30

I was flying from Cape Town to Bangalore through emirates flight. It is convenient to online check-in. I do it as I can choose an aisle seat. But for the second time, I got into problem while doing online checking for emirates. Probably the Internet connection was slow – in both occasions, emirates online application did not respond and I had to close the browser after 5-10 minutes of frustrating wait – staring at screen.

Here is the bug that frustrated me…

· I try to do an online check in and would like to change the seats.
· Application hangs when trying to save the changes.
· Close the browser.
· Try again to do check-in
· Get a message that the passengers have been checked in.

Fine – how will I know what are my seat numbers? How do I view my check-in details.

Apparently there are no easily reachable ways to gather information. Probably there is none. How do I search where is “view check-in details” or “view eBoarding Pass” or something similar? I tried site map, tried “Help” and tried “search”… could not figure out the link for viewing check-in details.

Is this a bug? If you are a tester will you catch this bug? If you are a developer will you accept that this is a bug? I am sure most people will say “if this is an intended functionality (I think, it is), then it should be documented requirement specifications. Once it is there, tester can write the test case and developer will make sure that the functionality is coded and tested”. Some testers might say this is “nice to have feature” …

What might have happened here? Requirements problem? Development problem? Or a Testing problem?

10 ways to reduce cost of software testing

2009-04-19T01:53:00.004+05:30

In current economic situations, IT folks worry about one thing – “reduce cost”. I have been frequently asked “how to reduce testing cost”. A no brainer answer would “do not do testing … at all”. How many buy this idea … can current breed of IT applications sustain with less or no testing at all? When I use the term testing –I am referring to “non-programmer” testing.

Here is my draft list of suggestions ...

1. Closely work with developers, do some parallel testing with them as the product/feature is getting developed
2. Identify and eliminate non-testing activities that occur in the name of process, documentation, management, metrics etc.
3. Analyze and profile every application under the portfolio to determine “stable” and “well tested” areas of the application. These areas should receive the least or no testing effort.
4. Analyze the test scripts suite and remove redundant, worn out ones. Aim to reduce scripted test repository as small as you can.
5. Review and reduce “regression testing” on the basis of “well tested/stable areas” of the application
6. Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
7. Plan testing in small but frequent cycles (Session based exploratory testing approach) – reduce planning and management overheads
8. Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
9. Cut down on lengthy test plans, testing reports, dashboards – switch to simple but frequent test reporting.
10. Simplify defect management process – reduce defect life cycle – resort to informal/quick defect communication.

Some this advice might look like a simple common sense (eliminate waste, focus on tasks that impact end result DIRECTLY). With so much selling happening about “testing tools”, “factory models”, “cheap and best testing services” – any common sense is difficult to come by.

How would IT community react to these suggestions – most likely response would “This would not work, how can we reduce testing, those test cases, processes, metrics, management practice?”. These suggestions would be most likely to be rejected on the grounds that testing cost needs to be reduced without “compromising quality”. Many IT folks think that quality comes from test scripts, processes, metrics, testing tools, automation etc. I am afraid quality is not such a simple thing.
Again, there are no free lunches here … if you are thinking about reducing cost of testing, there are always risks of impacting quality (roughly goodness or confidence in the product) in one or other way. If you approach the problem (cost vs quality) from a quality side (improve testing - test better, deeper and wider), then the chances of achieving good quality and also “some” cost benefits are more likely. However, if you approach it from “cost” side of the equation – you might do achieve that albeit some impact on overall goodness/quality of delivered product.

Note that some suggestions mentioned in this list call for some smart testers who can think on their feet, work with least supervision, least (optimum) documentation and processes and so on. I think, the focus should shift from process, tools, management, documentation to Skill. There can be problems in getting such resources in IT scenario (especially in outsourced/offshored world)

You have choice … which side you would like to approach the problem ..?

[update 20/Apr/2009] A colleague of mine reacted to this list saying "These are too risky suggestions and he would not recommend any of these. Business prudence is totally missing".

I think he was expecting to see some "low risk and high return" type of suggestions - like those "cheap" and "best" items. I still do not understand - there can be no risk free ways of reducing (testing) cost -unless you are totally spending like crazy without any thinking. We do not seem to have such risk free - free lunches - why fool ourselves and the client in believing such "non existent" things?

Another suggestion that came up was "Let us use standardized processes". How standardization can reduce cost? what is the cost bringing in standardization itself?
May be the expectation is that standardization will make each tester behave identical to another like robots. Are robots cheaper? may be? may be not .... They at least do not whine about working on week ends :)

Shrini

When "Process" stops working for you ....

2009-03-21T23:30:00.009+05:30

Other day I overheard a Test manager speaking to his team “As a CMMi Level 5 company, I don’t think we are following processes. We often talk English rather than showing supporting metrics that are back bones of any CMMi Level 5 organization. If you don’t measure then how can you improve? Considering the economic slowdown, it is high time that we should start showcasing our continuous productivity improvements or else we will lose the client"

What is happening here?

Most managers somehow (more so in current economic situations) confuse skill, human ingenuity and expertise to metrics/measurement. When customer cribs about “value” and quality of work delivered – she really is cribbing about people and their skill (not about metrics and measurements). When people hear about customer cribs … managers suddenly jump and say “let us collate some metrics and show client that we have delivered the value (which they will dump eventually)” and push the core issues about skill below carpet. This “hide and seek” game goes on until we lose the client. This pattern has to break and unfortunately I have no simple solution for that (probably no one has). Few of us appear to know the root of the problem now.

If following processes would ensure quality and being very serious about metrics is HOLY – then our problems would have been solved long ago …. Why people do not follow process? Is it because they are so tough and stress full to follow? Is it because they are difficult to understand? Probably people follow process and we have stopped being critical of whether process is doing any useful thing are not …. That is the start of the problem. Glorifying process beyond its own utility (ask process – it would probably say … beyond this, I cannot add any value). I understand process (whatever is the definition) provides some common framework within which people with diverse educational/technical/social background work to produce consistent output so that whole thing can be managed easily. Beyond certain point (this limit might vary from context), process cannot help any further. It calls for people's skill to deliver - process then becomes an enabler or mere Hygiene factor. Just walking or eating alone can not keep you healthy all the time. Do you know where is the limit beyond which "following process" can no longer help?

There is a big fuss about “using English than numbers” … Why there is so much faith on numbers? Why qualitative subjective wordings are such a waste? Why not we express everything in numbers all the time – our hunger, happiness, intelligence (yes there is IQ test), pain, sorrow, emotion (yes there is emotional quotient), commitment, enthusiasm, creativity and what not all human attributes are so rich and multi dimensional that poor numbers can express a minute part of them. And we refuse to use qualitative measures saying that “objective is better than subjective”. Many would like humans to behave as if they are machines so that they can be objectively measured. A sad reality…. Perils of advanced economic world. Hunger for objective interpretation of human attributes is probably has reached its crescendo. I am waiting for the downfall of that raise. Will it come?

There is a big deal about “improving productivity in testing .. We must meet SLA’s and show continuous improvement in productivity”. I am STRONG opponent of usage of the word “Productivity” in testing in general terms. When people say productivity, they typically refer to speed – number of units produced per unit of time. Much like in a shop floor assembly line. There might some portions of testing that one does that are “speed sensitive” but by and large skilled testing is not about “speed” more than it is about “coverage”, “identifying tough to find problems”, “asking right questions”, “seeking information”, “building on available information”, “investigation” and many more. Probably not more than 5% of good testing is speed sensitive… most of it is not … then what is the meaning of “productivity” when it is applied as “serious generalization” to all testing. I PROTEST ….

Finally, come’ on, let us accept there many ways we can improve (many) things without measuring them (at all) at least in poor numbers. We all do it in our day today interactions with our near and dear ones in family and those out side in society. So there are clear exceptions to the statement “you can not improve if you can not measure”. I strongly oppose the statement. Too poor generalization that suites machines and mechanical constructs well, than human beings in a social structure.

C-DLICE’ing in software Testing

2009-03-06T19:46:00.001+05:30

Let me take credit for making this pneumonic up: C-DLICE. I was listening to Michael Bolton's video interview on youtube. He said testing is more than verification, validation and confirmation. It is about Challenging claims, Discovery, Investigation, Learning and Exploring. Any skilled tester would do one or more of these activities as part of testing. By explicitly chaining them in a pneumonic, a tester can focus on a specific aspect of the interaction with the test subject.

Let me expand the pneumonic –

C – Confirmation. Other than traditional words like Verification and Validation (whatever may be the meanings of these terms) most people on this planet think that sole aim of doing testing is "confirmation". It is seen confirmation of claims made about the product, conformation of what developers "felt" that they created in response to requirement specifications that received and interpreted to their best of abilities. The confirmation about some specific user expectations (assumed to be routed through specifications into the software product). In basic form confirmation is somewhat like "Click this button, such and such thing should happen – Does it happen?" While confirmation is important aspect of testing, any testing that focuses on confirmation will become boring, brain dead and poor way to think about testing. Notions like anyone can do testing, process plays important role in testing, testing without test cases and requirements is not possible – are creation of confirmation oriented testing. I will not dwell upon challenges in confirmation oriented testing. And there is a big deal called "reference" against which you confirm – specifications. If your reference is wrong, ambiguous and incomplete – so will be your confirmation. That is the weakness of confirmatory testing.

Though my pneumonic is more about DLICE, I will still keep this "C" in there to remind us that confirmation may be as important as other letters in the pneumonic.

D- Discovery. While we test tester, we discover information about software, certain behaviors. It is like discovering an unknown island. As product grows bigger in size (in terms of codebase), discovery becomes important. No user would use the software as per the user manual. Discovering way in which software could be used and misused is important aspect in testing. Discovering is about finding information about unknown areas. For a growing software application, every time there is more to test than before – more to cover than before. Under such circumstances, you constantly discover the application, its variations, behaviors and so on.

L- Learning- This is a freaky one. A significant part of testing is implicitly spent on learning about everything that software under test. Be it business domain, technology domain, community of users using the software, cultural and social set up in the organization producing the software, we learn all time. Learn about how the software is constructed, deployed, distributed and so on. Often, I have seen people downplaying "learning" aspect of testing as they would like to position themselves as "experts".

I – Investigation As testers we investigate claims about the product. How people perceive the product? Investigate inconsistencies. Investigate bugs, Investigate impact of a new technology, software change on the over image of the software. Investigation is about focused information gathering, analysis on certain events. Examining the evidence etc. Investigation starts off as open ended.

C- Challenge (used as verb) – As testers we need to constantly challenge the assumptions, beliefs around how people think about software. What each stakeholder thinks about the capabilities of the software? Challenge the premises and so on. Challenging would require the design of tests, experiments etc to expose the weakness of an aspect of software.

E – Explore – Somewhat similar to Discovery, Exploration enables any information gathering exercise. Explore market conditions. Exploring is about taking a tour. Exploration helps in modeling the problem space. Exploration is more open ended than investigation.

Notice that each letter is has some overlap with others. You can learn while discovering or challenging a claim or exploring a feature. You can investigate something by exploring it or discovering it. You can challenge something by investigating it or learning about it and so on. One way to think about DLICE is – Discover like Magellan, Columbus, Learn like a learning a new language, Investigate like Sherlock Homes, Challenge like Lawyers, Explore like exploring moon's surface or deep African jungles.

Few practical themes to apply dlice'ing

When there is a new thing most people around you know little about - something that you do not understand well , then – Discover, Explore and Learn
When there is something that several others know but you do not – Learn through exploration, discovering
When there is"suspense" or "mystery" about a thing – Investigate – a defect, a strange behavior etc.
When there is some that is "well known" to you (you are pretty sure) about some claim – Challenge it and prove your point ( backed up by prior discovery, exploration, investigation and learning)

So, next time you feel bored doing testing, try switching your focus … try doing some investigation, discover new ways of using software or explore an area of software and so on. You would find that testing is always interesting but you were told about only one dimension of it (confirm, find bugs, check it passes tests) so you felt low or bored about doing testing that way ….

HAPPY C-DLICE'ing

Shrini

Context Driven Testing gets a boost – to grow stronger…

2009-01-05T01:37:00.001+05:30

It is a fantastic "new year" gift (and also week end feast to many like me who is spending better part of Christmas/new year in front of laptop) to all testers… Dr. Cem Kaner (along with James Bach) has posted about context driven testing with some new definitions and articulation. I belong to context driven testing community with many others. Context driven testing community has proclaimed its philosophy, guiding principles since its inception. As years went by, various groups of people (quite a few "unidentified" ones) started pushing false propaganda about our community. This I believe might have driven Cem and James to "rework" the original principles and overall articulation about context driven community.

I have personally been part of many discussions, where I heard people just attacking the theme of context driven testing by saying "Everything in this software world happens with the context. Only a fool would work without context … so talking big and calling oneself as context driven tester is no big deal. Context is forever one … every one applies the context to best of their ability and knowledge". The new articulation of "Context aware" - attempts to describe such people. If you think about a practice first, then tailor/modify that practice to the context – you would be a "context aware".

Another interesting point this post makes is that some people in agile software development community have found so much common with context driven testing that they were claiming agile and context driven testing as one and the same. With so much focus on people and their interactions – agile and context driven testing can be said to have common roots – "people focus" (as against process/standards focus). I would say Cem's articulation could have been much stronger when it comes to the differences between agile and context driven testing. Insistence of 100% automated unit tests, compulsory standup meetings, TDD is a must and all sorts of "standard" stuff is clearly deviation from agile manifesto (choice of people over Processes) as articulated by James Bach in his STAR West 2008. I see context driven testing community distancing from agile community on these areas …

Also look the new, cracked definition of context driven testing – "Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing." I am sure most of my friends who are not in the CDT (context driven community) will say … oh yes … that is a common sense. We work for stakeholders, customize our testing approaches to context … what is big deal? Why are you making common sense a big thing?" My response to such people would first "so called common sense in this case is not so common … so there is a big deal here. Secondly, thinking about practices first then context is "context aware" not "context driven".

Critical Focus on "best practice" has been rather mild I would say (appears in only 4 times in the post). Cem, apparently left the proponents of best practices, lightly by saying "Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context…. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven."

Context imperial testing is somewhat similar to what I normally refer as "goal displacement" (a term I learnt from James Bach). Instead of designing and adapting the practices to project /organization/group context, Context imperialists would RATHER change the project/group/organization itself to suit the "best practices that they are aware of. A recent example … someone said, we cannot afford to do testing this way as it will not allow us to collect the metrics that we need, so let us change the process so that we can collect and use the metrics". It is sad state of affairs that, people will get away with such "context imperialism".

Context driven testing community's view point on "detailed specifications, detailed test script documentation etc" has always been perceived as "means to promote exploratory testing". Many people, whom interacted, would immediately say "Oh!!! You mean to say do exploratory/adhoc testing then" in response to my statement that "In context driven community, we do not insist on detailed specification … In fact some of us think that is a crime to refuse to test without specifications". In reality, the crux of the matter is, in CDT, testers need to cope up with whatever information that they get and start from there to gather information that they need. In those situations where time, information, people with knowledge - are rare commodity, context driven testing is clear winner. It prepares its tester for such eventual realities.

Finally the assertion that "There are no context-driven techniques" – should put all those statements/viewpoints to rest such as "Exploratory testing is a context driven testing technique". Neither ET, nor context driven testing is a testing technique in itself…

Overall, this post is a great milestone in the history of Context driven testing … should be mentioned in context driven website.

Shrini

MS outlook as Alarm Clock: Is this a bug?

2009-01-04T23:58:00.002+05:30

[Background in the beginning … those who are interested in reading about bug I am talking about, skip first few paragraphs and go straight to Bug]

There is a saying that "Software users, most of the time, do not use the software as perceived by the designers or analyst". I happen to deploy Microsoft outlook as an alarm clock for me to help with a wake-up call. I am away from home, not carrying a cell phone, ipod is not a good tool for this purpose, buying an alarm clock for short duration would be a waste … hence zeroed in on Outlook meeting reminder feature as a software solution to help me to give wake up call.

I set up a recurring meeting of 0 minutes at say 7:00 AM everyday in the morning with reminder zero minutes. I thought this would work. I needed another hack … a sound alarm long enough for me to wake up. Most of small sound files shipped with windows/MS office were not helping me with this purpose. I had Yahoo messenger on my machine with an audio file that plays for few seconds. I decided to use that as reminder sound file. Even there was a problem. I thought 2-3 secs of Yahoo sound file might not be long enough to wake me up. So I needed to play this one file repeatedly say 7-8 times and record the whole thing so that I can get a sound file that plays for 8-10 seconds.

I went live with this setup, thinking that my next day's worries of waking up at specified time were over. It seems that UAT was not proper … first morning the alarm did not wake me up at all… when I investigated, I discovered that previous night before sleeping, I had muted the audio of the laptop … so far so good. I was waiting for next day to see if it works … Bingo … it did … exactly the way I wanted. No big deal I thought. However, I was happy to discover a low cost (zero indeed) technology to a problem in hand … software alarm clock.

I am sure there might be better ways … one could have written a small program in VB or Perl or Python to do this … with even snooze feature …. But, I think my solution worked for me … only to fail next day …. What ..? Yes, next day, the important day where I had some important meetings to attend… Outlook alarm failed … Is that a bug … Looks like I accidently stumbled on this bug …

[BUG] I think few hours before the scheduled appointment – alarm to kick off, a pop-up message about (Junk Email alert) came up prompting the user to take action. This is a modal window. Hence before dismissing this dialog box, outlook cannot proceed with any other pop-up windows like meeting reminders. So, my wake-up alarm did not get activated as there was a modal window waiting to be acted upon.

Is this a bug? May be or may not be …. Some might argue that this is not a bug as outlook is not supposed to be used this way. A developer may say that Junk email feature was required to be implemented as modal dialog box, it was assumed that people will act upon it … as they do for any other modal dialog box. Others may say it is too small problem to worry about …Some may even point out a work around for me … check on "Please do not show this dialog again"… so that my alarms will work without any problem. The Fix might be simple … (superficially) just make it as non modal window…. Or there can be other implications. I do not know... What if I had missed the flight back to India due to this software problem? What if I was delayed to all important meeting due to this software problem leading to huge financial loss?

What do you think?

Shrini

Metrics or a liquid in a container?

2008-12-31T03:48:00.003+05:30

I was responding to a thread on "productivity in software - when the term is relevant" on test republic and posted following paragraph about quantification and metrics, which I thought would benefit larger audience ....

People (especially managers) when trying to make things that are not (easily, at least without changing the form of task or thing)quantifiable, quantifiable - end up in changing the core of the thing.... that is goal displacement. I once heard a manager asking us to change the way we do testing because the method we were using was not quantifiable (having more human elements etc). We then changed the approach to testing to meet the quantification needs of testing rather than original needs of "information" and "evaluation". Following this, the manager was happy, metrics were available, everything was quantifiable .. but testing suffered. When that happened, manager managed to shift the blame to something else (totally unrelated) and gotten away. He could do that and get away with it because metrics enabled him to do so ... metrics being numbers have no story or no descriptive form on their own. Given a metric, you can tell any story that you want to tell and manipulate the world.

Software Metrics are like gases or liquids, they do not have their shape or form, they take the shape of the container in which they are placed. Be careful about this when making or dealing with things in software that are "quantifiable" ...

Shrini

Quantum Theory and Software Testing - Any connection?

2008-12-31T03:12:00.002+05:30

Atomic physics, Atomic structure and history of how all those brilliant scientists discovered (still discovering) atomic structure, electrons (dual nature of wave and particle) etc always fascinated me. More so now, than in my college days. Often I dream of going back to college days and debate with my teachers about stuff like atomic structure. It is tester in me, who curiously looks for finding some answers to questions that keep popping up in my mind.

I happen to pick up a book on quantum theory titled “In search of Schrodinger’s Cat” by John Gribbin. What attracted me is a quote attributed to famous scientist Niels Bohr, that appeared on the back cover of the book.

“Anyone who is not shocked by quantum theory has not understood it.”

I immediately applied it to current state of practice and perception about “software testing” and I would say

“Anyone who is not shocked by popular practice and theory of software testing has not understood it (especially the human element of it”.

I will be shocked when people make claims (what I call as "popular perceptions") about software testing such as “all testing should be documented, test cases are important for performing testing, test cases must be traceable to requirements, you cannot test without specifications, tester’s role is to find bugs, testing assures quality of the product, testing needs to be more process oriented than person dependent etc. I can say that those who are not shocked by such claims have not understood about software testing that relies on human elements (thinking, questioning, observations etc).

Quantum theory is fascinating so is software testing …. I am looking for more connections between these two … Can you help me? Or I am just dreaming?

Shrini

Can Software ever get IT right?

2008-11-19T23:39:00.001+05:30

Matt Heusser wrote this beautiful piece of writing about software development practices – quoting another famous Blogger Joel Spolsky.

"... which has a programming method in which programmers code stories based on notes written by designers that are based on requirements documents created by analysts that are assessments of what the customer actually wants. It's practically designed to get everything wrong, to insure that, no matter how ignorant the analysts and architects are on an issue, they'll find someone who knows even less to write the actual code ..."

It is interesting that with so many "loose" ends and human elements (thinking, question, modeling and analyzing), many still fancy the changes of "zero defect software", compare software to manufacturing, glorify processes to fix the problems in software that is INHERRINTLY designed to GO WRONG. If you look at the chain of analyst ->Designer->developer ->tester -> Customer, each one works with less or totally different set of information than all others.

Notice what Matt has to say – to make sure that they get the final software wrong – they will find someone who knows probably the least to write actual code !!!!

How can this (software) EVER GO RIGHT? Can this?

Read entire blog post here …

Shrini

Perils of Quantification – what harm metrics can do for you?

2008-11-19T13:30:00.003+05:30

This is a hurriedly written post (just to make sure that I do not lose the thought – "fieldstone" in Jerry Weinberg's terminology) – I plan to use this as a place holder for expanding ideas on this topic … Please bear with me for a while with this "being cooked" idea.

I stumbled on something that Michael Bolton said about metrics – in response a Google group discussion thread. Michael mentions "What you want to beware of, in particular, is turning rich information (stories about bugs, problems, risks, value) into impoverished data (numbers).

I think that is a great way (rather an interesting way) to think about "software metrics". To me, software metrics are great way to "squeeze", heavenly simplify" and "horribly trivialize" rich information about bugs, test ideas, problems, risks, value about software. While they provide a simplistic view of rich and often qualitative/subjective data/information – there is huge danger of "oversimplifying and information loss".

Many people argue with me saying "quantification" – associating something that we try to understand in term of numbers – is essential for science and engineering. Some even quote "you cannot improve anything that you cannot measure". I feel that the "urge" for measuring, notion of be quantitative /objective is simply "over emphasized". Let us consider the perils (ill effects) of quantification. Some entities lend themselves for quantification – say counting. Counting people, counting vehicles on road, counting fruits on a tree, marks a student scores in an exam. Many entities that are related to humans especially are difficult to quantify – tend lose lots of information when quantification is attempted. This is very true with software.

Consider following quantified information – what do you think? What do you lose when you quantify …

One tsunami
1 billion Indians
1.3 billion people in the world below the poverty line of 1$ /day
8 million people affected with AIDS disease
Software Quality of Six sigma
In 2003 there were 6328000 car accidents in the US.
Finally
6300 bugs in Windows 2000 ..

Notice that each of these numbers have rich information about loss of life, health of people, quality of life and so on. By squeezing rich information into a number, we lose the information. Numbers can be manipulated, argued in any way you want, they hide information, you can be cheated by numbers. Numbers are single dimensional where as information they tend to represent are often multidimensional.

"As proven by modern accounting scandals, you can make the numbers say whatever you want" – Mike Kelly

To be continued …

Shrini

A conversation on Automation ROI Part 1 …

2008-11-17T21:32:00.002+05:30

When automation is required, either by contract or due to technical constraints, ROI computation may not be helpful. Intangible factors may constitute the bulk of the return, and thus arithmetic computations won't indicate the real value of automation. Fortunately in these situations we often aren't faced with questions about the value of automation because it must be employed regardless.

-Doug Hoffman

Here goes a conversation with a colleague of mine who wanted me to help him with some ROI calculation for an automation project.

Colleague: Do you have a formula or framework for calculating ROI from automation?

Me: I might … first let me understand what you are looking for.

Colleague: It is simple man … Here is a client who is looking for investing in automation and she is interested in knowing the ROI so that she can take it to her boss with business case.

Me: That is good. What are the elements of ROI you are interested in knowing now?

Colleague: What do you mean?

Me: To me, ROI has three elements – a notion of investment (effort, money, time etc – all these can be interdependent in some way), a notion of "return" (called as benefits – some tangible, meaning quantified in terms of numbers, some intangible – soft benefits – qualitative measures) and finally a timeline usually in terms of direct measures like calendar months, or work/effort months OR indirect measures like number releases, number of test cycles, number of platforms covered etc. Which one is of interest to you…?

Colleague: All three … of course!!!

Me: Then you have some hard work to do gather information, data, expectations, some historical and some current.

Colleague: Well... I thought it is easy to find out ROI … I was told that there are many freely available ROI calculators especially catering to automation … are you aware of them?

Me: Yes, I have seen few of them … not so impressed … One problem that I have with most (or all) of these calculators is that a) they use a highly simplified model of testing that is totally out of context. Meaning you can just apply that to any project, any technology, any tool … you will have some numbers coming out … That is too good to believe 2) They equate automation to human testing literally 1:1 … In my opinion, automation is a different kind of testing – remove the human being (to the extent possible) and introduce the machine (automation script) – then think (dream, pray and wish) that program does EXACTLY like a human.

Colleague: This is too confusing …. Let me try to explain my problem in a different way. Customer is investing x dollars for automation, she wants to know when will she be able to recover the investment and when she will start reaping benefits (possibly without investing anything incrementally). How can we help her?

Me : OK … that is fair … Here we come again to same structure – x dollars (investment), when will she recover the investment (time lines) and when/what benefits she can expect, without possibly not incrementally investing (returns). Let us attack one by one … How your client wants to recover the investment?

Colleague: that is silly question … she wants to save manual testing effort by automating all or whatever is technically feasible. So, cycle time reduction is what she is looking at.

Me: So, the questions are – How much will be the cycle time reduction (assuming that that is possible and worth pursuing), by when, that reduction will be realized? What are the incremental benefits till that point of time? Right? Anything else I am missing?

Colleague: Good … I think now you have understood my problem … what next?

Me: Are all cycles of the same size? What happens to application under test for all these cycles (meaning does it under go change or not?) What is the current test cycle time (manual)? What all happens in a cycle? What are things under tester's control and what are not? When do you repeat cycle (assuming that you repeat cycles)?

Colleague: Oh!! My God … my head is spinning … I will have to get all the information and data... Are you sure these are required for ROI calculation? Anything else?

Me: Yes, at least in my opinion, to give a reasonable picture of ROI where R = cycle time reduction – I would need these. There are some more things that I would require to complete the equation … but let us get started with this ….

BTW, what makes your client believe that machine replicate what humans do? There are things machines are good at and there are things humans are good at. No matter what you do … I think in the context of test automation – machine cannot do what a sapient human tester can do (unless, human by design behave as though they are brain dead and emotionless).

Colleague: No …. No … not again … Do not try to confuse me … I will get you the details that you asked for… then let us fit an ROI formula. Please put you're the tester in you to sleep till then …

Me: (smiled) OK … Please bear in mind that "you cannot compare even one cycle of automated execution to same cycle of manual execution".

While my colleague is out to get the data that I asked for … what do you think? What have been your experiences of calculating ROI with automation … how did you deal with "improbable" yet simplistic model of treating automation execution equivalent to what human tester does – hence talking about cycle time reduction etc? What other returns (benefits) that automation provides, have been successful with your clients? How did you quantify them?

I work in IT services industry. Day in and out, I hear people asking me such things - while I attempt to explain them the hazards of the simplistic model of testing and automation they use in ROI, need for business case to push automation (that requires numbers and quantified measures) makes me to look for innovative ways to articulate what I want to say but in way that "business" people can agree on …

To be continued ….

Shrini

Extraas:

Here are 3 useful and well written papers on "Automation ROI"

ROI of Test Automation by Mike Kelly (2004)
Cost Benefit Analysis by Doug Hoffman (1999)
Bang for the Buck Test Automation by Elisabeth Hendrickson (2001)

Closing thought:

"… every time I hear "let's take a look at the ROI," or "it will increase your ROI," or "all we need to do is use the ROI calculator" some little part of me shrivels up and dies. It drives me insane. I refuse to believe that for products as complex and involved as automation and performance testing services (where you need to understand infrastructure, application architecture, business and use cases, deployment models, culture, risk tolerance, and the other aspects of the design, development, and testing taking place for a project) that you can so easily capture the ROI. If it were that easy you wouldn't be talking to me about it."

- Mike Kelly.

2 Notorious “E”’s of Testing

2008-11-13T14:33:00.002+05:30

Efficiency is doing things right; effectiveness is doing the right things – Peter Drucker

Let me make a dig on these two notorious and the most abused terms - testing effectiveness and efficiency. Raj Kamal has a post that discusses this aspect here.

To me, effectiveness has a notion of "degree of serving the purpose". For example, we can say "this measure" taken to curb the inflation has been effective (means it appears to have served its purpose). This medicine is effective in slowing down the disease. So when talking about effectiveness with respect to testing - we should map the results to the mission of testing and say the techniques, approaches that you have deployed served their purpose or not. Remember, as testers we serve our stakeholders. Different stakeholders have different expectations from testing. Testers form their mission to suit those expectations.

So, in order to be effective in testing, we need to understand the possible stakeholders, their expectations and which one to focus on. That would lead to testing mission. Any testing that happens, should serve the mission. Along the way, testers employ different approaches, techniques, tools and methods. Few of these would be "effective" in serving the mission and hence serve the stakeholder the information that they are interested in knowing, few many not. Therefore, if you are thinking about articulating about effectiveness in testing, think about stakeholders first, then their expectations, then the testing missions, then approaches, tools and techniques, finally link all of them to the results that you produce. I am not sure if a simplistic metric of an equation that counts "reified" entities like bugs and doing some math (like taking cube root of sum of all bugs and so on). Bugs are not real things but they are the emotions and opinions of frustrated "someone" who matter in your project context. Can you quantify frustration?

Also remember, since there could be multiple stakeholders (hence multiple testing missions), your testing (approach, tools and techniques) cannot be effective for all missions. Accept this fact and you don't have to be guilty about it. This becomes very visible when there are contradicting expectations and hence contradicting missions. Learn to negotiate with the stakeholders, try to iron out conflicts and state in your test strategy which missions you are focusing on and why.

Now, let me come to the term "efficiency". You might have heard people saying – "this vehicle is fuel efficient", "this equipment is energy efficient", "this worker is efficient". To me, the term efficiency is related to the notion of degree of conversion of deployed input (human and machine capital) to desired outcomes. Let us take the example of an internal combustion engine and put the definition of efficiency into perspective - The ratio of useful work to energy expended. Like effectiveness, identifying the best way to convert useful energy for testing to useful results that our stakeholders value – is never a simple task. There is no one right way to things also. To serve multiple stakeholders and testing missions, we as testers need to employ a diverse set of techniques, tools and methods. Hence there can be multiple ways to define "efficiency" with respect to software testing.

While Peter Drucker provides a simple framework for thinking about these terms, I would say it is too simplistic and rudimentary model to apply it to software testing. We neither have one "right" way of things nor a specific set of "right things to do". There are many right ways to do things and there are many right things to do. Who defines the notion of "right"? Our stakeholders. Therefore, it is very important to align our work as testers, to what stakeholder expect. First step towards this is to identify our stakeholders. Have you done that for your project?

I would like to highlight another thing here. Since the notions of efficiency and effectiveness as applied to software testing, are multi-dimensional and cannot be reduced simple set of numbers. Avoid temptation to simplify these parameters into simple metrics defined in terms of entities like bug counts and test cases counts etc. Think broadly and deeply, consider multiple stakeholders and testing missions.

In short, effectiveness deals with "fitness of approach/tool/techniques to serving mission" and efficiency deals with "conversation rate of deployed capital (humans and machines) to intended output". In other words, effectiveness is about "how powerful is your way of doing things" and efficiency is about "how well you do things". Both of these parameters are important indicators of testing work and are multidimensional in nature.

Shrini

Exploratory Testing - the state of the art, Evening Talk

2008-10-31T09:11:00.000+05:30

I am delivering a talk on "Exploratory Testing - The State of the art" at STeP-IN forum. This talk is happening at Intuit Campus at Bangalore on Nov 6th.

Find the announcement for this evening talk here.

I plan to cover mainly the advancements, tools, trends in last few years in the field of ET and shed light on controversies and myths associated with exploratory testing. There would discussions on SBTM, ET cheat sheets (Elizabeth Hendrikson), thoughts of Jonathan Kohl (analogy to music), Cem Kaner's thoughts on "ET after 23 years", Works of James Lindsay among others.

Here is a most popular myth... (can you beat this?)

"Exploratory testing is a technique"

Any suggestions, ideas are welcome...

See you there !!!

In the mean time, here are few posts that you can read about exploratory testing ...

Exploratory Testing Shock
18 Myths associated with ET
ET challenged

Shrini

Questioning Software - Bizzare?

2008-10-18T14:17:00.004+05:30

Rex Black does not like James Bach’s definition of testing “Questioning a product in order to evaluate it”. I am not sure why. During test2008 conference, this issue was brought up during a discussion. Rex said (paraphrase) “Questioning a lifeless thing like software is bizarre. I cannot question my dog”. I attempted to catch-up with him later next day to see if I can know more about his views. When I managed to get his few minutes, he took me to a pillar (painted with red) and pointing his finger to the pillar, said “Are you red”? He continued “I am asking the pillar. Am I getting an answer? “Questioning software is ridiculous and bizarre”. I thought I would get a chance to react to what he said. Rex being a busy man and did not have time for “bizarre/meaningless” debates, excused himself and went away.

Is questioning software really bizarre? I don’t think so. When Rex walked up to that red pillar and asked “Are you red” – what was he doing? – questioning the pillar. Right? He did question the pillar. Through his eyes, he could figure out that it was a red pillar. What is the problem then? May be, he was referring to the inability of the “lifeless” pillar not to answer him back in some human language that he could understand. Well, that was answering part- not the questioning part. Let us apply this to software, everything that we do as a part of testing can be thought of as a question that we ask (not necessarily in the same way as humans communicate) and Software does answer (unlike the pillar) in a subtle way. Thinking about testing a questioning process is a strong and powerful way to organize the thought process about testing. As testers we must develop skills to question, skills to interpret the answers, skills to improvise questioning and skills to analyze the subtle answers given by the software.

What do you say Mr. Black?

[Update] More on questioning, meanings and various interpretations of "questioning" software by Michael Bolton is here.

Shrini

Soft Part of Software Requirements ...

2008-09-14T16:35:00.005+05:30

Michael Bolton in response to a discussion on software requirements mentioned this ...

"... There are many requirements that are matters of opinion, aesthetics, value, usability, compatibility that can't be subjected to a formula, can't be anticipated in advance, and which change over time "

Really true ... what we typically ignore about software requirements is that --

1. Requirements evolver over time consuming, human interactions and communications

2. Formal languages and domain vocabularies have a place in eliciting requirements but we should not confuse them to provide completely unambiguous, clear, testable requirements.

3. Sometimes these formal languages and domain vocabularies are costly, time consuming or simply not feasible.

4. A software specification should cater not only to software program that is being developed and also for the usage and all human interaction related that the program.

What do you say?

Shrini

10 Commandments for Test Automation Outsourcing

2008-09-12T14:52:00.006+05:30

Most of my test automation experience and learning comes from working with IT groups and GUI based automation using COTS tools. Outsourcing of testing is a top agenda on today’s IT manager and automation happens to be one of the popular and “most frequently discussed” item.

I am regularly asked to formulate, present and consult automation projects in IT space. Based on this experience, I am attempting to formulate commandments - about 10 of them. I believe these are few considerations that an IT manager who is thinking about outsourcing test automation work.

Automation is not an answer to your testing problems such as limited testing bandwidth, limited time to test, poor application quality etc. These are true testing problems – one way to address them is to set your “manual testing” right then think of Automation. If your manual testing is poor, throwing automation in that will only cause that “poor” testing be completed quickly.

Automation is White Elephant While automation has clear benefits only when it is treated carefully. Automation is not a turnkey, as your application undergoes changes; automation solution needs to be maintained. Plus there are recurring tool costs, training costs and test case/project management over heads. Is your vendor telling about this sufficiently? Is your vendor downplaying this aspect? Watch out …

Anything that is quickly creatable – is quickly perishable Do not believe claims of “script creation in minutes”, “Automated script generation” etc. Any enterprise level serious automation is software development and requires clear attention and methodical approach to design/create and maintain.

Your business users can not create and own Automation solutions Let your business users/SME’s and domain experts do what they are best at – business support. Do not believe claims of business users creating and maintaining automation solutions. It requires good amount of testing and automation knowledge.

Judge your vendor by the questions they ask about automation Suggesting an automation solution requires thorough study of context of project, release cycles, current state of project, future expected life of application, business expectations, state of automation practice etc. Vendors who propose a “standard”, “gold plated” automation solution without asking questions to probe the context are “simply” selling you something – be skeptical about such vendors.
Higher % of offshoring in automation higher investments to make While moving work to offshore will give cost benefits, when it comes to automation there are number of factors to be considered while deciding how much of automation can happen at offshore. Key thing is about creation of application environment at offshore. If a local application environment can be created at offshore with tool licenses – higher degree of automation work can happen at offshore. It is important to note that for GUI centric/COTS automation tool based automation, both automation tool and application should reside on same machine. Hence pay attention to your application environment and feasibility of creating local environment at vendors offshore location before deciding the amount of work that can move to offshore. Even when such local environment is available at offshore, certain activities like acceptance testing, demo to users, and certain types of test cases will need to be done only at onsite. Be sensitive to these factors. Make sure your vendor asks about this and suggests suitable alternatives.

Pay attention to dependencies and Quality of current test Artifacts A successful outsourced automation requires that project dependencies are well understood by all stakeholders. Access of application from offshore, access to testers/developers/SMEs on test cases, Access to reviewers and code acceptance people are few dependencies that one needs to track as part of project. If automation is on the basis of existing manual test cases- make sure that these are detailed enough and available in a form that can be sent across to vendor offshore team.

Decide Acceptance criteria Formulation of Acceptance criteria is not item that is often given least importance while planning outsourced automation projects. Identify an internal owner in your organization who will accept the code/solution delivered by the vendor. Make sure that this person gets engaged in the project from the beginning and formulates the acceptance criteria along with vendor technical team. Failure to identify automation acceptance person from your end and getting a formal agreement on acceptance criteria can leave you “High and Dry” and leaves an open space for the vendor to deliver any “working” automation code but not necessarily the one that stays for long time.

Avoid linking an automation project (and its deliverables) to application release dates one common mistake committed by IT groups outsourcing automation is to link the automation deliverables to immediate ensuing product/application release and cut down time/effort for manual testing. One IT manager said to his project team “We are having a major release for application XXX in January and we have automation solution coming from a vendor by December. Since Vendor has promised that 90% of manual testing will be automated – let us plan to allocate 2 weeks of testing instead of 10 weeks planned earlier. As promised by vendor we can deploy automation and cut down testing by more than 50%”. What is the problem here? What if Automation delivery slips for the reasons beyond control of everyone? What if development delays? What if due to poor test case quality and fast changing application, automation scripts give inconsistent results? This will result in conflict - “believe automation” or “get some good manual testing done”? Such conflicts can severely impact your releases and hence business plans.

Record and Playback (RP) Automation is for Kids This one point can never be overemphasized. Many IT managers still feel that record playback features of industry standard automation tools can help them to create automation quickly and thereafter their own resources should be able to record and create/maintain automation scripts. However, the experience has shown time and again that RP approach is not beyond learning about how automation tool works with application and can not be used for real time, sustainable automation. If vendor proposes this as a part of the solution – you should be alert and suspect the abilities of this vendor to deliver automation solution

Do you agree with these commandments? Any different experience?

[update]
Some additional tips :
What a Vendor should ask you
- questions about your manual testing practice
- Your objectives of Automation and expectations
- Readiness to take up automation in terms of test cases, application state and environment
- Ask about your expectations on ROI

What vendor should suggest you
- Plan for automation maintenance when supplier is gone.
- Automation may or may not reduce cycle time – that depends upon nature of tests, application technology stack, nature of tool etc
- Automation may or may not reduce the cost of testing.

What to look for in an automation proposal
- Acceptance criteria
- Automation design details – how tests will be structured
- Environment related assumptions
- Pre-requisites about tool licenses, test cases, test data, access to developers, testers and business users (for clarifications about test cases)

You can not automate testing – all of testing as testing is intrinsically a human thinking and investigation activity. What you in reality claim to automate is some portions of testing – namely “test execution” of some select test cases. Note that activities like test design, bug investigation and logging, Test results verification etc are still to be done by a human tester. Automation will take you some places but not all. Unfortunately, the places where it does not take you – are the ones where real problems lie – only a skilled human tester can help.

Shrini

Automation reduces cycle time Part III

2008-08-26T07:39:00.001+05:30

I wrote about it earlier here and here

“If automation can not reduce cycle time, save testing cost and reduce manual testing errors, why do automation at all” said a colleague of mine in a discussion over mail chain yesterday.

After all, automation and test cycle time (whatever may be the meaning and definition) are so “inseparable” – the theme often evokes deep emotional, business, technological thoughts. While I agree with my colleague on the last item he mentioned “Reduce manual testing errors”, for others I would say they are true under a highly restrictive and idealistic conditions. This is similar to the practice in automobile engineering of specifying mileage of an automobile under “standard road/test conditions”. Such claims are made by automobile manufacturers to push their brand as the fuel efficient vehicle. There also goes a disclaimer that says “Actual mileage may vary depending upon the prevailing conditions on the road, driving habits etc”.

This is very true with respect to test automation too. While in ideal conditions and specific types of testing, automation can help in reducing “test execution” (not testing) time, in reality actual benefits from automation may vary depending upon the project context, state of development and testing practice and others.

I see a future where Automation tool vendors are forced to add a disclaimer to protect them from getting sued by someone who just purchased the tool expecting return as claimed in sales and marketing materials of the automation software.

As I mentioned here, testing cycle time is a complex variable that depends upon several parameters and many of these parameters are out off control of automation and even testing. Establishing a straight linear relation between automation and testing cycle time, according to me is a “horrible” and “unrealistic” simplification.

What is (are) the (fundamental) problem(s) that in testing, the automation is expected to solve?

I am afraid, there are very few testers or test mangers out there that begin their exploration into automation with this question. For many, the notion of automation appears to be well understood I am still struggling to answer this question in a context free sense. How can there be a universal solution to a vaguely or incompletely specified problem?

Dear reader, what do you think are the problems that automation is expected solve?

Shorten test cycle time? Reduce manual errors? Help to kill boredom and fatigue due to repeatability? Save dollars of testing cost? Bring consistent testing results? To supplement human testing capabilities?

To be continued …

Software - A game of questions and answers

2008-07-27T10:28:00.002+05:30

"The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question."
— Peter Drucker

"Testing is a questioning process in order to evaluate software" - James Bach

"Computers are useless. They can only give you answers." - Pablo Picasso

"One who asks a question is a fool for five minutes; one who does not ask a question remains a fool forever." - A Chinese proverb

Other day I was discussing with one of my colleague … somehow our discussion went about interpreting in simple terms the whole “game” of software development and testing. Here is what and how ended up in agreeing on the “simple” model to describe software and software lifecycle …(not SDLC but SLC)

Software development is about coming up with answers (and demonstrating those answers with an example) to the questions raised by testers, end users and other stakeholders.

Software testers ask questions about claims and capabilities of what software is supposed to do, take the questions to developers and project mangers and ask for answers.

Project manager or project sponsors scan these questions and pick up those that they think are worth “answering”, prioritize them and pass on the developers for providing answers and ways to demonstrate the answers. Before releasing the software to testers, developer do some question-answer session with buddy developers and leads (peer testing, unit testing and code reviews)

Developers then get on mission to analyze questions and develop/construct answers in the form of capabilities in the software and “release” to testers to check to see the answers are “satisfactory”. When developers do not get answers or feel that it takes relatively long time to find the answers – they turn to project manager with their analysis as why answer can not be made available immediately. Project manager then takes the decision of “deferring” those “unanswered” question to be taken up in future releases.
At times Testers, act on behalf of end users and other

Testers verify those answers and check to see they are OK … some times there will be follow-up questions or new questions (regression bugs/issues) and they are routed to developers via the project manager. This cycle repeats until there are new questions to be answered by the developers.

So … as long as there are questions to be answered about software … there will be the need of developers (who will provide answers) and there will be need of Project managers (to prioritize and check which questions need to be answered) and hence a software development project …

Guess what … it is software testers who drive the whole thing by asking relevant and important questions about software – about it’s claims and capabilities …

So … as important trait of a tester is to practice asking “good” questions …

Shrini

Software - A machine or an organism or ?

2008-07-23T05:41:00.003+05:30

“The first obvious difference between machines and organisms is the fact that machines are constructed, whereas organisms grow. …

Whereas the activities of a machine are determined by its structure, the relation is reversed in organisms - organic structure is determined by processes”

This is how Fritjof Capra opens up chapter 8 of his celebrated book “Turning point”
http://www.mountainman.com.au/capra_1.html

How do we understand software …? As a machine or an organism? Does software grow? How do we understand software by it structure or by observing (!!!) it behavior? What are useful models of software that help us to understand?

What kind of thing is this software? One thing is sure … it is not just code …

A general System …!!!! It is code and about various connected systems … can you think of all those systems connected with software system or systems?

Shrini

Are all best practices "worthless"? Testing Best Practices

2008-07-20T06:45:00.004+05:30

Other day I was quoting following from Jerry’s new book on testing to one of my colleague who is a “best practice” proponent.

…..The risks in these two situations are vastly different, so do you think I recommended the same testing process I used for finding a personal web-writing application? Do you imagine I recommended that my client install random freeware pacemakers into live patients until he found something he liked, or didn't dislike? Why not?

I took above sentences as reference and told him.. “Can you use software testing strategy that one uses for web application writing to that of an embedded software in a heart pace maker? Hence best practices are such a junk thing ...”

To that he was silent for a while answered --- I agree with your point that test strategy or approach used for web application cannot be applied for embedded software in pace maker … How about picking the practice from a same field/domain – will that not save the time, energy and effort for my client ? Let us say I develop a list of practices for a given field (embedded software used in human bodies) and keep “selling” them as best practices (jump start kit) for those clients who deal with such software? What is your opinion? Would you still say … best practices (in a context) are junk?

I did not have a good answer for him …. Then we discussed about “universal best practices” (I am not sure if such phrase exists as all best practices are universal in nature by default and context less??) such as “walking is good for health”,”Test considering end user scenarios”, “Do unit testing” “Do code review”, “Aspirin is good for heart”, “Drunken driving leads to accidents”, “Do meditation to calm your mind” etc. I told him about at least 3 contexts for each of these best practices where following best practices can lead to harmful effects.

After listening to me … he said … Shrini … you appear to be "making up" all these contexts to prove your point …I want you to answer my question – Are all generic best practices recommendations are worthless or fake? When customers want something readymade that will help them to jumpstart the work, they would like to see if I, as a consultant, can bring some “best practices” from my previous similar experiments. Is that expectation unreasonable?

I am thinking ... I don’t have a good answer for him … do you? I hope Jerry would have some answer …

Are there any "universal best practices" or by default all best practices are universal and context free? Will a best practice cease to remain as bet practice once it comes with a context?

[update] Quoting from Jerry's book again - "As humans - we are not perfect thinkers, we are affected by emotions and we are not clones. We are Imperfect, irrational, value driven,diverse humans - hence we test software and test our
testing AND hence test "best practices" that sales and marketing folks associate with software testing.

Shrini

Exploratory Testing SHOCK ....

2008-07-20T06:12:00.004+05:30

A colleague of mine other day expressed his struggle to make exploratory testing work for his time (scalability and making it as a best practice !!!) . He said "Exploratory testing is HIGHLY person DEPENDENT - that is the biggest problem for me ... Do you have any process document for doing best exploratory testing. I will have that included in our testing process framework. BTW that will help us in earning some kudos from our CMMI level 5 assessment team."

I said --" ... that is true ...why exploratory testing ... any good sapient testing is "person - human" dependent. A good testing requires thinking and inquisitive human mind. Are you planning to get testing done by machines, robots - so that your person dependency goes away? If yes .. kindly ask the CMMI team to order few robots for a Testing POC (proof of concept) "

He could not answer for a while ... then responded with low voice "I know you will say something like this only ... I have an answer… Automation!!!! I have raised a request for buying 10 licenses of this # 1 tool in the test tools market .. Howzzzzzzzzzzzzzzzt ?"

Now it is my turn to faint ....

Side Effects of Metrics/Statistics

2008-06-26T19:53:00.002+05:30

Jamie Dobson writes this piece of "reality" with respect to statistics/metrics and numbers.

“... that human beings will always work toward their defined success criteria."

True and very revealing for all metric enthusiasts. Just let people know what they will be measured on - they will modify their work pattern and output to suite positively on the measurement criteria. For example, a test team is measured on number of bugs the team logs, you will see more and more bugs and if a test team is measured on number of test cases they execute, you will see testers executing increasing number of test cases.

One thing that happens is what I refer as "goal displacement" - Goal of doing “required” work getting displaced by doing *that* work in *that* way as described/interpreted by measurement criteria. Can you see a problem here?

When working with social/cultural setup involving human beings, introduction of "monitoring/measuring", typically causes a "shift" in a overall behavior of the group towards "what is being measured" instead of "what is required". We tend to believe that people behave the same with or without a measurement system in place – we are wrong

This is the side effect that I am referring when a metrics program is introduced in a software project setup....

Are you aware of this? What steps can be taken to address the side effects?

Shrini

Software Testing certifications Part II

2008-06-24T14:49:00.000+05:30

Dr Cem Kaner posted a note to Software testing yahoo group on the topic “software testing certifications”. I thought, it would really make lots of sense and value to a discussion about software testing certifications to share those views here.

Dr Kaner quotes following in his note.
http://www.channelinsider.com/c/a/Careers/VARs-IT-Certs-More-About-Marketing-Less-About-Skills/

Continuing my thoughts on certifications, here is something that I would like to add … on the basis of Dr Kaner’s notes.

1. Certifications have value as “marketing” aid and most confuse them to be as means of getting knowledge or experience or learning.
2. IT organizations and service providers use their “certified staff” as “proof” of their well trained staff to their clients.
3. Certifications do have place in hiring. Whether you like it or not, organizations still use certifications as main filtering mechanisms in hiring just like a college engineering degree.
4. Certifications matter for those who are in the initial stages of their career especially those who are looking to get their foot in testing field. Most of the time, certifications get them a call to the interview.
5. Certifications can get you an interview call, might even get you a job but there after it is your skill and work that “keeps” you on job. Do not mistake certification for life time warranty for the job.
6. One very common argument in favor of certification is that “certification help in knowing the testing vocabulary” – This is true to some extent. But while going for certification with this objective, keep in mind that - there are no universally accepted authorities that define and mandate testing terminologies and terms and practices vary across the board.
7. Certification enthusiasts claim that certifications are means of learning and gaining knowledge in the subject. WRONG … there are better ways of studying and learning than going for certification
8. With the help of internet, thanks for Google and other search engines, today the information is everywhere, just look around you can learn lot and gain knowledge by effectively searching the web, reading blogs, writing blog and engaging in conversation with other in the community.

Now about let me talk about those certifications that hold value in today’s world

• CISSP (Certified Information Systems Security Professional] To earn a CISSP, candidates must have five years of experience and endorsement from any professional certified by (ISC)2, the organization that awards CISSP certifications.
• CCNA (Cisco Certified Internetwork Expert]) – this one especially my favorite as it requires the candidate to demonstrate the knowledge as part of the exam in a lab environment – e.g fixing a faulty router.

To summarize, certifications tend to be of value for some (hiring managers and new entrants) and there are some examples of good certifications that test the skill of the candidate. Do not confuse certifications to “learning” and “knowledge” – most of the current software testing certifications are to be used as “marketing” tools.

What if Automation finds bugs ....? Good thing or bad thing?

2008-06-13T15:29:00.006+05:30

Ryan in response to my post on "cycle time reduction and automation", mentions that "It is an accurate statement that automation will not improve cycle time if it finds bugs, where bugs would have otherwise gone undetected. However, I believe the claims automation vendors make, is based on the fact that manual testing is also going to uncover those bugs."

So, when automation finds bugs, your cycle time increases. When people claim cycle time reduction, they "overlook" this fact. why? Those bugs will be found by manual testing too. This may or may not be the case. Bugs discovered by human testing and Automation tend to be of different types.

Now, let us track that trail of what happens a bug is discovered -
In automation - situation could be bit tricky especially when the automation tests, logs are bigger. An error/bug reported by an automation bug needs to be checked to see if it is a bug in automation code or a bug in application or bug in data setup or some timing or synchronization related problem (in GUI automation scenario). Let us say you have 5-7 pages of log file - you will have to scan/read through the log file an locate the bug. You might have to do execute failed automated test manually (and corresponding data setup etc).
In manual testing, human tester can easily trace and follow the bug trail and document the bug. At a high level, bug investigation and isolation tasks tend to become relatively low.

Hence, when automation discovers a bug - things get really problematic.

If one were to cut down cycle time by automation or otherwise, they HAVE to make sure either "no bugs are discovered" or "any discovered bugs are IGNORED" or "bugs that are discovered, if fixed, not tested again and other regression testing is done ....

Can automation control or influence any of above events - prevents bugs being discovered or igonore the bugs if accidently discovered or mandate that bugs fixes will not be subsequently tested?

For the sake of argument, let us suppose that both human test cycle and Automation find same number of bugs ... and take out "bugs" portion of test cycle, how automation can save test cycle time? On what parameters this cycle time reduction by Automation depends ?

Type of test - nature of interactions between "test execution agent" (human or an automated script) and nature of verifications (during and post execution).

GUI Forms with lots of data input fields - can result in quick form fill tests when automated (zero think time).
Tests that require longer processing time can not gain from automation as automation can not speed up the processing time.
Tests that require visual inspection - window titles, error messages, color, tool tip and other GUI centric elements - are better tested manunally as programmatic tests would mean lots of investment. Human testes are quicker and cheaper in such cases.
Result verification that requires detailed analysis, text file processing, database check etc are good candidates for gaining cycle time.

Thus, there are parameters that are beyond the reach of automation ... hence the notion of cycle time reduction has to be really, really taken with "caution".

Shrini

A catalogue of Test Automation Benefits ...

2008-06-13T14:42:00.004+05:30

Other day, some one asked me “in your opinion, what are the real benefits of automation”. That triggered ideas for me to write this post… Let me attempt to consolidate and list all the benefits people claim around “automation”

Real: In my opinion, these are “real” and achievable benefits.

- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency
- Quick – No think time while running tests. (Computer program does not think – So not useful in those cases where you need to think as you execute. How does automation program help you to gain speed when thinking is required?)
- Supplements human ability to spot software bugs
- Helps to run big number of data combinations – can test robustness
- Helps in multi platform combinations (OS/browser/database and other application setups)
- Repeat the testing (test execution) done for configuration A in Configuration B also.
- Hence increased Test coverage
- Following non test execution tasks
- Generate and manage special set of test data
- Large Volume Test comparisons
- Automated workflows and alerts
- Environment setup

Fake: These benefits are on transition point – where the focus slowly starts drifting from “real” to “imaginary” and hence “fake”. These are false promises that are realistically not possible.

- Improvement in application quality (automation can not improve application quality – even testing can not … only developers and business analysts can)
- Improvement in Test process (Test process is pre-requisite for automation)
- Non technical people can do automation – no programming language required
Improved test planning (not sure how)
- By executing a set of test cases for a specified number of times – ROI from automation can be realized. Say in 10 executions –automation test pays for itself after that is “cost saving” all along

Conditional: These are the benefits that are realizable and could be reasonable but under a very strict set of conditions – “once in a while” cases. When these benefits are stated, mostly the conditions that must be fulfilled to realize these benefits, are not stated. This makes these benefits look as though there are real and universally applicable

- Simply – saves time.
- Test effort reduction
- Improved Time to market
- Improved test productivity
- 24x7 testing possible
- Knowledge retention

Totally outrageous: There are some really outrageous and are more or less like a typical sales pitch. People, who make and believe such claims, do not seem to understand human side of testing, testing itself and automation. They are just believe that machines are better than human testers.

- An automation test execution is equal to an hour of skilled human testing
- Automation can replicate human interactions
- Reduced dependency on human testers
- Solves problem of resource crunch and less time available for testing
- Reduced defect rate – fewer defects

Anything I missed?

What is a bug ... A new meaning ...

2008-06-06T00:55:00.007+05:30

Let me give a try to this one-liner ...(short post)

"A software bug is a reflection of the mind of a confused human user"

Analyse this statement .....

[Some updates]
Above one-liner proposed by me, seem to have generated interest in some ... Let me clarify further ....
This one liner of mine is a beginning of an effort to understand human thinking process while he/she sees a bug. A human goes through a series of emotions while dealing with a software bug. A dominant emotion among these is "confusion" - state of perplexity, chaos, uncertainty.
Let us say you are running a test, observing what is happening and doing a quick comparison with what were expecting ... Suddenly something "unexpected" happens, your mind starts remodelling all that is happening, you were expecting "x" to happen, where as you are seeing "a", "b" and "c" are happening - that is contradicting your model .... your heart beat goes up, blood pressure goes up ... lots of physiological changes happen ... your brain tries to stabilize, create a new model on the fly, comprehend a things on the ground and after a while, you calm down and begin to put up an explanation of what is happening and why? A bug report comes out at the end ....

So Bhargavi --- the life cycle of a bug begins with a "confused" mind of a tester. A confusion is a mental state where you are not able to comprehend and reason to the situations and problems that you are subjected to (imagine a traffic cop at traffic jam on road, a kid on first day in a new school etc). It is the confusion that triggers the thinking in the mind of tester, tester follows the trail of the thinking and builds up a new model and finally when is mind is settled, comes up with a explanation. So when a bugs gets nailed down, mind becomes calm.

Even an obvious bug - when you look at it deeply is a result of thinking process triggered by "confusion" what is expected and what is observed. As a normal human reaction to a state of confusion, you reason out the things and make them clear - then bug becomes obvious ...
If an obvious bug is at level "0", confused state is at "-1" level. Also the "sense" of confidence is always followed by a subtle at times quick state of confusion. When try to open your car door in wrong way, open your cupboard keys in wrong position, when you are trying to open a door by pulling that opens only on "push" mode ... our mind goes through a quick state of confusion ... you quickly notice the situation and gain your calm.

It is deep cognitive process .... Psychologists will be able to explain better ... "Psychology of a BUG" ?

Shrini

Goal of Testing and a quotable quote

2008-06-02T15:20:00.002+05:30

Michael Bolton in response to this post from Steve Rowe mentioned this gem …

“Do our automated tests take into account the notion that different people might value different things, and that one of the tester's primary goals is to recognize different constituencies and the ways in which their values might be threatened? “

Many people in our community think that as testers our goal is:

To find bugs
To prove that the application "works" as per the specifications.
Run a bunch of tests and report the results,
Develop some automation to speed up the execution of tests

But to “analyze”, “investigate” the value systems/perceptions of different stakeholders of a software product and explore various possibilities where stakeholder’s value is threatened. This involves among other things – finding some bugs, running some tests, writing some automation etc. Note that End user or customer of the software is an important stakeholder.

This is statement from Michael is an extension or probably logical conclusion of the famous statement that is often associated with testing “Be customer’s advocate” (or “think like customer”). Michael seems to suggest that it is not only customer whom we should consider, as testers we think about all stakeholders and explore what each of these stake holders value.

A stakeholder is a person is who is affected by the success or failure of a project or the Actions and inactions of a product or the effects of service – Cem Kaner

In one of the comments for the same post, I found another worth quoting statement from Ben Walther …

“There will always be inputs into your system that violate the assumptions made in building it. A computer generated test will not be able to violate such assumptions.”

Very true … that is why testing is so challenging and exciting

Shrini

Mission - Test Estimation model

2008-05-29T21:22:00.003+05:30

It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.—Fred Brooks

Steve McConnell opens his famous book "Software Estimation: Demystifying the Black Art " with above statement ... Development community is fortunate to have some one like Steve McConnell to help them with solving this puzzle called "Estimation"

I am working on developing a test estimation model. I think in today’s testing world, this is a biggest and the most complex testing problem to be solved. As I am reading and researching about this topic and formulating my initial thoughts, I am thinking about testing, testing models, questioning, thinking, modeling, bug investigation, test-stop criteria, non-linearity, size of testing work and so on.

Following is a list of challenges/questions that I am searching answers for …

1. Model of Testing: Testing is an evaluation activity as against Development that is a construction activity. How to think about estimating a “evaluation/Analysis” based activity?

2. Sapient Testing model that requires critical thinking, questioning, modeling is essentially a non linear activity where as development can be a relatively bounded activity of starting from a spec to a working code. For example if a particular testing task takes x hours, 10 such units will take 10x hours … we can not make such extrapolation right?

3. Sizing testing activity is still a big problem – what is the unit of testing task? Atom? Or molecule? What is the building block?

4. Tasks like bug finding, investigation, test design are difficult to quantify?

5. When do we stop testing? What are exit criteria for test cycle? This will set upper limit for testing scope?

6. How many test cycles we need? How do we estimate?

7. What are the human factors involved the process of test estimation?

8. How do we address problem of "slipped bugs" in case of plain and straight forward "scripted testing" (write test cases and execute them word by word and log the bugs and end of the cycle). What about bugs or tests that come out of specs, exploratory tests? (Something that we worry heavily in IT services Industry – how much testing to do? There are penalty clauses for missed bugs)

9. What about productivity numbers used. In estimation -> we first talk about sizing the work, then apply productivity figures to arrive at effort then split the effort into schedule. How do you deal with numbers like – 10 test cases executed per person per day? Should you believe them or question what a test case is?

10. Is estimation a guess work?

11. Is estimation similar to the work of a fortune teller, weather forecaster, Election analyst, Stock Market analyst, A punter in horse race, or a gambler? After all in test estimation we tend to predict the future right?

Any views? Are these questions important while arriving at a test estimation model?

Interesting quotes on Prediction:

"Those who have knowledge, don't predict. Those who predict, don't have knowledge.” - Lao Tzu
“Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” - Peter Drucker

Shrini

Can Automation reduce cycle time or improve time to market?

2008-05-27T19:45:00.003+05:30

Continuing on this discussion on test effort and manual testing – there is another popular variation in which Automation tool vendors claim “improved time to market” and/or “reduced cycle time”. In this post, let me dissect these claims and see how true and credible is this claim.

First of all let freeze what one mean by “Time to market” and “Cycle time”. Let me define the terms as bellow in the context of a traditional/Waterfall model type of software development.

Time to market is a time window between the time your start development (requirements and so on) till you ship the product in market for general public consumption. Depending upon whether you are doing a major release or minor release, this window may span from few months to few years (as in the case of Windows VISTA)

Cycle (when used without any qualification indicates a cycle of development and Testing) time is a time window for a software product under development. The cycle time can be divided into Development time and testing time. A development cycle starts with deliberation of requirements, design of features to be implemented. A test cycle starts with a development team “releasing” the code for testing to begin and test cycle ends with test team completing the planned testing for that cycle. During this period development team can fix the bugs reported. Hence for all practical purposes, a cycle time implies a time window between start of design/requirements until test team completing the testing/development team is ready to release next build.

So it is apparent from above definitions that cycle time is a subset of time to market.

Automation can reduce cycle time (less of a problem) only if (check how many items or situations that “automation” can control)

Automated tests run without reporting ANY bugs on the software (A bug reported by automation means – some investigation and confirmation by manual execution that the bug reported by automation is indeed a bug with the application)
Automated tests DO NOT report any run time errors (A runtime error by a automated test means some investigation and re-run.)
Development team INSTANTLY fixes any bugs reported by Automation and these fixes are so well done that there will be no further verification (manual or automated) required
Manual testing (a very small portion indeed) that happens in the cycle does not report any bugs. All the manual tests pass.
If manual testing reports some bugs those bugs will be fixed INSTANTLY without requiring any further verificationBug reporting time, triage, investigation (if any) is so small that they are negligible.

Automation can reduce/Improve time to Market only if –

All the items mentioned under “Cycle time” and
Business stakeholders do not worry about outstanding bugs. They take the decision to ship the product as soon as Automation test cycle is completed (because automation cycle is NOT expected to report any bugs). So the end of the automation test cycle, shipping the product is a logical thing to follow.

If you analyze these situations, you would notice that many of the factors that influence cycle time or time to market are not under the control of “test automation”. These factors are to do with development team, quality of the code, quality of requirements, number and nature of the bugs reported by both manual and automated test execution and above stakeholder’s decision about those bugs reported. One can not claim that there will be cycle time reduction or improved time to marked JUST because x% of test cases are automated. A big and unrealistic generalization - only an automation tool vendor can afford to make.

So next time when someone says automation reduces time (either cycle time or time to market) – do quiz them and say “What do you mean”?

Bonus Question : Can automation accelerate your Testing? If yes, under what circumstances?

Next Post : Can automation address the IT problem of Limited (human) resoures and tight deadlines?

Shrini

When method dictates Goal – See Goal displacement in action - II

2008-05-17T11:26:00.008+05:30

I wrote about Goal displacement part I here. Surprisingly .. no comments yet ... :(

Here is example # 2

Software Testing Certifications: A conversation between a certification enthusiast (CE) and a critique (CC – that is me)

CE : Do you know about this certification for software testing, this is very popular in Europe and US?
CC: Yes, I do but I am not sure if this really helps in evaluating skills of our testers.

CE: I think it does. I have heard about exam it seems pretty exhaustive and coves all aspects of testing.
CC: Umm … what do you think the certification is testing? What and how does it evaluate testing skill of the candidate?

CE: Certification tests the “knowledge” of the tester, familiarity to terms, definitions and experience of the candidate…
CC : Really? How? How a certification tests knowledge and experience of the tester?

CE : By carefully selected questions and evaluation by testing experts. The certification exam is based on a body of knowledge and questions – mostly objective type. I think one can rely on the exam.
CC : So … you are saying an objective (yes/no and multiple choice) type question paper is used to test the skill and experience of a tester. Does the exam allows theory type of questions? Can the tester debate on an issue? Does the exam involve any kind of putting the tester into real testing situation? Does it observe the tester in action ?

CE : Come’on, how it is possible? Certification bodies have their limitations, they can not set up “practical exams” to watch tester doing testing and then rate. Do you want certification body to set up a audio/video facility to allow for debates, questioning and real time situation simulation?
CC : Don’t you think, that you would right and reasonable way to assess the skill of a tester not using a fixed set of questions that emphasizes on memory recall, reproduction of text of study material?

CE: That is right … Look at feasibility of having such exams … what about cost of administering such tests? What about evaluation? It would be costly. That is why the certification bodies might have created a scheme of tests that are easier to evaluate and conduct on mass scale. This will enable a relative cheaper exam and allow more people to take the exam Right?
CC: Well …good point. But what is the goal of certification exams? What they attempt to achieve?

CE: Evaluate and Assess skills and experience of a software tester.
CC: But your current exam seems to be structured in such way that it is easier to administer and evaluate.

CE : Ummm… that is correct.
CC: See this is what I call as Goal displacement. Certification bodies wanted to design, administer and evaluate a system of exam to evaluate and assess skill of a tester. But they seem to have taken the path of designing an exam system that is easier to administer.
The goal of software testing certifications is to act as a mechanism of evaluating skills in software testing and provide a benchmark for the talent in that space. As the popularity of such certifications grow, so grows the need for system for mass administration and evaluation. This causes the change in the examination pattern and evaluation mechanism to facilitate the mass administration. Any part of the exam that is good from skill evaluation perspective will be dumped if that does not lend itself to easier evaluation of exam results.

Side Note:

Software estimation models (especially test estimation models) often suffer from Goal displacement problems. Other day I told my colleage " The reason why we struggle with test estimates is that we use simplified model of testing while estimating but reality is different hence estimates are typically go wrong". For that he said -- "OK .... we know that actual testing models are complex (non linear and involve critical thinking/Questioning etc). That is why we use simplified models of testing ".... How strange ..!!!!

This is another example of goal displacement right away ... To make estimation possible - change the testing model itself ... use a simple one. So the goal of using a good and resonable testing model that happens to be complex (and does not lend itself that easily to estimation) is replaced by a goal of finding a testing model that is simple and easier to estimate ...

what should be your goal - good testing or estimation?

Shrini

Can Automation reduce human testing effort ?

2008-05-17T10:33:00.004+05:30

This post is for all those IT managers who are considering outsourcing testing and listening/talking to various vendor presentations on testing and automation.

I was in a conversation with a client other day. He asked me “Can you reduce the number of resources that you have deployed currently for regression testing of this application to say by half in next six months or so by using “Automation”?

I was not shocked to hear this since for past few years, I have seen many clients in IT application testing space have had similar expectations out of outsourced testing and hence on automation. In an Outsourced IT application testing domain, this is how a service provider positions Automation – A means to cut down the cost of testing (resources).

Client’s expectation of reduction in testing resources (with no reference to scope of testing) using automation in a specified time frame stands on following assumptions or beliefs.

- Regression testing is simply executing a set of test cases and reporting results nothing more than that.
- Regression testing gives the confidence that nothing is broken – all that was working previously is “intact”
- Automated test cycle means zero human involvement hence percentage of automation should directly result in proportionately reduction in human effort of testing. For example if 50% of test cases are automated, effectively manual test cycle time can be reduced to that percentage.
- Automation is turn-key. Once you have an automation solution developed, one can just keep using it unlimited number of times without any extra cost and effort
- Operationally, automated test cycle ALWAYS takes less (fraction of manual test cycle version) time – there would be no additional effort required to investigate results and chase script failures.

Let us look at costs of creating and owning automation (note that majority of these are of recurring types)

- Tool evaluation, proof of concept and other initial investigation costs
- Automation tool cost and recurring licenses/upgrade costs
- Resource Training costs
- Costs associated with automation environment (development and execution) – servers, applications, connectivity etc
- Costs associate with manual test cases cleanup, rework, test data, clarifications and any other effort to facilitate automation
- Automation development/ testing/review and acceptance costs
Costs of setting up and maintenance of automation execution environment
- Costs of investigation of automation results, failures
- Costs of re-runs (testing automation code fixes) and any required additional human testing
- Costs of maintaining automation code – design and structure of automation extent and frequency of application changes, automation tool changes (new versions etc) and change in application platform determine this cost component

How do these cost components compare with costs associated with human testing? Will an automated test cycle always be quicker than human testing cycle? Is speed of excution all that matters to you? Can these two versions of testing cycle be compared? Can one hour of automated testing (test execution) be compared with 1 hour of skilled human testing?

Test automation if not “appropriately” applied can be highly expensive and painful. If you are dependent on automation for achieving project GoLive timelines – be aware you could be risking the release. When someone says they can reduce testing effort by automation – be sure they are selling you something and they do not understand fully what is testing and what automation is.

Test automation is High risk item in business. Before you invest money or even before you make any business decision that based on automation capability – make sure you, as an IT manager is aware of darker side of automation.

If you are getting a sense that I am discouraging Automation and regression testing – Yes, I am. Considering the approach that today’s IT World is taking automation – I am against such approaches to Automation – but NOT to all other forms of automation that can happen.

So next time you sit in a vendor presentation and hear this "Test automation reduces manual test cycle time and effort" - Please ask "HOW" and "UNDER WHAT CONDITIONS". I am sure the presenter will have tough time in answering your question as these are not a common questions to these "automation snake oil" seller. In case if you get an answer, make sure to check relevance of those parameters to YOUR project and application context.

Shrini

I am a sapient Tester …

2008-04-26T00:32:00.003+05:30

Here is my first and a very crude attempt write a poem (if it can be called as a poem) on sapient tester ...

I understand that testing is a sapient process *
(* A sapient process is any process that relies on skilled humans)
I question things around me
I model things to understand their behavior
I learn from related fields in testing (ones that are close require higher levels of human cognition)
I use my brain and I constantly try to improve my thinking
I don’t believe in rote processes that mechanize human thinking
I don’t use documentation, KA/KT* as essentials for getting started with testing
I can figure out my way in understanding the software I am testing.
I use automation as aid in manual testing not as a replacement
I use techniques as heuristics
I just don’t simply follow the written instructions and procedures - I question them
I collaborate with developers, other testers, users, support personnel and other stakeholders to understand wide range of perspectives.
I understand depth and value of human testing
I understand that by automating a test, I might be loosing some thing.
I am sapient tester... Are you?

Association of software testing (AST) publishes a magazine called "sapient testing"

James Bach wrote about sapient processes here

* KA - knowledge Acquisition and KT - knowledge Transfer

Shrini

Thinking about Zero defect software - Attaining software Nirvana ...

2008-04-26T00:27:00.000+05:30

“Humans have always proved impossible is possible … Dreaming, Desire and passion is key to achieve “impossible”. Wright Brothers dreamt about “flying” – they achieved it. Once, reaching moon was impossible – now? You need to give your life to it …Zero defect software is possible …” You need to aim high”.

This is how, a colleague of mine argued with me when we were discussing about Zero defect software. He believed that many software like IBM mainframe apps are running with near zero defects … (there could be defects but of cosmetic nature)

The notion of defects, number…

No defect is bigger or smaller … Let us say there is a 0.5 second delay in system response … Will this matter? Is this a bug … depends upon which software we are talking about? Context is important …

When people talk about software processes, discipline and making an attempt to achieve zero defect … they are often of the opinion that …
Human beings make mistakes deliberately … many or all can be avoided if there is a second eye or a watch dog
Human do sloppy work unless controlled – given a choice no one would do a good job (if no one is observing)
Humans require constrained and regulated environment
Humans - overlook

It is this possibility that people require discipline makes others (mangers especially) to think that zero defect is possible. People who vociferously argue in favor of zero defect software – think so because they feel that it is because of human laziness and other related aspects – defects are introduced. Put a system of governance, policing – you will achieve zero defect. How true is this?

Chasing the definitions of Quality ....

2008-04-26T00:19:00.003+05:30

Why do we have so many meanings/interpretations/definitions of quality - A colleague of mine asked recently. I did not have any ready answer for him ... I said ... after a pause ... "There are so many ways to see the quality". Because, I think every one, whoever attempted to define quality (from Juran to Weinberg, ISO to CMMI), did so wearing a "user's" hat while they were not real users. It is like doing a role play temporarily - a third party view.

Michael Bolton mentioned this elegantly...

Quality is not an attribute of a thing or a thing (most of us believe that is in a thing). It is a relationship between the user and the product.

So extending Michael’s view point - quality is a two way and intimate relationship between the user and product. There is no room for third party.

So as a third party everyone can come up with "possible" meanings of the term quality.

The moment we understand that it is relationship on a timescale (perception of quality does change over time) not something built into the thing ... we will be good. Beware a notion of good quality does not alway remain so .... as time changes so does the perception about quality in the minds of the beholder.

What does this mean to we, testers?

- As testers we do a third party assessment of (to best of our abilities) the relation ship between a product and it's anticipated user base.
- Things like fast, great UI, cheap, reliable are only manifestations of relationship between user and product - not real things or attributes about quality.

- Quality is observer dependent - quality can be viewed always and only viewed from the eyes of the user's frame of reference
- User's taste, perception of quality for a product changes. Hence there can not be any "one-time", "fits all size" kind of testing/assessment for a product (more so .. if it a software product)

Who is next in the line with a definition of quality?

Shrini

When method dictates outcome and Goal – See Goal displacement in action - I

2008-04-26T00:13:00.004+05:30

I first heard (came to know) about this word from James Bach and this immediately caught my attention. I started seeing it everywhere. The phrase "goal displacement" is generally referred to a situation where method of achieving something dictates the out come and goal. The process by which the means used to achieve a goal becomes more important than the goal itself.

Here is an example that I think illustrates what I mean by “Goal Displacement”.

A model of Test Automation

Manager: I am quite impressed with overall progress this automation project. But I would like see some metrics and measurements to quantitatively monitor future projects in this area – like number of scripts produced per day, individual automation engineer’s productivity etc
Me: Ummm … right now, our model of automation development does not itself to such measurements. We don’t treat overall work in terms of scripts and measure engineer’s productivity. I am afraid; I don’t have an easy answer to your question

Manager: I understand your concerns. But I need those metrics.
Me: Here is a deal, how about modifying the whole process of automation such that it is easier to measure and exactly built according to metrics that we would like to measure (regardless of what outcome we would like produce)?

Manager: I did not understand that … Can you not just introduce them to existing model without affecting anything else - especially the output?
Me: To accommodate the kinds of things that you are asking, I will have to modify the way we develop things – which I am afraid will affect the quality of our deliverables. I don’t see any way to make those measurements in current model.

Manager: Metrics and measurements are not suppose to change the process instead I heard that metrics help to control and improve the process.
Me: In this case they do. Do you want me go ahead and change? I see a serious, negative side effect of changing the process/model to suite metrics.

Manager: Ummmm … Let us talk about it some other day (walked away)

What is happening here? Manager wanted to see some metrics to see how the automation team is performing. I simply said "You can have those metrics but at the cost of changing a process that is apparently working fine and producing the results that make you happy". So here, the introduction of metrics (or a measurement process) causes a shift in the goal of producing good automation code to goal of having a process that measurable in some way. It is like saying if current process (way of doing things) does not cater to a new related expectation, possibly a conflicting one with original goal (a displaced goal) - then forget about goal - do whatever change required to meet new goal or expectation.

Stay tuned for two more examples of goal displacement ...
Shrini

Software Testing certification - To be or not to be ..

2008-04-25T23:25:00.003+05:30

I am yet open up my thoughts on certification ... also I have not been very critical of certification programs. Here is an attempt to publish my personal views on this topic ..

In 2003 I took CSTE exam ( A link from QAI is here) - as asked my manager and cleared with flying colors - 90% above ...My preparation for the exam was painful ... I had scratched the CBOK (reference material) all over as I did not agree with most of material - which seemed to copy paste from a software engg text book and about 20 years old stuff. I could memorize lots of stuff without sufficiently challenging it. I passed the exam ... did I learn anything new that helped my work --- NONE other than some terms and their meanings (questionable).

I also constantly watched the kinds of people who passed such exams .... I must the quality was bad. Passing was easy ....In my opinion, most of certifications (I am a certified Java programmer too) suffer from this problem. There is so much focus on memorizing things not about learning.... There is little space for debates and questioning... That is you questions do not have "comments" field .. or "I don't agree with premise of the question because...." kind of open ended response from the candidate .. Who has the time to site and evaluate all responses ...?

Discussions on certifications often take "Emotional" angle - people say exams like BE and certifications help one to improve skill and knowledge .. I would say they only provide reading material .. rest is in your hand (rather in mind) . I strongly feel that skill comes first and gets developed by practicing, doing, spending real good time with dedication not by passing an exam that tests your memory retention capabilities.

The problem I see with testing certifications is that - they don't test testers skill to do testing ... (a practical exam like our science lab in schools might help) instead they test tester's memory retention skills.

Are there any certification exams that watch tester testing stuff (with some video etc) and then give "good to go" Tag? No .. that would be expensive and tough exam to administer right ...

An example of goal displacement ... certification exams appear to be designed for easier and large scale (yes/No or choice) kind of assessment than complex/tougher assessment of watching tester in action and rating his/her level of skill.

Why goal displacement -- Some group of people in testing thought - we need an exam to test our testers .. so what kind of exam should we have? ... something that is easier to administer or something that tests tester in action... They prefer to choose the first one ... So the goal of having a good certification got shifted to having a certification exam that is easier to evaluate ...why ? Good certification exam requires complex models and involve lengthy/costly administration. It costs lots of money...

Having said all this ... its your call .. I am neither in favor or against to certifications in testing (some people 's daily bread comes by running those exams -- let us not take away that ...). It is a big business in testing industry today. Being testers, let us evaluate what is good for each one of us (considering each ones technical and educational background) and take the decision.

Any alternative to certification ....?
Focus on skill ... how many of us can go to an interview and challenge the interviewer "Give me a software to test ... give me an hour ... come back I will have a defect/issue list ready". How many of us can take this challenge of "testing anything, anywhere, under any time frame" ? Managers in IT companies need to see evidence that this person can test ... he has tester attitude...

In absence of such evidence, managers resort to something that easy to check (do you have this certification?). If you are fresher or beginner in software testing field ... and struggling to get a break or good job ... You have two options ... Get certified and get a job .. (there after it is your skill that keeps you on job not the certification) other option ... practice testing, read stuff, read books, learn programming language, debate, sharpen your analytical skills, write blog, discuss with others in the field .. build a credible profile for your self (Google should be able to find you) and go confidently to any interview and say "give me stuff to test ..."

First one is easy path ... some money and about few weeks of reading, take the exam your are done... second one is tough one.. Requires you to commit to the profession ... requires your learn things, may take years to be good at testing, many years to build a credible profile that speaks for itself ...

People choose first one mostly as it is easy ... So I would not say all who support certification are not skilled people and take short cut to success neither all those who oppose certification are good skilled testers …

You want to be certified? it is your call -- What ever you do, be a tester by heart - test everything/question everything until you are convinced

Read James Bach on Certification here

Michael Bolton on why I am not certified

Ben Simo on his favorite certification

Balancing on the other side -- AST certification debate

Shrini

18 Myths Associated with Exploratory Testing ...

2008-03-17T08:59:00.004+05:30

Myths are public dreams; dreams are private myths – Joseph Campbell
Science must begin with myths, and with the criticism of myths - Karl Popper

Instead of writing on what is exploratory testing, to me, it looks easier to start of explaining what is not exploratory testing. For those who are hearing this word for the first time – let me remind you this is a testing approach that involves simultaneous test design and execution with an emphasize on learning. The term exploratory testing is coined by Dr Cem Kaner around 1983.

Exploratory testing as approach has a legacy. Its predecessor “Ad-hoc” testing is a perceived as spoilt kid. Until recently, I was under the impression that adhoc testing is quick and monkey type of testing (often connected with sloppy testing) where you just play with application in order to find some bugs that are out of specification. I was proved wrong during a rapid software testing workshop that I attended at Toronto. James Bach mentioned in that class that the word “Adhoc” means “to the purpose”. Most people believe (as I did initially) that Adhoc testing is undirected, random clicking of mouse and navigation. This is probably the reason, why Dr Kaner decided to name the baby differently so that the “stigma” associated with the approach of testing goes off.

I think Dr Kaner did not stop at just naming the baby, he along with James Bach, Michael Bolton, Jonathan Kohl, Jonathan Bach and others did lots of research and practice related to “exploratory testing”. Today, if many of us talk so confidently about this – it is due to the pioneering work done by these people.

Let me also remind that exploratory testing is not a testing technique but an approach and opposite of exploratory testing is “scripted testing”.

There so much of confusion, skepticism and disbelief associated with this approach to testing. Based on my interactions with people from all walks of testing world, I have drawn a partial list of myths that are associated with exploratory testing.

Let us go straight into the list of myths …

Why bother about ET?
1. ET -- every one does ET while doing testing - why bother? why have a special name?
2. ET is some kind of snake oil - people use this term to indicate some mystic thing to make money
3. ET will not work in my context - I am not even willing to give it a try as I am not convinced we should give a try. I know it for sure.
4. ET does not seem to have come under the radar of Gartner or Forrester's -- it might not be that popular.

About the form of ET
5. ET - is an unstructured, ad-hoc (meaning sloppy) process.
6. ET is nothing but doing testing without any formal test cases.
7. ET is instant bug hunting process.

About ways of doing ET
8. ET should always be done after completion of all planned scripted testing (if time permits) If you have time (while you wait for a new build), do ET to use your time productively.
9. ET is not predictable and repeatable (was it meant to be repeatable?)
10. ET shows scant respect for well established Test techniques - how can you do testing without using any of those time tested techniques

About skills required for ET
11. ET requires in depth domain expertise hence only domain experts can do it - it is not every ones cup of tea
12. ET seems to require special skills (domain, quick learning etc) - hence we do not do it.
13. ET is highly person dependent, unacceptable to us - we are resource starved industry.

About suitability of ET in a context
14. Our quality process standards require that every test effort be substantiated by a detailed report of what test cases are executed, on what platform, what data is used, how many test cases passed, how many failed etc. ET is not so good in producing such report. ET does not provide enough evidence/proof of having executed testing.

About perceived Value of ET
15. ET is not process oriented and is not methodical (requires highly skilled disciplined, responsible testers). Anything that is so “person/Skill” oriented is strict no-no in our environment (process driven)
16. ET can not be outsourced - even if it is we can not assess the progress - it is an uncontrollable process.
17. ET can not be automated - I am looking to reduce spend on testing - ET can not help me there.
18. ET seems to be useful in only in those environments where there are no requirements but ours is a very structured process. Why to use ET?

Why these are myths and not the truths – that is an exercise to the readers. I welcome each one you, the reader to challenge my claim that these are myths and lets debate …

Shrini

Here comes my 100th post ...

2008-03-17T08:18:00.001+05:30

4 Years, 100 Posts, 200+ Comments, 15+ Commentators, about 5000+ page views (statistics since Dec 2007)

It all started in July 2004, on 23rd I posted my first blog post. Why did I start blogging? I just wanted to join the bandwagon of my colleagues and make presence in blog-o-sphere. . I could not blog much in my first year. I struggled to decide what to write and often felt choked after writing few paragraphs or sentences. Mostly my initial posts referred another post or some news which I announced or made reference in my blog. I did not have any significant stuff to offer to the readers. I think this is what people call as “writers block”. I suffered with that in my initial blogging days.

Things started to change as I wrote more, attracted few best minds in our industry to my blog. As I started receiving more comments, I, all of a sudden, started to see lot more ideas to write and debate on. Many times I asked myself, why do I blog? Got only one solid voice from inside … “I wish to communicate …I have something to tell to testing world”. So, all these months and years, the notion of “idea or urge to communicate” drove my blogging. I felt very happy and satisfied after a posted a blog post. It was like winning a game, or scoring good marks in an exam. A good (in my own terms) blog post always energized me.

After posting, I would monitor my blog for comments. Depending upon the content of the post, I expect 2-3 comments in 24 hours of the post going live. Sometimes I loose sleep if I don’t comments on a post (having a content that I believe is thought provoking and interesting). I then, shamelessly send mails to my frequent readers (l have a list) asking them (explicitly) to comment on the specific post. As far as possible I avoid doing this as I feel this is similar to a well dressed girl going round and asking “how do I look” - a desperate attempt to grab attention. Usually best responses come without asking for it.

In my opinion, a success of blog can be judged in terms of the number and quality of comments. Number – because it is an indicative of number of readers of the blog who not only spent time with the blog also went further to write a comment. Most of the blog readers today use blog as one way communication tool – it is not. Some well moderated blogs like the ones from James Bach, are effective 2 way communication mechanism.

Many times, I think of getting on with “Google Adsense” …in current shape and size, my blog even may qualify for it. I am resisting … I personally do not like the blogs with “google adsense” as it distracts from reading. I see a blog as a library of interesting topics often focused on a specific area. Anything else is distraction for me. I don’t want to loose that appeal from my blog by opening it up to Google.

As a blogger I must avoid doing

without leaving any clue, stop blogging for months and then come back and say “ sorry It has been long time since my last post” … Readers tend to develop some inertia when then see a lull … So keep blogging (one in a month at least .. this time can vary from individual to individual).

Discourage or block comments or make the process of commenting too tedious. To me comments are very important as they are only available mechanism of getting feedback from my blog readers.

As blog reader of other blogs, I would not like if some one does not respond to my comment (especially where I challenge the post content). So if a reader of my blogs posts a comment asking me to clarify – that would be my first priority.

Post reference to other blogs without having your own thoughts of ideas

Short posts like “I am back or here is an interesting thing” … No short posts unless there is real need.

These are my own rules of blogging – I plan to follow them religiously and make sure my readers get best experience of reading my blog.

Benefits of blogging (at least to me)
This blog helped me in connecting many unknown faces. I have few students asking for some help on a topic on testing, I got someone asking me to do a software product review for me. I got several job offers from the readers (that were looking for hiring me), I made new friends via my blog. I have great names in testing World like James Bach, Cem Kaner, Jerry Weinberg – commenting on the posts of my blog.

My message to my blog readers –

Keep checking my blog and generously comment on the posts. You can also use my blog as a platform for debate on something other commentator said about a post. For example you might want to react to a comment posted by Pradeep on my post. I welcome that. This will make my blog as multi fold communication – a discussion forum. As far as possible avoid giving very general comments like “Great post or I liked this blog etc …” As I welcome the appreciation (at times I need it) but would like those items to be in 1:1 mail to me (shrinik@gmail.com) – for a reader, comments like “ I like this post or this is a great blog” not very useful. I encourage my readers to challenge me more often – critically analyze the views presented here … write about it and leave a link. I think such challenging comments are quite useful to me and other readers as it presents a different view point. Do write to me about the list of topics that you would like me to blog.

There is one contradiction that I am not able to solve. If I am blogging for myself (for my satisfaction) should I bother about my readers – their choices, likes/dislikes? Should I worry about responding to the comments?

Should you bother about me blogging or not …. What do you say – Dear Reader?

Software Requirement Heuristic ...All requirements are Vague

2008-02-13T12:54:00.001+05:30

Here is a heuristic - "All formal/written requirements are vague – only exception is when requirements are collected using a process that does not involve any human”

(Wait … if at all such process exists, where it came from? There must have been a human being such a process needs to be designed by a human … …hence there is a human element here too … Is above heuristic, an axiom or hypothesis or law?)

So there will be always gaps between "Customers view", "end user view" and programmers view". ET can help you to explore these gaps.

Jerry Weinberg in his book on Exploring requirements.. - gives the example of a nursery rhyme "Mary had a little lamb" and goes on to prove that there can be at least more than a dozen ways of interpreting above sentence ... Remember we typically deal with requirements documents of 50-60 pages ... Imagine the amount of misinterpretation possibilities....

I was discussing with this with Michael Bolton, recently. I said “I can say for sure that as long as requirements are written in English, they will be ambiguous. English is a funny language". He immediately shot back and said "you can say same thing with Hind, Tamil, French or any other human language"... I laughed and said “well, you are right, how about this ... One way to get unambiguous requirements is to express them formal languages like computer programming languages ... say Java. "Class MyClass" in java is nearly unambiguous". Michael continued "Here, the ambiguities come, not from written requirement but from the process of translating what is there in one’s mind to formal language".

That brought us to the real truth about software requirements ... "As long as one human tries convert what is there in her mind to some thing that written form; mind to document or mind to mind data transfer .... There will be ambiguities"

People can say a thing in different words but might mean one thing or they might say one thing and might mean differently....

Welcome to the world of software ... a discipline with high human content ....hence we need skilled software testers to keep world "lighted"

First thing you need to acknowledge is Requirements are fallible - they can be wrong - horribly incorrect (so will be corresponding program). If you base your entire testing only on requirement - you can go wrong by miles ... beware of this requirement based testing trap...

BTW there is a type of testing called requirement based testing ... can you imagine how narrow it can be?

You might have heard of people counting requirements … Can we count requirements as we count apples and pizzas?

What do you say about this concept of Requirements Traceability Matrix (RTM) – a holy thing that most swear by? Fundamental principle on which the concept is that “requirements and test cases can be counted”

Please… for God’s sake understand requirements, bugs, test cases are not real things and can not be counted in a way you count physical things, they are mere concepts, ideas …

Reductionism and Test Techniques - who, what?

2008-01-16T15:31:00.000+05:30

Scientific reductionism is an undeniably powerful tool, but it can mislead us too, especially when applied to something as complex as, on the one side, a food, and on the other, a human eater. It encourages us to take a mechanistic view of that transaction: put in this nutrient; get out that physiological result.
- Michael Pollan

I have heard statements like the ones below several times … I am sure many of you have.

- Orthogonal Array or pair wise technique reduces the number of test cases and hence can optimize the test effort

- Use of orthogonal array based testing (Test design technique) has demonstrated to produce superior test plans that improve testing productivity by a factor of 2.

- Equivalence class partitioning (ECP) is a functional testing technique that systematically reduces the number of tests from all the possible data inputs and/or outputs, and it provides a high degree of confidence that any other data in one particular subset will repeat the same result.

- Boundary value technique catches the bugs occurring near boundaries. Use of boundary value technique increases the effectiveness of test coverage.

What do you notice common in all these statements? Faith and reductionism. When you say “Technique XXX does this or help in this [reduce the number of test cases], you are forgetting that it is not the technique but it is YOU as someone who is making an assumption/assertion.

For example, when you say “Orthogonal array technique reduces the number test cases” – you actually saying, I as a tester using this technique making assumption that only pair wise test cases generated by this technique matter – rest I ignore.
Or
When some one says “ECP technique systematically reduces the number of test cases” – that is a view of reductionism. It is tester's hypothesis, judgment and assertion that certain sets of data values CAN be treated equivalently. ECP does not "reduce" ANYTHING by itself systematically or otherwise. ECP's principle per se is that there are groups of data that can be modeled to be treated identically by the AUT

I can say that, this is a real difference. This shifts the focus from the technique to the person who is using it. You then can not shift the blame to the technique if it fails.

As testers we use lots of tools, techniques and heuristics to understand, operate and observe the test objects. Every time I use a technique or a heuristic, It is my explicit choice and I make a set of assumptions and assertions. The goodness and applicability of results of the test or technique, directly depends upon these assumptions and assertions. Technique/tool has no role in it.

Pradeep Soundararajan and Michael Bolton have it here – A focus on tester not on test or a tool or technique…

http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html

If you take a gun and kill a person – what do you say? Gun killed the person?

Shrini

Percipient Subject and Software Quality

2008-01-08T13:37:00.000+05:30

Jerry Weinberg in his famous book “An Introduction to General Systems Thinking” mentions quoting Albert Einstein…

“Belief in external world independent of percipient subject is foundation of all science”

Let me extend this to Quality (or Software Quality) …

“Belief in external world independent of percipient subject is essential for the notion of Quality in software world”

All definitions of Quality (except that of Jerry) from Juran, Deming and others somehow assumed the existence of “external world independent of percipient subject”. Hence Quality has been more or less defined in absolute terms.

Let me remind you -
- Quality is value to some person (one who matters)
- Quality is not an attribute of a *thing” – we can not measure Quality but can only assess it in “relative terms”
- Quality is in beholder’s perceptions.
- Quality is not an intrinsic attribute of any “thing”
(Quotes from Jerry, Michael Bolton and others…)

Bonus Tip:

“Belief in external world independent of percipient subject is essential for the success of a metrics program”

Most of Metrics programs that are claimed to be successful stand on this belief.

Howzzzzzzzat?

Shrini

7 Habits of successful Testers

2007-12-31T17:41:00.000+05:30

Here comes the last post for the year 2007 - to be expanded later ...

Self Driven or high levels of Inner drive for learning new things – No fear of unknown.
Spontaneous – Thinks on the feet – Good in emergency response.
Agile and adaptable.
Love for Science (Physics/Chemistry), Mathematics and Philosophy.
Love for problem, Puzzles.
Hunger for self Expression – Writing, speaking.
Organized Skepticism and constantly challenge their own thoughts

Shrini

Exploratory Testing challenged - Part I

2007-12-29T16:57:00.001+05:30

Here is a conversation that I had recently with one of my manager. We were discussing about merits and demerits of “Exploratory Testing” (ET) as a testing approach. This manager is a die-hard fan of “scripted testing” and apparently swears by “Quality/Factory” school of Testing.

Me: We should propose ET for this
Manager: Why? What are the benefits?

Me: ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
Manager: Why our test scripts fail to find those bugs? I trust that our specifications are exhaustive and our scripts are thoroughly reviewed and signed off by the business experts. Test scripts should be able to find nearly all bugs that I consider important to be fixed.

Me: Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”. Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts. There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space. So, there is good possibility that scripts will not find all bugs that potentially be discovered.

Manager: What are other benefits of ET?
Me: ET helps the test coverage and provide enhanced bug finding capabilities over scripted testing – especially to nullify “Pesticide Paradox” associated with scripts.

Manager: How? What is this pesticide paradox?
Me: Just as pests in soil over a period of repeated application of a specific pesticide acquire immunity to it and fail to die or show up – software bugs become immune to repeated application of specific Test cases. Over the period of time developers are become aware of test cases that are executed and *specially* test to make sure that new build of the application good just enough to pass those tests. As result of this, there is a “false” sense of stability and quality of the application.

Manager: So... Test cases wear out … why so?
Me: Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment. Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.

Manager: what else …?
Me: Quoting James Bach – “The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper. There's a lot of value in that way of testing. But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.”

Manager: What are other attributes of ET?
Me: Cem Kaner states that exploratory tests provide
- Interactive
- Concurrence of cognition and execution
- Creativity
- Drive towards fast results
- De-emphasize archived testing materials

Manager: I heard another related term “adhoc testing” is this similar to ET?
Me: Yes and no … Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing. Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.

Manager: What is specific about ET vis-à-vis scripted testing?
Me: ET is more of investigative approach where as scripted testing is more of “validation” or “conformance” oriented. In scripted Testing, the tests, the sequences, data, variations etc is pre-defined where as in ET, test design/execution and learning all happen more or less at the same time.

Manager: I heard that ET requires “experience” and “Domain knowledge”. Can an average tester do a good ET?
Me: I am not sure how you define “average tester”, “experience” and “domain knowledge”. I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others. Domain knowledge certainly helps in ET but I do not consider it as mandatory.

Manager: Fair enough… What types of bugs ET can discover?
Me: It depends upon what kind of bugs you want to discover. ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application. ET can be configured to cater to specific investigative missions. You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.

Manager: I notice all along you argued like a “purist” in Testing. I am more of a Business owner; I would need to relate every dollar I spent to the return or the benefit that it gives.

Me: No … I would not call myself as purist, not at least here. I bring in lots of business considerations in my recommendations related to testing. ET provides a way of optimizing testing efforts by time-boxed sessions with charters. Depending upon the nature of the information stakeholders are looking for, ET sessions can be accurately planned.

Manager: Yes …

Me: Let us say you have 5000 scripts for an application and they pass all the time. Would you be worried?
Manager: Ummmm… It depends upon the context. But mostly I would not worry about it. I interpret that as a sign of enhanced maturity of those specific application areas. It is quite possible that there are no bugs that I should worry about - the “scripts passing” is a “confirmation” of that fact.

Me: What if this trend continues and next 5 cycles of testing also do not produce any bugs? Would you be worried then?
Manager: No …not at all In fact, I would reduce the size of scripts being executed to say half – 2500 as the application has become stable. This is an indication for me to "cut down" testing effort, I can possibly look at automation as well.

Me: Here is a twist, what if your ALL scripts are passing but your are seeing bugs (either detected by other means or by the customers) .. Would not you doubt your test cases?
Manager: Depends upon the kinds of bugs I see … If I were doubt something or someone at all … I would doubt test results, testers integrity and project management in general. Test scripts are less likely to be at “falult”. That would process issue - we would need to tighten the process.

Me: OK … What corrective action you would take then? What you steps will you take?
Manager: I would immediately order a thorough Root Cause analysis of the defects and identify what is causing them in the first place. Tighten the development, configuration and deployment process. I would strictly enforce the processes (improved) in Testing and mandate the testers to correctly and meticulously execute the scripts and report the relevant results correctly.

Me: What if you still find bugs outside your scripts?
Manager: That would be a “hypothetical question” – not likely to happen. In any case my focus would be to improve the testing process and strengthen Test scripts. Again, if you are finding still finding bugs – probably those bugs would be “obscure” type – I might not have to bother about them…

Manager: Good… I am still not convinced that ET can give the bang for the buck. As someone who is interested in predictability and repeatability of testing, I am interested in a test process that can scale.
Me: Ummmmm … OK… what is a testing process? Is this something that “actually happens” or “something that is intended”? Is repeatability and predictability - all you care?
Manager: You are too much … there is a limit to asking questions …I don’t think this discussion is leading to any good … Let us talk about it some other time [walks out of the room]

I am continuing my discussion with this manager and post the views and continued discussions in Part 2....

A very happy new year to all …

Shrini

Advantages of "highly repeatable tests" ...

2007-12-14T18:30:00.000+05:30

I was reading Ben simo's post on "what is software testing" - a meticulously created list of quotes about software testing. The beauty of this post is that it traces the history of software testing”. One good way to read this post is to evalauate each statement or the quote mentioned, with respect to it's relevance to software testing.

I stumbled upon this GEM from James Bach.

“Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.”

- James Bach,Test Automation Snake Oil, 1996

So, if you have an excellent set of "highly" repeatable tests, in terms of execution (automatable sequence of actions) and in terms of results (pass of fail) - congratulations, you have successfully managed to find a set of test cases or scenarios where the software is least likely to fail - meaning you will not (or do not expect to) see bugs/problems

But ... wait .. is that your testing mission?

I heard someone yelling from my back… “Yes .... that is what we expect in regression testing. But sometimes occasionally we do find bug as developer made a mistake that was caught or tester [by mistake] deviated from scripted test sequence [a process issue or discipline issue]"

What do you say?

Shrini

Dr Kaner on Software Metrics ...

2007-11-21T14:01:00.000+05:30

Here is another gem from Dr Kaner .. this time around it is about Software Metrics ...

A rare insight into metrics world ....

http://www.artima.com/forums/flat.jsp?forum=106&thread=218013&start=30#287847

This is in responce to a thread by Alberto Savoia

http://www.artima.com/forums/flat.jsp?forum=106&thread=218013&start=0&msRange=15

I keep looking for such insightful replies and posts ... I hope readers are liking it and getting benefited by it ...

Shrini

Further on Testing as a career ..

2007-11-17T14:46:00.000+05:30

Following this post on career in software testing , I found an interesting comment/viewpoint from Jeff Fry's post (more preciously a comment to his post by one "Steve Sandvik") on "why do you enjoy testing".

Jeff Fry's post itself is a very good post that goes in details about "Testing, Career, Enjoyment and few suggestions for tester to stufy read".

Following are Steve Sandvik's comments that worth "consideration"

"...Yes, it may be my first formal job testing software, but as so many people in testing like to point out, nearly any experience or learning has some translation to testing, if you know how to apply it. 15 years of power plant operation and maintenance experience provides an awfully large number of troubleshooting and investigation opportunities.

Identify the fields outside of your industry where, for lack of a better description, good forensic skills and an agile mind (not to be confused with an Agile mind) are at a premium. Industrial equipment field service, process and generation operations, and auditing are a few I can think of off the top of my head. "

And these comments about "in-born" testing qualities

" ...I’m not sure whether truly great testers are born or made, but I think there’s at least a component of most of them that falls firmly into the born camp–in the same way most writers would write whether or not they were paid to do it, I suspect that most people who seriously take up testing as a career for its own sake rather than as a stepping stone to something else approach the world in a certain way even when they’re not formally testing. I know I approach things from what seems to me to be a testing perspective most of the time."

I am filling my blogs with few interesting career related suggestions ... I hope my blog readers are enjoying ...

Shrini

Michael Bolton on "Software Bugs"

2007-11-16T21:35:00.000+05:30

Continuing my earlier post on Dr Cem Kaner's comment, this time is about sharing views about software bugs by another leading light of context driven testing community - Michael Bolton.

Here are his views about "software bugs" (Again, these Michael's views in response to a question on the forum about Bugs - whole his reply stands on its own, I believe)

Personally, this is best advise that I have ever seen with respect to handing bugs by testers and how that decision impacts other stakeholders ... Read on ...

[Michael Bolton : Quote]

When I'm a tester, I'm concerned about trying to drive the project. As a project manager, that was my job. As a tester, my job was--and is--to ferret out information of any kind about the application that helps the project manager to achieve her goals. For me, this has a couple of implications.

First, I don't merely observe the product; I have to observe the things around the product--the platform, the systems with which the product interacts, the business processes, the anticipated or unanticipated users of the product, and so on. I try to be leery of recommendations to fix specific bugs, because in the past I spent too long going the other way--believing that I'm running the project when I'm not. (I'm arguably not a multimillionaire at least in part because in one company where I worked, company project managers had abdicated quality decisions to the testers and developers, which meant that we had a great, largely bug-free product that missed its market window by about a year.)

Second, there is one particular kind of bug that I will try to sell: bugs that make testing harder or slow it down. My goal is to reveal information about the product. Even if we do great testing, there are some things that we won't know about the product. Things that impinge on testing pose the risk of us knowing even less than we would otherwise.

So, with at least one eye firmly fixed on the context and the best judgement I can muster, I will advocate strongly

- to fix immediately bugs that block deeper or broader testing;
- to add testability (logging, scriptable interfaces, configurability, controllability, installability) to the product such that we can increase test coverage;
- to fix immediately trivial-looking bugs that add distraction and noise to the project effort--for example, typos that absolutely everyone will notice and report, such that the reporting and processing of the report will take time away from other coverage.

However, I also remind myself that we testers are vulnerable to representativeness bias--bugs that look trivially simply might be hard to fix, bugs that seem gnarly might be insignificant to the end-user, bugs that look hideously complex might have easy fixes, and so on. So I try to tell the absolute best story that I can about the bug and its worst ramifications, but I also acknowledge that I might not have the whole story about the technical or business reasons to fix or to defer a bug

[Michael Bolton: Unquote]

Shrini

Dr. Cem Kaner on Software Testing as a Career

2007-11-16T21:18:00.000+05:30

I take this opportunity to share few on great discussions happening at "software-testing" Yahoo group, to all my blog readers. For the purpose of focus, I am not sharing the entire thread and the beauty of the reply is such that you can read this without knowing or referring the original post that initiated this discussion....

There were actually two replies by Dr Kaner - I am taking the liberty of rearranging few paragraphs from both replies in order to givea specific flow to the whole thing. The purpose of this post is to share the words of wisdom and experience for all those who would like pursue the career in “Software Testing

[Dr Kaner: Quote]

Let me start by distinguishing between a CAREER and a JOB. A CAREER involves a long-term, intentional focus on a field or type of work. A JOB is a temporary assignment with a particular employer. My career is focused on improving the satisfaction and safety of software users and developers. My current job is as a professor. I have also held jobs as a tester, test manager, programmer, human factors analyst, software development manager, technical publications manager, development director, organization development consultant, salesperson, software development consultant, and attorney focused on the law of software quality. Each of these has addressed different aspects of what has been, to me, the same career. People define their own careers. Many people define their career in terms of traditional categories (programmer, tester, lawyer, teacher), but the choice belongs to the person, not the category.

When you make a choice ("I am an X" or "My career is X"), that choice is both inclusive (Xness is in your path) and exclusive (if Yness is not part of Xness, and Xness is not part of Yness, then "I am X" means also "I am not Y"). When someone defines their career as "tester," I think that definition is too narrow.

I see software development as a bundle of coordinated tasks, including programming, design, testing, usability evaluation, modeling, documentation, development of associated training, project management, etc. Very few people would do all of these as part of the same job. Fewer would do them all on the same project or in the same week. But working at one company as a tester and another company later as a programmer is not inconsistent with calling myself a software developer at either/both companies

I don't generally encourage my students to pursue software testing AS A CAREER. They can make that decision later, after they have more experience. I prefer to encourage them to try SOFTWARE DEVELOPMENT as a career -- to me, development includes testing. And that they take a job doing serious, skilled testing as PART of that career. Most of the best testers I know have significant experience outside of testing and apply that experience to what they do as testers or test managers.

I think that testing is a fine choice for a first job--for some people--but that doesn't make it a first career. It becomes a first career only for the person who says, "This, testing, is my career." I don't recommend that people make a decision to narrow their career that much, early in their career. Let them explore the field more, in their next few jobs, before they lock themselves into something.

I think that some people are good at both programming and testing, some people are good at both writing and testing, some people are good at design and testing, very few people are good at every software development task. So I think it is inappropriate to say that someone shouldn't be considered a software developer because they are good at some aspects of development but not others. Most (all?) of the hidebound process-pushers that I know in the field have never done serious professional work outside of testing. From their narrow perspective, they think they know more about how to manage a development project than the people who retain their testing services. Instead of trying out their ideas as project managers (where they will be accountable if they fail) these process advocates undermine the projects they work on by trying to control things they don't understand with rigid policies and procedures, standards and superstitions, whose costs and impacts are beyond their imagination. We have too many of these people in our field. We need more people who have a broader view of the tremendous value that testing can offer--within its limited role--and are glad to embrace a service-provider role that provides that value.

I think some fresh engineers should start their career with a job in programming, others with testing, others writing, others with human factors assessment, others with configuration management, others with data analysis. I think that choice should depend on what motivates the particular person.

What makes testing worth spending time on--as a job and maybe as a career?

We are professional investigators. Rather than building things, we find ways to answer difficult questions about the quality of the products or services we test. Our job--if we choose to do it well--requires us to constantly learn new things, about the product, its market, its implementation, its risks, its usability, etc. To learn these, we are constantly developing new skills and new cognitive structures in a diversity of fields. It also requires us to communicate well to a diverse group of people. We ALSO get to build things (test tools), but very often, we build to our own designs, which can be more satisfying than building an application that does something we'll never personally do (or want to do). Learning to do good software testing requires learning to do critical thinking well, and to back it up with empirical research. Not everyone will like to do testing. Not every engineer or programmer will have the skills or the interest to do professional-level testing. But for those of us who enjoy critical thinking, experimentation, and keeping the human relevance of what we do always in mind, there is nothing else like it in software development (except, for some people on some projects, requirements analysis backed with rapid prototyping and prototype-based research).

[Dr Kaner: Unquote]

Shrini

Tester's world of Possibilities

2007-11-05T12:05:00.000+05:30

Probable impossibilities are to be preferred to improbable possibilities. -- Aristotle
The future belongs to those who see possibilities before they become obvious. -- John Sculley

Recently, I challenged a fellow tester about testing “Notepad ->File -> Save As” functionality. I asked him to zoom on (focus) only Text files and investigating about file names (say base file name – one without the extension) and come up with testing ideas.

He started with domain testing approach and said that file names that can be supplied to the program could be classified as:

Valid File Names – those where file creation succeeds.
Invalid File names – where file creation fails and no new file gets created.

Then he went on thinking about possible values in terms of these “classes”. I argued with him about his classification – why only think about valid and invalid file names? Can you think about other possibilities …?

Few examples that I gave are –
What if file is created but it can not be opened to view?
What if the file is created but is read only?
What if the file is created but notepad application crashes during the file creation?
What if the file is created but notepad crashes while opening such a file?
What if the file is created but it takes 10 minutes to load the file?
what if the file is created but can not be searched using Windows – Search?
And so on …
My friend said “well all of these can be considered as invalid file names” … To that I said “But as per your initial classification, no file gets created for an invalid name ….!!!!”
My friend continued “ These all are possibilities but not real values …”. I said “That is exactly is my point. As a curious tester, I think about all possibilities and then investigate on those possibilities. My work starts from that point where other wash off their hands saying “We are done”.

What do you think are the possibilities when a file is entered for Notepad -> File->Save As dialog? Keep your investigations on the lines probing the file name parameter…Can you describe the "big picture"?

Shrini

100 Questions or 100/10000 Test cases in 20 minutes …

2007-10-23T22:57:00.000+05:30

That is what I would have named a blog post that my buddy Rapid software tester (context driven tester too) Pradeep Soundararajan wrote few days ago.

If every question is a test case or represents say 10 test cases, can you create 100-1000 test cases in say 20 minutes… Any James Bach student, context driven and or rapid tester will demonstrate that it is possible ….

Without any KA/KT, training, domain knowledge, no pages of documentation …? Plain vanilla testing at its best. Unbelievable right?

One might say --- that is cheating …!!! you may scream… that is not a test case. Where are steps? Where are the detailed information about application? Can a novice, low skilled tester use this? Can this be used for next five years? Can that be automated? And run in night when people are sleeping and give the results in the morning (in plain pass fail – way)?

The answer is “No”

Please note, while we create lots “frills” or decoration around real test and call it as test case – at its core a test is a question that we ask the program and program responds to it with one more answer. Some of these answers we can observe, assess, report while lot others go un-noticed. And while this happens, the environment or platform also responds to the question.

Questioning is a key attribute of a skilled tester. Questioning is also important aspect of learning. Unfortunately, right from our childhood (remember your father and mother saying “this kid is too much – asks too many questions. You will know when you grow up”!!!), through our education and now in the Job, Questioning is not encouraged.

Why?

Questioning is considered as disrespect or contempt
Questioning is indiscipline
Questioning is disturbing
Questioning is embarrassing when answer is not available
Questioning sometime is considered as silly and stupid
Intelligent people do not respond to silly questions
Questioning at times make everyone think – that stalls the progress in some cases
Questioning in a group is considered as bad
Questioner is a labled as “trouble creater”
Question = Trouble, more work, Road block
Oh Gosh, we have not thought about this at all … this is terrible, what do we do now – is not easily coming response for a question

Today’s testers are forced to follow processes, documents, checklists and other “standard” things. If one were to follow the kind of thinking that Pradeep displayed – testing can happen with least information – a lot can happen in small time. I have heard testers who can not start testing (or test design) until they get specification, training and other supporting material. This is a damaging trend for testing profession.

As a famous punch-line of Café Coffee Day (popular chain of coffee joints in India) – “A lot can happen over coffee", goes, can I say “A lot of testing can happen in 20 minutes for any application”.

Just give me the *stuff* to test … I will flood you queries that can potentially lead (if investigated and answered) to an arsenal of information about application under test …

Shrini

Types of Equivalence: Equivalence Class Partitioning - II

2007-10-07T10:05:00.000+05:30

Following this post of mine, I have been studying deep into understanding of this technique. Here are few more thoughts related to “equivalence”.

Here is ECP in nutshell - “Group a set of tests or data supplied for an application. Assert that all the tests/data belonging to group will teach you “same thing” (application behavior). Hence it is “sufficient” to use only one value/test from the group”.

Fundamental to ECP is the concept of “Equivalence”. Most of the authors or proponents of this technique give examples of date, integer fields and demonstrate identification of classes and equivalence. For example if you consider a date field in “NextDate” program, using “generally accepted rules” governing the usage of dates in Gregorian calendar – you can identify some classes – All the dates in the month of January can be considered as equivalent (except first and last day of January and first and last month of the century – which are boundaries). These “canned” classes appear to be applicable for every application that has date field in a “next date” function. Another example would be a field of integers (1-100) – most authors have mentioned the example of 2-99 as one equivalence class meaning all numbers in the range 2-99 would be treated “alike”.

I would call such equivalence that can be arrived without knowing anything about application, it logic and programmatic implementation details as “Universal equivalence”. It is easier to explain the concept of ECP using “universal equivalence” – date and integer fields are the most popular examples. But I see a danger here – the way ECP is explained using “universal equivalence” – it leaves out lots of key details such as basis for equivalence.

What are other forms of Equivalence?

Functional Logic equivalence - Consider the “Age (1-150)” field. Application logic might enforce that Age range 1-16 considered as one eq class (Kids) and others like 17-45 (Adults) and 60-99 (Senior Citizen). This kind of equivalence is very straight forward, easy to derive. Often specifications help us to arrive at such equivalence classes. Here is where the classic examples of “valid” and “invalid” EQ classes seems to have been originated.

If one were to go by pure functional logic equivalence, it would be sufficient to model Age parameter having three eq. classes and hence one value taken each from these classes (3 in all) would provide “complete” test coverage from an ECP perspective.
Dr Cem Kaner calls this as “specified equivalence”.

Implementation Equivalence – This is where one deep dives into how data is processed (validated, accepted, rejected), passed around (within application components) and eventually stored or discarded after use. Here we would talk about programming language (data types), software platform (OS and other related programs) and the hardware platform.

Dr Kaner identifies another two sets of equivalence - “Risk based” and “Subjective”. If the equivalence is in the eyes of tester (“these two tests appear to teach me same thing”), this form of equivalence is called “Subjective” equivalence. If a notion of equivalence is established targeting a specific class of risks or errors, it is referred as “risk based” equivalence.

Thus one way to apply ECP effectively is to start with universal equivalence and go on refining the sets EQ classes as we go deep into the application and platform (add, modify, delete the classes and their definitions). Implementation Equivalence seems to be the lowest or the last in the chain overrides the specifications of classes as determined by higher levels of equivalence (universal or functional logic type)

One question to spice up the discussion – Is ECP a black box technique?
Yes if we restrict to “universal and functional logic” equivalence.
No if we deep dive into code of the application and look around at platform (software and hardware)

What do you think?

[ Update ]

ECP attempts "simplify" a big picture (data domain with infinite set of possible values"). When attempting to apply ECP for a data variable, best starting point would be "what is that big picture I am trying simplify using ECP"? This is a top-down approach - model, understand, analyse, hypothesize the big picture then go to next level and then think about EQ. classes. I have people mostly approaching this from "bottom up" approach - think about valid and invalid classes first (or even actual values) then if possible think about the big picture.

Which approach you think is a useful one to start with?

BTW, there is "Equivalance principle" by Einstein related to theory of relativity. Can I say equivalence as applicable software tests is "relative" in nature?

Shrini

More on definition of Test Automation ...

2007-09-30T09:25:00.000+05:30

While discussing about "definition" of "test automation", with Ben Simo I left a comment at Ben's blog post. I am sharing that here for the benefit of my blog readers ...

(You would also find some debate about "model based testing" in comments)

Test automation involves - Automation Design and Execution of automated tests ...

Design: An act of translation of some question that one asks regarding a feature of an software application into some programmable instructions so that the question being asked is modeled with reasonable accuracy (courtesy: Michael Bolton) In simple words translating a testing question into a program

Execution: Use of a machine (hence a set of computer programs) to support any aspect of Testing with Testing at the center of whole scheme of things.
(derived from the definitions by James Bach and Michael B)

As I continue to write about definitions and terminologies around test automation, let me reiterate - Nothing called "Automated Testing" exists in this world....

Shrini

Shrini

Simple things and me ...

2007-09-29T00:27:00.000+05:30

World likes simplicity – Pictures, Colors, Sound, Odor are used to abstract some complex behaviors.

Consider following –
Software Project Status – Red, Green and Yellow
Project Size – Small, Big, Medium
Test cases – Simple, Medium, complex
Testing – Gather requirements, write test cases (some times even prepare them), execute them, check if the pass, if pass report and go home and if fail log a bug and go home

Automation – Create an automated script using the state of the art Automation tool. Store the script in test management tool and execute it from it. See the result logged automatically. If script fails (I mean reports failure), automatically make another script to log the bug

Knowledge Transfer – As simple as Fund transfer in bank. Transfer the relevant documents to the subject – transfer. If there are clarifications there always our best friend “issue list”, “action item tracker”, “clarification list” – many names but all is one.

Knowledge acquisition – As simple as one country invades and acquires another country. Just acquire – by force

What is common between each one of these – simplicity.

I become restless when

- People tell me about testing project status in terms of Red, green or yellow.
- People start approaching testing as sequence of actions – test design, execution, bug logging and regression
- People start describing a bunch of test cases or software features as Simple, medium and complex – SMC model.
- Testers getting classified as great, lousy based on the number of test cases executed or number (not the type) of bugs logged.

To me, all of these are simplifications of some complex things that we are trying to understand. My frustration is “Why things can be so simple”? “There must be something that is hidden behind this simple thing”

Imagine, if Newton were to think – when apple fell on his head – “No big deal – it had to fall so it fell down – Gosh – my head is aching. Why did sit bellow this tree?” Same thing applies to Archimedes. Jump out bath tub and run …?

While thinking about simplicity about some things that we deal with helps us to get started off – important to understand that it was just a beginning.

While testers and most of the managers are happy about such simplifications - a skilled tester is always worried about such simplifications and often attempts to find the loop wholes in simplified models, notions, beliefs, description etc…

Nature’s simplest things have hidden inside then greatest and deepest mysteries. That would a fascination journey – we just need vehicle capable moving into it – Our imagination and curiosity. Today’s skilled testers are blessed with this imagination and curiosity. The journey has just begun …

Do you think post has horribly and outrageously simplified the seriousness and intent behind the post? Well, I can not be too serious …

Automation Dreams - Thinking about END in the begining ...

2007-09-27T20:08:00.000+05:30

"Cherish your visions and your dreams as they are the children of your soul; the blue prints of your ultimate accomplishments."

- Napolean Hill

Thinking about “end” in the mind helps me most of the cases such as planning, checking and carrying out various day –today tasks like performing 4-5 interconnected tasks, leaving home for a week/month long journey, arranging my daughter’s birthday etc. Thinking about “END” while you are about to begin or at the time of beginning a task - can help you to visualize the entire stretch of journey from start to end, anticipate various milestones, problems and as many as details as possible. This can prove to be a powerful mind modeling concept if you train yourself in visualizing from the END and work backward from there to the beginning.

Let us apply this to Test Automation …..
- How do you describe a successful automation?
- How can one describe steady state automation in deployment?
- Can you trace a journey from the beginning of an automation initiative and the end where your automation has become “obsolete” or has reached a steady state?
- Is there any END for Test Automation as a software project?

Automation initiatives have end goal(s) to achieve – cut testing cost by certain percentage, increase test coverage by certain percentage etc. In a way every automation project has some dreams to realize. Can we describe those dreams related to automation?

Let me give it a try ….

I will have an automation suite that runs “unattended” for up to 8 hours. (Length of Execution)
My automation suites covers 40% of my regression tests for product X. (Automation coverage)
I will have an automation suite whose results I can trust the most or When my Automated Test fails – I am sure that is bug (Reliability and Trust)
I have 40% of my test cases automated. Hence my testing effort now on will be 40% less (You can dream and these are DREAMS …. )
My smoke test suite is automated – I can now make changes to the code more frequently even when there is time crunch.
And so on ..

What are you Automation Dreams? Can you describe them?

Physicians, Surgeons, X Ray Lab Technicians ....

2007-09-13T14:16:00.000+05:30

Heard of this rant?
“We would like our business users, domain experts and subject matter experts write automated scripts (tests?)”
“Our next generation automation tool – allows persons with ZERO programming and testing knowledge generate automation scripts in MINUTES”

Think about it …. Will you ask a Physician (Medical Doctors) or Surgeon to work as XRay or Ultra Sound Lab Technician and vice versa? Or An Aneathesian to perform a surgical operation?
Each one in professional life are known to have some core competencies. Best bet will be to utilize each one to their strengths….

Business users know business domain while Automation engineers and Tester know programming and software testing.

IMHO, asking business users to do “main stream” testing and automation is sure recipe for “failure”.

Any views? Do you think the analogy presented here is logical?

More as I hear from you ….

Shrini

Inattentional Blindness ...

2007-09-11T13:30:00.000+05:30

Sajjad has an interesting article on "inattentional blindness" and Exploratory Testing here

Sajjd mentions that - "Exploratory testing is supposed to be better at minimizing inattentional blindness". This kicks off a thread of thinking in my mind - Why it might be so? what elements in ET helps one to minimize inattentional blindness?"

Let me take a guess and answer --
May be it is "thinking between alternating polarities" while doing ET - doing vs explaining, fast vs slow, reading vs doing, focussing vs defocussing. In my opinion inattentional blindness happens due to "heavy" focus on one or more "atomic" aspects of bigger object under observation. one possible solution is to defocus as often as you can.

A related phrase (a kind of antonym) I use is "your eyes will see what you would like to see". This is more dominant theme in scripted testing where a tester is "pre-programmed" observe only expected results.

Here is a wikipedia page http://en.wikipedia.org/wiki/Inattentional_blindness

Do not forget to read James Bach's comments for this post -

"Testing is a lot like fishing. You try to create conditions that maximize your chances for catching something tasty. Exploratory testing is a little more like fishing, because just as with real life fishermen ......"

Shrini