I am a sapient Tester …

Here is my first and a very crude attempt write a poem (if it can be called as a poem) on sapient tester ...

I understand that testing is a sapient process *
(* A sapient process is any process that relies on skilled humans)
I question things around me
I model things to understand their behavior
I learn from related fields in testing (ones that are close require higher levels of human cognition)
I use my brain and I constantly try to improve my thinking
I don’t believe in rote processes that mechanize human thinking
I don’t use documentation, KA/KT* as essentials for getting started with testing
I can figure out my way in understanding the software I am testing.
I use automation as aid in manual testing not as a replacement
I use techniques as heuristics
I just don’t simply follow the written instructions and procedures - I question them
I collaborate with developers, other testers, users, support personnel and other stakeholders to understand wide range of perspectives.
I understand depth and value of human testing
I understand that by automating a test, I might be loosing some thing.
I am sapient tester... Are you?

Association of software testing (AST) publishes a magazine called "sapient testing"

James Bach wrote about sapient processes here

* KA - knowledge Acquisition and KT - knowledge Transfer

Shrini

Thinking about Zero defect software - Attaining software Nirvana ...

“Humans have always proved impossible is possible … Dreaming, Desire and passion is key to achieve “impossible”. Wright Brothers dreamt about “flying” – they achieved it. Once, reaching moon was impossible – now? You need to give your life to it …Zero defect software is possible …” You need to aim high”.

This is how, a colleague of mine argued with me when we were discussing about Zero defect software. He believed that many software like IBM mainframe apps are running with near zero defects … (there could be defects but of cosmetic nature)

The notion of defects, number…

No defect is bigger or smaller … Let us say there is a 0.5 second delay in system response … Will this matter? Is this a bug … depends upon which software we are talking about? Context is important …

When people talk about software processes, discipline and making an attempt to achieve zero defect … they are often of the opinion that …
Human beings make mistakes deliberately … many or all can be avoided if there is a second eye or a watch dog
Human do sloppy work unless controlled – given a choice no one would do a good job (if no one is observing)
Humans require constrained and regulated environment
Humans - overlook

It is this possibility that people require discipline makes others (mangers especially) to think that zero defect is possible. People who vociferously argue in favor of zero defect software – think so because they feel that it is because of human laziness and other related aspects – defects are introduced. Put a system of governance, policing – you will achieve zero defect. How true is this?

Chasing the definitions of Quality ....

Why do we have so many meanings/interpretations/definitions of quality - A colleague of mine asked recently. I did not have any ready answer for him ... I said ... after a pause ... "There are so many ways to see the quality". Because, I think every one, whoever attempted to define quality (from Juran to Weinberg, ISO to CMMI), did so wearing a "user's" hat while they were not real users. It is like doing a role play temporarily - a third party view.

Michael Bolton mentioned this elegantly...

Quality is not an attribute of a thing or a thing (most of us believe that is in a thing). It is a relationship between the user and the product.

So extending Michael’s view point - quality is a two way and intimate relationship between the user and product. There is no room for third party.

So as a third party everyone can come up with "possible" meanings of the term quality.

The moment we understand that it is relationship on a timescale (perception of quality does change over time) not something built into the thing ... we will be good. Beware a notion of good quality does not alway remain so .... as time changes so does the perception about quality in the minds of the beholder.

What does this mean to we, testers?

- As testers we do a third party assessment of (to best of our abilities) the relation ship between a product and it's anticipated user base.
- Things like fast, great UI, cheap, reliable are only manifestations of relationship between user and product - not real things or attributes about quality.

- Quality is observer dependent - quality can be viewed always and only viewed from the eyes of the user's frame of reference
- User's taste, perception of quality for a product changes. Hence there can not be any "one-time", "fits all size" kind of testing/assessment for a product (more so .. if it a software product)

Who is next in the line with a definition of quality?

Shrini

When method dictates outcome and Goal – See Goal displacement in action - I

I first heard (came to know) about this word from James Bach and this immediately caught my attention. I started seeing it everywhere. The phrase "goal displacement" is generally referred to a situation where method of achieving something dictates the out come and goal. The process by which the means used to achieve a goal becomes more important than the goal itself.

Here is an example that I think illustrates what I mean by “Goal Displacement”.

A model of Test Automation

Manager: I am quite impressed with overall progress this automation project. But I would like see some metrics and measurements to quantitatively monitor future projects in this area – like number of scripts produced per day, individual automation engineer’s productivity etc
Me: Ummm … right now, our model of automation development does not itself to such measurements. We don’t treat overall work in terms of scripts and measure engineer’s productivity. I am afraid; I don’t have an easy answer to your question

Manager: I understand your concerns. But I need those metrics.
Me: Here is a deal, how about modifying the whole process of automation such that it is easier to measure and exactly built according to metrics that we would like to measure (regardless of what outcome we would like produce)?

Manager: I did not understand that … Can you not just introduce them to existing model without affecting anything else - especially the output?
Me: To accommodate the kinds of things that you are asking, I will have to modify the way we develop things – which I am afraid will affect the quality of our deliverables. I don’t see any way to make those measurements in current model.

Manager: Metrics and measurements are not suppose to change the process instead I heard that metrics help to control and improve the process.
Me: In this case they do. Do you want me go ahead and change? I see a serious, negative side effect of changing the process/model to suite metrics.

Manager: Ummmm … Let us talk about it some other day (walked away)

What is happening here? Manager wanted to see some metrics to see how the automation team is performing. I simply said "You can have those metrics but at the cost of changing a process that is apparently working fine and producing the results that make you happy". So here, the introduction of metrics (or a measurement process) causes a shift in the goal of producing good automation code to goal of having a process that measurable in some way. It is like saying if current process (way of doing things) does not cater to a new related expectation, possibly a conflicting one with original goal (a displaced goal) - then forget about goal - do whatever change required to meet new goal or expectation.

Stay tuned for two more examples of goal displacement ...
Shrini

Software Testing certification - To be or not to be ..

I am yet open up my thoughts on certification ... also I have not been very critical of certification programs. Here is an attempt to publish my personal views on this topic ..

In 2003 I took CSTE exam ( A link from QAI is here) - as asked my manager and cleared with flying colors - 90% above ...My preparation for the exam was painful ... I had scratched the CBOK (reference material) all over as I did not agree with most of material - which seemed to copy paste from a software engg text book and about 20 years old stuff. I could memorize lots of stuff without sufficiently challenging it. I passed the exam ... did I learn anything new that helped my work --- NONE other than some terms and their meanings (questionable).

I also constantly watched the kinds of people who passed such exams .... I must the quality was bad. Passing was easy ....In my opinion, most of certifications (I am a certified Java programmer too) suffer from this problem. There is so much focus on memorizing things not about learning.... There is little space for debates and questioning... That is you questions do not have "comments" field .. or "I don't agree with premise of the question because...." kind of open ended response from the candidate .. Who has the time to site and evaluate all responses ...?

Discussions on certifications often take "Emotional" angle - people say exams like BE and certifications help one to improve skill and knowledge .. I would say they only provide reading material .. rest is in your hand (rather in mind) . I strongly feel that skill comes first and gets developed by practicing, doing, spending real good time with dedication not by passing an exam that tests your memory retention capabilities.

The problem I see with testing certifications is that - they don't test testers skill to do testing ... (a practical exam like our science lab in schools might help) instead they test tester's memory retention skills.

Are there any certification exams that watch tester testing stuff (with some video etc) and then give "good to go" Tag? No .. that would be expensive and tough exam to administer right ...

An example of goal displacement ... certification exams appear to be designed for easier and large scale (yes/No or choice) kind of assessment than complex/tougher assessment of watching tester in action and rating his/her level of skill.

Why goal displacement -- Some group of people in testing thought - we need an exam to test our testers .. so what kind of exam should we have? ... something that is easier to administer or something that tests tester in action... They prefer to choose the first one ... So the goal of having a good certification got shifted to having a certification exam that is easier to evaluate ...why ? Good certification exam requires complex models and involve lengthy/costly administration. It costs lots of money...

Having said all this ... its your call .. I am neither in favor or against to certifications in testing (some people 's daily bread comes by running those exams -- let us not take away that ...). It is a big business in testing industry today. Being testers, let us evaluate what is good for each one of us (considering each ones technical and educational background) and take the decision.

Any alternative to certification ....?
Focus on skill ... how many of us can go to an interview and challenge the interviewer "Give me a software to test ... give me an hour ... come back I will have a defect/issue list ready". How many of us can take this challenge of "testing anything, anywhere, under any time frame" ? Managers in IT companies need to see evidence that this person can test ... he has tester attitude...

In absence of such evidence, managers resort to something that easy to check (do you have this certification?). If you are fresher or beginner in software testing field ... and struggling to get a break or good job ... You have two options ... Get certified and get a job .. (there after it is your skill that keeps you on job not the certification) other option ... practice testing, read stuff, read books, learn programming language, debate, sharpen your analytical skills, write blog, discuss with others in the field .. build a credible profile for your self (Google should be able to find you) and go confidently to any interview and say "give me stuff to test ..."

First one is easy path ... some money and about few weeks of reading, take the exam your are done... second one is tough one.. Requires you to commit to the profession ... requires your learn things, may take years to be good at testing, many years to build a credible profile that speaks for itself ...

People choose first one mostly as it is easy ... So I would not say all who support certification are not skilled people and take short cut to success neither all those who oppose certification are good skilled testers …

You want to be certified? it is your call -- What ever you do, be a tester by heart - test everything/question everything until you are convinced

Read James Bach on Certification here

Michael Bolton on why I am not certified

Ben Simo on his favorite certification

Balancing on the other side -- AST certification debate

Shrini

18 Myths Associated with Exploratory Testing ...

Myths are public dreams; dreams are private myths – Joseph Campbell
Science must begin with myths, and with the criticism of myths - Karl Popper

Instead of writing on what is exploratory testing, to me, it looks easier to start of explaining what is not exploratory testing. For those who are hearing this word for the first time – let me remind you this is a testing approach that involves simultaneous test design and execution with an emphasize on learning. The term exploratory testing is coined by Dr Cem Kaner around 1983.

Exploratory testing as approach has a legacy. Its predecessor “Ad-hoc” testing is a perceived as spoilt kid. Until recently, I was under the impression that adhoc testing is quick and monkey type of testing (often connected with sloppy testing) where you just play with application in order to find some bugs that are out of specification. I was proved wrong during a rapid software testing workshop that I attended at Toronto. James Bach mentioned in that class that the word “Adhoc” means “to the purpose”. Most people believe (as I did initially) that Adhoc testing is undirected, random clicking of mouse and navigation. This is probably the reason, why Dr Kaner decided to name the baby differently so that the “stigma” associated with the approach of testing goes off.

I think Dr Kaner did not stop at just naming the baby, he along with James Bach, Michael Bolton, Jonathan Kohl, Jonathan Bach and others did lots of research and practice related to “exploratory testing”. Today, if many of us talk so confidently about this – it is due to the pioneering work done by these people.

Let me also remind that exploratory testing is not a testing technique but an approach and opposite of exploratory testing is “scripted testing”.

There so much of confusion, skepticism and disbelief associated with this approach to testing. Based on my interactions with people from all walks of testing world, I have drawn a partial list of myths that are associated with exploratory testing.

Let us go straight into the list of myths …

Why bother about ET?
1. ET -- every one does ET while doing testing - why bother? why have a special name?
2. ET is some kind of snake oil - people use this term to indicate some mystic thing to make money
3. ET will not work in my context - I am not even willing to give it a try as I am not convinced we should give a try. I know it for sure.
4. ET does not seem to have come under the radar of Gartner or Forrester's -- it might not be that popular.

About the form of ET
5. ET - is an unstructured, ad-hoc (meaning sloppy) process.
6. ET is nothing but doing testing without any formal test cases.
7. ET is instant bug hunting process.


About ways of doing ET
8. ET should always be done after completion of all planned scripted testing (if time permits) If you have time (while you wait for a new build), do ET to use your time productively.
9. ET is not predictable and repeatable (was it meant to be repeatable?)
10. ET shows scant respect for well established Test techniques - how can you do testing without using any of those time tested techniques

About skills required for ET
11. ET requires in depth domain expertise hence only domain experts can do it - it is not every ones cup of tea
12. ET seems to require special skills (domain, quick learning etc) - hence we do not do it.
13. ET is highly person dependent, unacceptable to us - we are resource starved industry.

About suitability of ET in a context
14. Our quality process standards require that every test effort be substantiated by a detailed report of what test cases are executed, on what platform, what data is used, how many test cases passed, how many failed etc. ET is not so good in producing such report. ET does not provide enough evidence/proof of having executed testing.

About perceived Value of ET
15. ET is not process oriented and is not methodical (requires highly skilled disciplined, responsible testers). Anything that is so “person/Skill” oriented is strict no-no in our environment (process driven)
16. ET can not be outsourced - even if it is we can not assess the progress - it is an uncontrollable process.
17. ET can not be automated - I am looking to reduce spend on testing - ET can not help me there.
18. ET seems to be useful in only in those environments where there are no requirements but ours is a very structured process. Why to use ET?

Why these are myths and not the truths – that is an exercise to the readers. I welcome each one you, the reader to challenge my claim that these are myths and lets debate …

Shrini

Here comes my 100th post ...

4 Years, 100 Posts, 200+ Comments, 15+ Commentators, about 5000+ page views (statistics since Dec 2007)

It all started in July 2004, on 23rd I posted my first blog post. Why did I start blogging? I just wanted to join the bandwagon of my colleagues and make presence in blog-o-sphere. . I could not blog much in my first year. I struggled to decide what to write and often felt choked after writing few paragraphs or sentences. Mostly my initial posts referred another post or some news which I announced or made reference in my blog. I did not have any significant stuff to offer to the readers. I think this is what people call as “writers block”. I suffered with that in my initial blogging days.

Things started to change as I wrote more, attracted few best minds in our industry to my blog. As I started receiving more comments, I, all of a sudden, started to see lot more ideas to write and debate on. Many times I asked myself, why do I blog? Got only one solid voice from inside … “I wish to communicate …I have something to tell to testing world”. So, all these months and years, the notion of “idea or urge to communicate” drove my blogging. I felt very happy and satisfied after a posted a blog post. It was like winning a game, or scoring good marks in an exam. A good (in my own terms) blog post always energized me.

After posting, I would monitor my blog for comments. Depending upon the content of the post, I expect 2-3 comments in 24 hours of the post going live. Sometimes I loose sleep if I don’t comments on a post (having a content that I believe is thought provoking and interesting). I then, shamelessly send mails to my frequent readers (l have a list) asking them (explicitly) to comment on the specific post. As far as possible I avoid doing this as I feel this is similar to a well dressed girl going round and asking “how do I look” - a desperate attempt to grab attention. Usually best responses come without asking for it.

In my opinion, a success of blog can be judged in terms of the number and quality of comments. Number – because it is an indicative of number of readers of the blog who not only spent time with the blog also went further to write a comment. Most of the blog readers today use blog as one way communication tool – it is not. Some well moderated blogs like the ones from James Bach, are effective 2 way communication mechanism.

Many times, I think of getting on with “Google Adsense” …in current shape and size, my blog even may qualify for it. I am resisting … I personally do not like the blogs with “google adsense” as it distracts from reading. I see a blog as a library of interesting topics often focused on a specific area. Anything else is distraction for me. I don’t want to loose that appeal from my blog by opening it up to Google.


As a blogger I must avoid doing

without leaving any clue, stop blogging for months and then come back and say “ sorry It has been long time since my last post” … Readers tend to develop some inertia when then see a lull … So keep blogging (one in a month at least .. this time can vary from individual to individual).

Discourage or block comments or make the process of commenting too tedious. To me comments are very important as they are only available mechanism of getting feedback from my blog readers.

As blog reader of other blogs, I would not like if some one does not respond to my comment (especially where I challenge the post content). So if a reader of my blogs posts a comment asking me to clarify – that would be my first priority.

Post reference to other blogs without having your own thoughts of ideas

Short posts like “I am back or here is an interesting thing” … No short posts unless there is real need.

These are my own rules of blogging – I plan to follow them religiously and make sure my readers get best experience of reading my blog.

Benefits of blogging (at least to me)
This blog helped me in connecting many unknown faces. I have few students asking for some help on a topic on testing, I got someone asking me to do a software product review for me. I got several job offers from the readers (that were looking for hiring me), I made new friends via my blog. I have great names in testing World like James Bach, Cem Kaner, Jerry Weinberg – commenting on the posts of my blog.

My message to my blog readers –

Keep checking my blog and generously comment on the posts. You can also use my blog as a platform for debate on something other commentator said about a post. For example you might want to react to a comment posted by Pradeep on my post. I welcome that. This will make my blog as multi fold communication – a discussion forum. As far as possible avoid giving very general comments like “Great post or I liked this blog etc …” As I welcome the appreciation (at times I need it) but would like those items to be in 1:1 mail to me (shrinik@gmail.com) – for a reader, comments like “ I like this post or this is a great blog” not very useful. I encourage my readers to challenge me more often – critically analyze the views presented here … write about it and leave a link. I think such challenging comments are quite useful to me and other readers as it presents a different view point. Do write to me about the list of topics that you would like me to blog.

There is one contradiction that I am not able to solve. If I am blogging for myself (for my satisfaction) should I bother about my readers – their choices, likes/dislikes? Should I worry about responding to the comments?

Should you bother about me blogging or not …. What do you say – Dear Reader?

Software Requirement Heuristic ...All requirements are Vague

Here is a heuristic - "All formal/written requirements are vague – only exception is when requirements are collected using a process that does not involve any human”

(Wait … if at all such process exists, where it came from? There must have been a human being such a process needs to be designed by a human … …hence there is a human element here too … Is above heuristic, an axiom or hypothesis or law?)

So there will be always gaps between "Customers view", "end user view" and programmers view". ET can help you to explore these gaps.

Jerry Weinberg in his book on Exploring requirements.. - gives the example of a nursery rhyme "Mary had a little lamb" and goes on to prove that there can be at least more than a dozen ways of interpreting above sentence ... Remember we typically deal with requirements documents of 50-60 pages ... Imagine the amount of misinterpretation possibilities....

I was discussing with this with Michael Bolton, recently. I said “I can say for sure that as long as requirements are written in English, they will be ambiguous. English is a funny language". He immediately shot back and said "you can say same thing with Hind, Tamil, French or any other human language"... I laughed and said “well, you are right, how about this ... One way to get unambiguous requirements is to express them formal languages like computer programming languages ... say Java. "Class MyClass" in java is nearly unambiguous". Michael continued "Here, the ambiguities come, not from written requirement but from the process of translating what is there in one’s mind to formal language".

That brought us to the real truth about software requirements ... "As long as one human tries convert what is there in her mind to some thing that written form; mind to document or mind to mind data transfer .... There will be ambiguities"

People can say a thing in different words but might mean one thing or they might say one thing and might mean differently....

Welcome to the world of software ... a discipline with high human content ....hence we need skilled software testers to keep world "lighted"

First thing you need to acknowledge is Requirements are fallible - they can be wrong - horribly incorrect (so will be corresponding program). If you base your entire testing only on requirement - you can go wrong by miles ... beware of this requirement based testing trap...

BTW there is a type of testing called requirement based testing ... can you imagine how narrow it can be?

You might have heard of people counting requirements … Can we count requirements as we count apples and pizzas?

What do you say about this concept of Requirements Traceability Matrix (RTM) – a holy thing that most swear by? Fundamental principle on which the concept is that “requirements and test cases can be counted”

Please… for God’s sake understand requirements, bugs, test cases are not real things and can not be counted in a way you count physical things, they are mere concepts, ideas …

Reductionism and Test Techniques - who, what?

Scientific reductionism is an undeniably powerful tool, but it can mislead us too, especially when applied to something as complex as, on the one side, a food, and on the other, a human eater. It encourages us to take a mechanistic view of that transaction: put in this nutrient; get out that physiological result.
- Michael Pollan

I have heard statements like the ones below several times … I am sure many of you have.

- Orthogonal Array or pair wise technique reduces the number of test cases and hence can optimize the test effort

- Use of orthogonal array based testing (Test design technique) has demonstrated to produce superior test plans that improve testing productivity by a factor of 2.

- Equivalence class partitioning (ECP) is a functional testing technique that systematically reduces the number of tests from all the possible data inputs and/or outputs, and it provides a high degree of confidence that any other data in one particular subset will repeat the same result.

- Boundary value technique catches the bugs occurring near boundaries. Use of boundary value technique increases the effectiveness of test coverage.


What do you notice common in all these statements? Faith and reductionism. When you say “Technique XXX does this or help in this [reduce the number of test cases], you are forgetting that it is not the technique but it is YOU as someone who is making an assumption/assertion.

For example, when you say “Orthogonal array technique reduces the number test cases” – you actually saying, I as a tester using this technique making assumption that only pair wise test cases generated by this technique matter – rest I ignore.
Or
When some one says “ECP technique systematically reduces the number of test cases” – that is a view of reductionism. It is tester's hypothesis, judgment and assertion that certain sets of data values CAN be treated equivalently. ECP does not "reduce" ANYTHING by itself systematically or otherwise. ECP's principle per se is that there are groups of data that can be modeled to be treated identically by the AUT

I can say that, this is a real difference. This shifts the focus from the technique to the person who is using it. You then can not shift the blame to the technique if it fails.

As testers we use lots of tools, techniques and heuristics to understand, operate and observe the test objects. Every time I use a technique or a heuristic, It is my explicit choice and I make a set of assumptions and assertions. The goodness and applicability of results of the test or technique, directly depends upon these assumptions and assertions. Technique/tool has no role in it.

Pradeep Soundararajan and Michael Bolton have it here – A focus on tester not on test or a tool or technique…

http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html

If you take a gun and kill a person – what do you say? Gun killed the person?

Shrini

Percipient Subject and Software Quality

Jerry Weinberg in his famous book “An Introduction to General Systems Thinking” mentions quoting Albert Einstein…

“Belief in external world independent of percipient subject is foundation of all science”

Let me extend this to Quality (or Software Quality) …

“Belief in external world independent of percipient subject is essential for the notion of Quality in software world”

All definitions of Quality (except that of Jerry) from Juran, Deming and others somehow assumed the existence of “external world independent of percipient subject”. Hence Quality has been more or less defined in absolute terms.

Let me remind you -
- Quality is value to some person (one who matters)
- Quality is not an attribute of a *thing” – we can not measure Quality but can only assess it in “relative terms”
- Quality is in beholder’s perceptions.
- Quality is not an intrinsic attribute of any “thing”
(Quotes from Jerry, Michael Bolton and others…)

Bonus Tip:

“Belief in external world independent of percipient subject is essential for the success of a metrics program”

Most of Metrics programs that are claimed to be successful stand on this belief.

Howzzzzzzzat?

Shrini

7 Habits of successful Testers

Here comes the last post for the year 2007 - to be expanded later ...

• Self Driven or high levels of Inner drive for learning new things – No fear of unknown.
• Spontaneous – Thinks on the feet – Good in emergency response.
• Agile and adaptable.
• Love for Science (Physics/Chemistry), Mathematics and Philosophy.
• Love for problem, Puzzles.
• Hunger for self Expression – Writing, speaking.
• Organized Skepticism and constantly challenge their own thoughts

Shrini

Exploratory Testing challenged - Part I

Here is a conversation that I had recently with one of my manager. We were discussing about merits and demerits of “Exploratory Testing” (ET) as a testing approach. This manager is a die-hard fan of “scripted testing” and apparently swears by “Quality/Factory” school of Testing.

Me: We should propose ET for this
Manager: Why? What are the benefits?

Me: ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
Manager: Why our test scripts fail to find those bugs? I trust that our specifications are exhaustive and our scripts are thoroughly reviewed and signed off by the business experts. Test scripts should be able to find nearly all bugs that I consider important to be fixed.

Me: Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”. Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts. There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space. So, there is good possibility that scripts will not find all bugs that potentially be discovered.

Manager: What are other benefits of ET?
Me: ET helps the test coverage and provide enhanced bug finding capabilities over scripted testing – especially to nullify “Pesticide Paradox” associated with scripts.

Manager: How? What is this pesticide paradox?
Me: Just as pests in soil over a period of repeated application of a specific pesticide acquire immunity to it and fail to die or show up – software bugs become immune to repeated application of specific Test cases. Over the period of time developers are become aware of test cases that are executed and *specially* test to make sure that new build of the application good just enough to pass those tests. As result of this, there is a “false” sense of stability and quality of the application.

Manager: So... Test cases wear out … why so?
Me: Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment. Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.

Manager: what else …?
Me: Quoting James Bach – “The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper. There's a lot of value in that way of testing. But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.”

Manager: What are other attributes of ET?
Me: Cem Kaner states that exploratory tests provide
- Interactive
- Concurrence of cognition and execution
- Creativity
- Drive towards fast results
- De-emphasize archived testing materials

Manager: I heard another related term “adhoc testing” is this similar to ET?
Me: Yes and no … Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing. Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.

Manager: What is specific about ET vis-à-vis scripted testing?
Me: ET is more of investigative approach where as scripted testing is more of “validation” or “conformance” oriented. In scripted Testing, the tests, the sequences, data, variations etc is pre-defined where as in ET, test design/execution and learning all happen more or less at the same time.

Manager: I heard that ET requires “experience” and “Domain knowledge”. Can an average tester do a good ET?
Me: I am not sure how you define “average tester”, “experience” and “domain knowledge”. I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others. Domain knowledge certainly helps in ET but I do not consider it as mandatory.

Manager: Fair enough… What types of bugs ET can discover?
Me: It depends upon what kind of bugs you want to discover. ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application. ET can be configured to cater to specific investigative missions. You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.

Manager: I notice all along you argued like a “purist” in Testing. I am more of a Business owner; I would need to relate every dollar I spent to the return or the benefit that it gives.

Me: No … I would not call myself as purist, not at least here. I bring in lots of business considerations in my recommendations related to testing. ET provides a way of optimizing testing efforts by time-boxed sessions with charters. Depending upon the nature of the information stakeholders are looking for, ET sessions can be accurately planned.

Manager: Yes …

Me: Let us say you have 5000 scripts for an application and they pass all the time. Would you be worried?
Manager: Ummmm… It depends upon the context. But mostly I would not worry about it. I interpret that as a sign of enhanced maturity of those specific application areas. It is quite possible that there are no bugs that I should worry about - the “scripts passing” is a “confirmation” of that fact.

Me: What if this trend continues and next 5 cycles of testing also do not produce any bugs? Would you be worried then?
Manager: No …not at all In fact, I would reduce the size of scripts being executed to say half – 2500 as the application has become stable. This is an indication for me to "cut down" testing effort, I can possibly look at automation as well.

Me: Here is a twist, what if your ALL scripts are passing but your are seeing bugs (either detected by other means or by the customers) .. Would not you doubt your test cases?
Manager: Depends upon the kinds of bugs I see … If I were doubt something or someone at all … I would doubt test results, testers integrity and project management in general. Test scripts are less likely to be at “falult”. That would process issue - we would need to tighten the process.

Me: OK … What corrective action you would take then? What you steps will you take?
Manager: I would immediately order a thorough Root Cause analysis of the defects and identify what is causing them in the first place. Tighten the development, configuration and deployment process. I would strictly enforce the processes (improved) in Testing and mandate the testers to correctly and meticulously execute the scripts and report the relevant results correctly.

Me: What if you still find bugs outside your scripts?
Manager: That would be a “hypothetical question” – not likely to happen. In any case my focus would be to improve the testing process and strengthen Test scripts. Again, if you are finding still finding bugs – probably those bugs would be “obscure” type – I might not have to bother about them…

Manager: Good… I am still not convinced that ET can give the bang for the buck. As someone who is interested in predictability and repeatability of testing, I am interested in a test process that can scale.
Me: Ummmmm … OK… what is a testing process? Is this something that “actually happens” or “something that is intended”? Is repeatability and predictability - all you care?
Manager: You are too much … there is a limit to asking questions …I don’t think this discussion is leading to any good … Let us talk about it some other time [walks out of the room]

I am continuing my discussion with this manager and post the views and continued discussions in Part 2....

A very happy new year to all …

Shrini

Advantages of "highly repeatable tests" ...

I was reading Ben simo's post on "what is software testing" - a meticulously created list of quotes about software testing. The beauty of this post is that it traces the history of software testing”. One good way to read this post is to evalauate each statement or the quote mentioned, with respect to it's relevance to software testing.

I stumbled upon this GEM from James Bach.

“Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.”

- James Bach,Test Automation Snake Oil, 1996

So, if you have an excellent set of "highly" repeatable tests, in terms of execution (automatable sequence of actions) and in terms of results (pass of fail) - congratulations, you have successfully managed to find a set of test cases or scenarios where the software is least likely to fail - meaning you will not (or do not expect to) see bugs/problems

But ... wait .. is that your testing mission?

I heard someone yelling from my back… “Yes .... that is what we expect in regression testing. But sometimes occasionally we do find bug as developer made a mistake that was caught or tester [by mistake] deviated from scripted test sequence [a process issue or discipline issue]"

What do you say?

Shrini

Dr Kaner on Software Metrics ...

Here is another gem from Dr Kaner .. this time around it is about Software Metrics ...

A rare insight into metrics world ....

http://www.artima.com/forums/flat.jsp?forum=106&thread=218013&start=30#287847

This is in responce to a thread by Alberto Savoia

http://www.artima.com/forums/flat.jsp?forum=106&thread=218013&start=0&msRange=15

I keep looking for such insightful replies and posts ... I hope readers are liking it and getting benefited by it ...

Shrini

Further on Testing as a career ..

Following this post on career in software testing , I found an interesting comment/viewpoint from Jeff Fry's post (more preciously a comment to his post by one "Steve Sandvik") on "why do you enjoy testing".

Jeff Fry's post itself is a very good post that goes in details about "Testing, Career, Enjoyment and few suggestions for tester to stufy read".

Following are Steve Sandvik's comments that worth "consideration"

"...Yes, it may be my first formal job testing software, but as so many people in testing like to point out, nearly any experience or learning has some translation to testing, if you know how to apply it. 15 years of power plant operation and maintenance experience provides an awfully large number of troubleshooting and investigation opportunities.

Identify the fields outside of your industry where, for lack of a better description, good forensic skills and an agile mind (not to be confused with an Agile mind) are at a premium. Industrial equipment field service, process and generation operations, and auditing are a few I can think of off the top of my head. "

And these comments about "in-born" testing qualities

" ...I’m not sure whether truly great testers are born or made, but I think there’s at least a component of most of them that falls firmly into the born camp–in the same way most writers would write whether or not they were paid to do it, I suspect that most people who seriously take up testing as a career for its own sake rather than as a stepping stone to something else approach the world in a certain way even when they’re not formally testing. I know I approach things from what seems to me to be a testing perspective most of the time."

I am filling my blogs with few interesting career related suggestions ... I hope my blog readers are enjoying ...

Shrini

Michael Bolton on "Software Bugs"

Continuing my earlier post on Dr Cem Kaner's comment, this time is about sharing views about software bugs by another leading light of context driven testing community - Michael Bolton.

Here are his views about "software bugs" (Again, these Michael's views in response to a question on the forum about Bugs - whole his reply stands on its own, I believe)

Personally, this is best advise that I have ever seen with respect to handing bugs by testers and how that decision impacts other stakeholders ... Read on ...

[Michael Bolton : Quote]

When I'm a tester, I'm concerned about trying to drive the project. As a project manager, that was my job. As a tester, my job was--and is--to ferret out information of any kind about the application that helps the project manager to achieve her goals. For me, this has a couple of implications.

First, I don't merely observe the product; I have to observe the things around the product--the platform, the systems with which the product interacts, the business processes, the anticipated or unanticipated users of the product, and so on. I try to be leery of recommendations to fix specific bugs, because in the past I spent too long going the other way--believing that I'm running the project when I'm not. (I'm arguably not a multimillionaire at least in part because in one company where I worked, company project managers had abdicated quality decisions to the testers and developers, which meant that we had a great, largely bug-free product that missed its market window by about a year.)

Second, there is one particular kind of bug that I will try to sell: bugs that make testing harder or slow it down. My goal is to reveal information about the product. Even if we do great testing, there are some things that we won't know about the product. Things that impinge on testing pose the risk of us knowing even less than we would otherwise.

So, with at least one eye firmly fixed on the context and the best judgement I can muster, I will advocate strongly

- to fix immediately bugs that block deeper or broader testing;
- to add testability (logging, scriptable interfaces, configurability, controllability, installability) to the product such that we can increase test coverage;
- to fix immediately trivial-looking bugs that add distraction and noise to the project effort--for example, typos that absolutely everyone will notice and report, such that the reporting and processing of the report will take time away from other coverage.

However, I also remind myself that we testers are vulnerable to representativeness bias--bugs that look trivially simply might be hard to fix, bugs that seem gnarly might be insignificant to the end-user, bugs that look hideously complex might have easy fixes, and so on. So I try to tell the absolute best story that I can about the bug and its worst ramifications, but I also acknowledge that I might not have the whole story about the technical or business reasons to fix or to defer a bug

[Michael Bolton: Unquote]

Shrini

Dr. Cem Kaner on Software Testing as a Career

I take this opportunity to share few on great discussions happening at "software-testing" Yahoo group, to all my blog readers. For the purpose of focus, I am not sharing the entire thread and the beauty of the reply is such that you can read this without knowing or referring the original post that initiated this discussion....

There were actually two replies by Dr Kaner - I am taking the liberty of rearranging few paragraphs from both replies in order to givea specific flow to the whole thing. The purpose of this post is to share the words of wisdom and experience for all those who would like pursue the career in “Software Testing

[Dr Kaner: Quote]

Let me start by distinguishing between a CAREER and a JOB. A CAREER involves a long-term, intentional focus on a field or type of work. A JOB is a temporary assignment with a particular employer. My career is focused on improving the satisfaction and safety of software users and developers. My current job is as a professor. I have also held jobs as a tester, test manager, programmer, human factors analyst, software development manager, technical publications manager, development director, organization development consultant, salesperson, software development consultant, and attorney focused on the law of software quality. Each of these has addressed different aspects of what has been, to me, the same career. People define their own careers. Many people define their career in terms of traditional categories (programmer, tester, lawyer, teacher), but the choice belongs to the person, not the category.

When you make a choice ("I am an X" or "My career is X"), that choice is both inclusive (Xness is in your path) and exclusive (if Yness is not part of Xness, and Xness is not part of Yness, then "I am X" means also "I am not Y"). When someone defines their career as "tester," I think that definition is too narrow.

I see software development as a bundle of coordinated tasks, including programming, design, testing, usability evaluation, modeling, documentation, development of associated training, project management, etc. Very few people would do all of these as part of the same job. Fewer would do them all on the same project or in the same week. But working at one company as a tester and another company later as a programmer is not inconsistent with calling myself a software developer at either/both companies

I don't generally encourage my students to pursue software testing AS A CAREER. They can make that decision later, after they have more experience. I prefer to encourage them to try SOFTWARE DEVELOPMENT as a career -- to me, development includes testing. And that they take a job doing serious, skilled testing as PART of that career. Most of the best testers I know have significant experience outside of testing and apply that experience to what they do as testers or test managers.

I think that testing is a fine choice for a first job--for some people--but that doesn't make it a first career. It becomes a first career only for the person who says, "This, testing, is my career." I don't recommend that people make a decision to narrow their career that much, early in their career. Let them explore the field more, in their next few jobs, before they lock themselves into something.

I think that some people are good at both programming and testing, some people are good at both writing and testing, some people are good at design and testing, very few people are good at every software development task. So I think it is inappropriate to say that someone shouldn't be considered a software developer because they are good at some aspects of development but not others. Most (all?) of the hidebound process-pushers that I know in the field have never done serious professional work outside of testing. From their narrow perspective, they think they know more about how to manage a development project than the people who retain their testing services. Instead of trying out their ideas as project managers (where they will be accountable if they fail) these process advocates undermine the projects they work on by trying to control things they don't understand with rigid policies and procedures, standards and superstitions, whose costs and impacts are beyond their imagination. We have too many of these people in our field. We need more people who have a broader view of the tremendous value that testing can offer--within its limited role--and are glad to embrace a service-provider role that provides that value.

I think some fresh engineers should start their career with a job in programming, others with testing, others writing, others with human factors assessment, others with configuration management, others with data analysis. I think that choice should depend on what motivates the particular person.

What makes testing worth spending time on--as a job and maybe as a career?

We are professional investigators. Rather than building things, we find ways to answer difficult questions about the quality of the products or services we test. Our job--if we choose to do it well--requires us to constantly learn new things, about the product, its market, its implementation, its risks, its usability, etc. To learn these, we are constantly developing new skills and new cognitive structures in a diversity of fields. It also requires us to communicate well to a diverse group of people. We ALSO get to build things (test tools), but very often, we build to our own designs, which can be more satisfying than building an application that does something we'll never personally do (or want to do). Learning to do good software testing requires learning to do critical thinking well, and to back it up with empirical research. Not everyone will like to do testing. Not every engineer or programmer will have the skills or the interest to do professional-level testing. But for those of us who enjoy critical thinking, experimentation, and keeping the human relevance of what we do always in mind, there is nothing else like it in software development (except, for some people on some projects, requirements analysis backed with rapid prototyping and prototype-based research).

[Dr Kaner: Unquote]

Shrini

Tester's world of Possibilities

Probable impossibilities are to be preferred to improbable possibilities. -- Aristotle
The future belongs to those who see possibilities before they become obvious. -- John Sculley

Recently, I challenged a fellow tester about testing “Notepad ->File -> Save As” functionality. I asked him to zoom on (focus) only Text files and investigating about file names (say base file name – one without the extension) and come up with testing ideas.

He started with domain testing approach and said that file names that can be supplied to the program could be classified as:

Valid File Names – those where file creation succeeds.
Invalid File names – where file creation fails and no new file gets created.

Then he went on thinking about possible values in terms of these “classes”. I argued with him about his classification – why only think about valid and invalid file names? Can you think about other possibilities …?

Few examples that I gave are –
What if file is created but it can not be opened to view?
What if the file is created but is read only?
What if the file is created but notepad application crashes during the file creation?
What if the file is created but notepad crashes while opening such a file?
What if the file is created but it takes 10 minutes to load the file?
what if the file is created but can not be searched using Windows – Search?
And so on …
My friend said “well all of these can be considered as invalid file names” … To that I said “But as per your initial classification, no file gets created for an invalid name ….!!!!”
My friend continued “ These all are possibilities but not real values …”. I said “That is exactly is my point. As a curious tester, I think about all possibilities and then investigate on those possibilities. My work starts from that point where other wash off their hands saying “We are done”.

What do you think are the possibilities when a file is entered for Notepad -> File->Save As dialog? Keep your investigations on the lines probing the file name parameter…Can you describe the "big picture"?

Shrini

100 Questions or 100/10000 Test cases in 20 minutes …

That is what I would have named a blog post that my buddy Rapid software tester (context driven tester too) Pradeep Soundararajan wrote few days ago.

If every question is a test case or represents say 10 test cases, can you create 100-1000 test cases in say 20 minutes… Any James Bach student, context driven and or rapid tester will demonstrate that it is possible ….

Without any KA/KT, training, domain knowledge, no pages of documentation …? Plain vanilla testing at its best. Unbelievable right?

One might say --- that is cheating …!!! you may scream… that is not a test case. Where are steps? Where are the detailed information about application? Can a novice, low skilled tester use this? Can this be used for next five years? Can that be automated? And run in night when people are sleeping and give the results in the morning (in plain pass fail – way)?

The answer is “No”

Please note, while we create lots “frills” or decoration around real test and call it as test case – at its core a test is a question that we ask the program and program responds to it with one more answer. Some of these answers we can observe, assess, report while lot others go un-noticed. And while this happens, the environment or platform also responds to the question.

Questioning is a key attribute of a skilled tester. Questioning is also important aspect of learning. Unfortunately, right from our childhood (remember your father and mother saying “this kid is too much – asks too many questions. You will know when you grow up”!!!), through our education and now in the Job, Questioning is not encouraged.

Why?

• Questioning is considered as disrespect or contempt
• Questioning is indiscipline
• Questioning is disturbing
• Questioning is embarrassing when answer is not available
• Questioning sometime is considered as silly and stupid
• Intelligent people do not respond to silly questions
• Questioning at times make everyone think – that stalls the progress in some cases
• Questioning in a group is considered as bad
• Questioner is a labled as “trouble creater”
• Question = Trouble, more work, Road block
• Oh Gosh, we have not thought about this at all … this is terrible, what do we do now – is not easily coming response for a question

Today’s testers are forced to follow processes, documents, checklists and other “standard” things. If one were to follow the kind of thinking that Pradeep displayed – testing can happen with least information – a lot can happen in small time. I have heard testers who can not start testing (or test design) until they get specification, training and other supporting material. This is a damaging trend for testing profession.

As a famous punch-line of Café Coffee Day (popular chain of coffee joints in India) – “A lot can happen over coffee", goes, can I say “A lot of testing can happen in 20 minutes for any application”.

Just give me the *stuff* to test … I will flood you queries that can potentially lead (if investigated and answered) to an arsenal of information about application under test …

Shrini

Types of Equivalence: Equivalence Class Partitioning - II

Following this post of mine, I have been studying deep into understanding of this technique. Here are few more thoughts related to “equivalence”.

Here is ECP in nutshell - “Group a set of tests or data supplied for an application. Assert that all the tests/data belonging to group will teach you “same thing” (application behavior). Hence it is “sufficient” to use only one value/test from the group”.

Fundamental to ECP is the concept of “Equivalence”. Most of the authors or proponents of this technique give examples of date, integer fields and demonstrate identification of classes and equivalence. For example if you consider a date field in “NextDate” program, using “generally accepted rules” governing the usage of dates in Gregorian calendar – you can identify some classes – All the dates in the month of January can be considered as equivalent (except first and last day of January and first and last month of the century – which are boundaries). These “canned” classes appear to be applicable for every application that has date field in a “next date” function. Another example would be a field of integers (1-100) – most authors have mentioned the example of 2-99 as one equivalence class meaning all numbers in the range 2-99 would be treated “alike”.

I would call such equivalence that can be arrived without knowing anything about application, it logic and programmatic implementation details as “Universal equivalence”. It is easier to explain the concept of ECP using “universal equivalence” – date and integer fields are the most popular examples. But I see a danger here – the way ECP is explained using “universal equivalence” – it leaves out lots of key details such as basis for equivalence.

What are other forms of Equivalence?

Functional Logic equivalence - Consider the “Age (1-150)” field. Application logic might enforce that Age range 1-16 considered as one eq class (Kids) and others like 17-45 (Adults) and 60-99 (Senior Citizen). This kind of equivalence is very straight forward, easy to derive. Often specifications help us to arrive at such equivalence classes. Here is where the classic examples of “valid” and “invalid” EQ classes seems to have been originated.

If one were to go by pure functional logic equivalence, it would be sufficient to model Age parameter having three eq. classes and hence one value taken each from these classes (3 in all) would provide “complete” test coverage from an ECP perspective.
Dr Cem Kaner calls this as “specified equivalence”.

Implementation Equivalence – This is where one deep dives into how data is processed (validated, accepted, rejected), passed around (within application components) and eventually stored or discarded after use. Here we would talk about programming language (data types), software platform (OS and other related programs) and the hardware platform.

Dr Kaner identifies another two sets of equivalence - “Risk based” and “Subjective”. If the equivalence is in the eyes of tester (“these two tests appear to teach me same thing”), this form of equivalence is called “Subjective” equivalence. If a notion of equivalence is established targeting a specific class of risks or errors, it is referred as “risk based” equivalence.

Thus one way to apply ECP effectively is to start with universal equivalence and go on refining the sets EQ classes as we go deep into the application and platform (add, modify, delete the classes and their definitions). Implementation Equivalence seems to be the lowest or the last in the chain overrides the specifications of classes as determined by higher levels of equivalence (universal or functional logic type)

One question to spice up the discussion – Is ECP a black box technique?
Yes if we restrict to “universal and functional logic” equivalence.
No if we deep dive into code of the application and look around at platform (software and hardware)

What do you think?

[ Update ]

ECP attempts "simplify" a big picture (data domain with infinite set of possible values"). When attempting to apply ECP for a data variable, best starting point would be "what is that big picture I am trying simplify using ECP"? This is a top-down approach - model, understand, analyse, hypothesize the big picture then go to next level and then think about EQ. classes. I have people mostly approaching this from "bottom up" approach - think about valid and invalid classes first (or even actual values) then if possible think about the big picture.

Which approach you think is a useful one to start with?

BTW, there is "Equivalance principle" by Einstein related to theory of relativity. Can I say equivalence as applicable software tests is "relative" in nature?

Shrini