Dr. Cem Kaner on Software Testing as a Career

I take this opportunity to share few on great discussions happening at "software-testing" Yahoo group, to all my blog readers. For the purpose of focus, I am not sharing the entire thread and the beauty of the reply is such that you can read this without knowing or referring the original post that initiated this discussion....

There were actually two replies by Dr Kaner - I am taking the liberty of rearranging few paragraphs from both replies in order to givea specific flow to the whole thing. The purpose of this post is to share the words of wisdom and experience for all those who would like pursue the career in “Software Testing

[Dr Kaner: Quote]

Let me start by distinguishing between a CAREER and a JOB. A CAREER involves a long-term, intentional focus on a field or type of work. A JOB is a temporary assignment with a particular employer. My career is focused on improving the satisfaction and safety of software users and developers. My current job is as a professor. I have also held jobs as a tester, test manager, programmer, human factors analyst, software development manager, technical publications manager, development director, organization development consultant, salesperson, software development consultant, and attorney focused on the law of software quality. Each of these has addressed different aspects of what has been, to me, the same career. People define their own careers. Many people define their career in terms of traditional categories (programmer, tester, lawyer, teacher), but the choice belongs to the person, not the category.

When you make a choice ("I am an X" or "My career is X"), that choice is both inclusive (Xness is in your path) and exclusive (if Yness is not part of Xness, and Xness is not part of Yness, then "I am X" means also "I am not Y"). When someone defines their career as "tester," I think that definition is too narrow.

I see software development as a bundle of coordinated tasks, including programming, design, testing, usability evaluation, modeling, documentation, development of associated training, project management, etc. Very few people would do all of these as part of the same job. Fewer would do them all on the same project or in the same week. But working at one company as a tester and another company later as a programmer is not inconsistent with calling myself a software developer at either/both companies

I don't generally encourage my students to pursue software testing AS A CAREER. They can make that decision later, after they have more experience. I prefer to encourage them to try SOFTWARE DEVELOPMENT as a career -- to me, development includes testing. And that they take a job doing serious, skilled testing as PART of that career. Most of the best testers I know have significant experience outside of testing and apply that experience to what they do as testers or test managers.

I think that testing is a fine choice for a first job--for some people--but that doesn't make it a first career. It becomes a first career only for the person who says, "This, testing, is my career." I don't recommend that people make a decision to narrow their career that much, early in their career. Let them explore the field more, in their next few jobs, before they lock themselves into something.

I think that some people are good at both programming and testing, some people are good at both writing and testing, some people are good at design and testing, very few people are good at every software development task. So I think it is inappropriate to say that someone shouldn't be considered a software developer because they are good at some aspects of development but not others. Most (all?) of the hidebound process-pushers that I know in the field have never done serious professional work outside of testing. From their narrow perspective, they think they know more about how to manage a development project than the people who retain their testing services. Instead of trying out their ideas as project managers (where they will be accountable if they fail) these process advocates undermine the projects they work on by trying to control things they don't understand with rigid policies and procedures, standards and superstitions, whose costs and impacts are beyond their imagination. We have too many of these people in our field. We need more people who have a broader view of the tremendous value that testing can offer--within its limited role--and are glad to embrace a service-provider role that provides that value.

I think some fresh engineers should start their career with a job in programming, others with testing, others writing, others with human factors assessment, others with configuration management, others with data analysis. I think that choice should depend on what motivates the particular person.

What makes testing worth spending time on--as a job and maybe as a career?

We are professional investigators. Rather than building things, we find ways to answer difficult questions about the quality of the products or services we test. Our job--if we choose to do it well--requires us to constantly learn new things, about the product, its market, its implementation, its risks, its usability, etc. To learn these, we are constantly developing new skills and new cognitive structures in a diversity of fields. It also requires us to communicate well to a diverse group of people. We ALSO get to build things (test tools), but very often, we build to our own designs, which can be more satisfying than building an application that does something we'll never personally do (or want to do). Learning to do good software testing requires learning to do critical thinking well, and to back it up with empirical research. Not everyone will like to do testing. Not every engineer or programmer will have the skills or the interest to do professional-level testing. But for those of us who enjoy critical thinking, experimentation, and keeping the human relevance of what we do always in mind, there is nothing else like it in software development (except, for some people on some projects, requirements analysis backed with rapid prototyping and prototype-based research).

[Dr Kaner: Unquote]

Shrini

Tester's world of Possibilities

Probable impossibilities are to be preferred to improbable possibilities. -- Aristotle
The future belongs to those who see possibilities before they become obvious. -- John Sculley

Recently, I challenged a fellow tester about testing “Notepad ->File -> Save As” functionality. I asked him to zoom on (focus) only Text files and investigating about file names (say base file name – one without the extension) and come up with testing ideas.

He started with domain testing approach and said that file names that can be supplied to the program could be classified as:

Valid File Names – those where file creation succeeds.
Invalid File names – where file creation fails and no new file gets created.

Then he went on thinking about possible values in terms of these “classes”. I argued with him about his classification – why only think about valid and invalid file names? Can you think about other possibilities …?

Few examples that I gave are –
What if file is created but it can not be opened to view?
What if the file is created but is read only?
What if the file is created but notepad application crashes during the file creation?
What if the file is created but notepad crashes while opening such a file?
What if the file is created but it takes 10 minutes to load the file?
what if the file is created but can not be searched using Windows – Search?
And so on …
My friend said “well all of these can be considered as invalid file names” … To that I said “But as per your initial classification, no file gets created for an invalid name ….!!!!”
My friend continued “ These all are possibilities but not real values …”. I said “That is exactly is my point. As a curious tester, I think about all possibilities and then investigate on those possibilities. My work starts from that point where other wash off their hands saying “We are done”.

What do you think are the possibilities when a file is entered for Notepad -> File->Save As dialog? Keep your investigations on the lines probing the file name parameter…Can you describe the "big picture"?

Shrini

100 Questions or 100/10000 Test cases in 20 minutes …

That is what I would have named a blog post that my buddy Rapid software tester (context driven tester too) Pradeep Soundararajan wrote few days ago.

If every question is a test case or represents say 10 test cases, can you create 100-1000 test cases in say 20 minutes… Any James Bach student, context driven and or rapid tester will demonstrate that it is possible ….

Without any KA/KT, training, domain knowledge, no pages of documentation …? Plain vanilla testing at its best. Unbelievable right?

One might say --- that is cheating …!!! you may scream… that is not a test case. Where are steps? Where are the detailed information about application? Can a novice, low skilled tester use this? Can this be used for next five years? Can that be automated? And run in night when people are sleeping and give the results in the morning (in plain pass fail – way)?

The answer is “No”

Please note, while we create lots “frills” or decoration around real test and call it as test case – at its core a test is a question that we ask the program and program responds to it with one more answer. Some of these answers we can observe, assess, report while lot others go un-noticed. And while this happens, the environment or platform also responds to the question.

Questioning is a key attribute of a skilled tester. Questioning is also important aspect of learning. Unfortunately, right from our childhood (remember your father and mother saying “this kid is too much – asks too many questions. You will know when you grow up”!!!), through our education and now in the Job, Questioning is not encouraged.

Why?

• Questioning is considered as disrespect or contempt
• Questioning is indiscipline
• Questioning is disturbing
• Questioning is embarrassing when answer is not available
• Questioning sometime is considered as silly and stupid
• Intelligent people do not respond to silly questions
• Questioning at times make everyone think – that stalls the progress in some cases
• Questioning in a group is considered as bad
• Questioner is a labled as “trouble creater”
• Question = Trouble, more work, Road block
• Oh Gosh, we have not thought about this at all … this is terrible, what do we do now – is not easily coming response for a question

Today’s testers are forced to follow processes, documents, checklists and other “standard” things. If one were to follow the kind of thinking that Pradeep displayed – testing can happen with least information – a lot can happen in small time. I have heard testers who can not start testing (or test design) until they get specification, training and other supporting material. This is a damaging trend for testing profession.

As a famous punch-line of Café Coffee Day (popular chain of coffee joints in India) – “A lot can happen over coffee", goes, can I say “A lot of testing can happen in 20 minutes for any application”.

Just give me the *stuff* to test … I will flood you queries that can potentially lead (if investigated and answered) to an arsenal of information about application under test …

Shrini

Types of Equivalence: Equivalence Class Partitioning - II

Following this post of mine, I have been studying deep into understanding of this technique. Here are few more thoughts related to “equivalence”.

Here is ECP in nutshell - “Group a set of tests or data supplied for an application. Assert that all the tests/data belonging to group will teach you “same thing” (application behavior). Hence it is “sufficient” to use only one value/test from the group”.

Fundamental to ECP is the concept of “Equivalence”. Most of the authors or proponents of this technique give examples of date, integer fields and demonstrate identification of classes and equivalence. For example if you consider a date field in “NextDate” program, using “generally accepted rules” governing the usage of dates in Gregorian calendar – you can identify some classes – All the dates in the month of January can be considered as equivalent (except first and last day of January and first and last month of the century – which are boundaries). These “canned” classes appear to be applicable for every application that has date field in a “next date” function. Another example would be a field of integers (1-100) – most authors have mentioned the example of 2-99 as one equivalence class meaning all numbers in the range 2-99 would be treated “alike”.

I would call such equivalence that can be arrived without knowing anything about application, it logic and programmatic implementation details as “Universal equivalence”. It is easier to explain the concept of ECP using “universal equivalence” – date and integer fields are the most popular examples. But I see a danger here – the way ECP is explained using “universal equivalence” – it leaves out lots of key details such as basis for equivalence.

What are other forms of Equivalence?

Functional Logic equivalence - Consider the “Age (1-150)” field. Application logic might enforce that Age range 1-16 considered as one eq class (Kids) and others like 17-45 (Adults) and 60-99 (Senior Citizen). This kind of equivalence is very straight forward, easy to derive. Often specifications help us to arrive at such equivalence classes. Here is where the classic examples of “valid” and “invalid” EQ classes seems to have been originated.

If one were to go by pure functional logic equivalence, it would be sufficient to model Age parameter having three eq. classes and hence one value taken each from these classes (3 in all) would provide “complete” test coverage from an ECP perspective.
Dr Cem Kaner calls this as “specified equivalence”.

Implementation Equivalence – This is where one deep dives into how data is processed (validated, accepted, rejected), passed around (within application components) and eventually stored or discarded after use. Here we would talk about programming language (data types), software platform (OS and other related programs) and the hardware platform.

Dr Kaner identifies another two sets of equivalence - “Risk based” and “Subjective”. If the equivalence is in the eyes of tester (“these two tests appear to teach me same thing”), this form of equivalence is called “Subjective” equivalence. If a notion of equivalence is established targeting a specific class of risks or errors, it is referred as “risk based” equivalence.

Thus one way to apply ECP effectively is to start with universal equivalence and go on refining the sets EQ classes as we go deep into the application and platform (add, modify, delete the classes and their definitions). Implementation Equivalence seems to be the lowest or the last in the chain overrides the specifications of classes as determined by higher levels of equivalence (universal or functional logic type)

One question to spice up the discussion – Is ECP a black box technique?
Yes if we restrict to “universal and functional logic” equivalence.
No if we deep dive into code of the application and look around at platform (software and hardware)

What do you think?

[ Update ]

ECP attempts "simplify" a big picture (data domain with infinite set of possible values"). When attempting to apply ECP for a data variable, best starting point would be "what is that big picture I am trying simplify using ECP"? This is a top-down approach - model, understand, analyse, hypothesize the big picture then go to next level and then think about EQ. classes. I have people mostly approaching this from "bottom up" approach - think about valid and invalid classes first (or even actual values) then if possible think about the big picture.

Which approach you think is a useful one to start with?

BTW, there is "Equivalance principle" by Einstein related to theory of relativity. Can I say equivalence as applicable software tests is "relative" in nature?

Shrini

More on definition of Test Automation ...

While discussing about "definition" of "test automation", with Ben Simo I left a comment at Ben's blog post. I am sharing that here for the benefit of my blog readers ...

(You would also find some debate about "model based testing" in comments)

Test automation involves - Automation Design and Execution of automated tests ...

Design: An act of translation of some question that one asks regarding a feature of an software application into some programmable instructions so that the question being asked is modeled with reasonable accuracy (courtesy: Michael Bolton) In simple words translating a testing question into a program

Execution: Use of a machine (hence a set of computer programs) to support any aspect of Testing with Testing at the center of whole scheme of things.
(derived from the definitions by James Bach and Michael B)

As I continue to write about definitions and terminologies around test automation, let me reiterate - Nothing called "Automated Testing" exists in this world....

Shrini

Shrini

Simple things and me ...

World likes simplicity – Pictures, Colors, Sound, Odor are used to abstract some complex behaviors.

Consider following –
Software Project Status – Red, Green and Yellow
Project Size – Small, Big, Medium
Test cases – Simple, Medium, complex
Testing – Gather requirements, write test cases (some times even prepare them), execute them, check if the pass, if pass report and go home and if fail log a bug and go home

Automation – Create an automated script using the state of the art Automation tool. Store the script in test management tool and execute it from it. See the result logged automatically. If script fails (I mean reports failure), automatically make another script to log the bug

Knowledge Transfer – As simple as Fund transfer in bank. Transfer the relevant documents to the subject – transfer. If there are clarifications there always our best friend “issue list”, “action item tracker”, “clarification list” – many names but all is one.

Knowledge acquisition – As simple as one country invades and acquires another country. Just acquire – by force

What is common between each one of these – simplicity.

I become restless when

- People tell me about testing project status in terms of Red, green or yellow.
- People start approaching testing as sequence of actions – test design, execution, bug logging and regression
- People start describing a bunch of test cases or software features as Simple, medium and complex – SMC model.
- Testers getting classified as great, lousy based on the number of test cases executed or number (not the type) of bugs logged.

To me, all of these are simplifications of some complex things that we are trying to understand. My frustration is “Why things can be so simple”? “There must be something that is hidden behind this simple thing”

Imagine, if Newton were to think – when apple fell on his head – “No big deal – it had to fall so it fell down – Gosh – my head is aching. Why did sit bellow this tree?” Same thing applies to Archimedes. Jump out bath tub and run …?

While thinking about simplicity about some things that we deal with helps us to get started off – important to understand that it was just a beginning.

While testers and most of the managers are happy about such simplifications - a skilled tester is always worried about such simplifications and often attempts to find the loop wholes in simplified models, notions, beliefs, description etc…

Nature’s simplest things have hidden inside then greatest and deepest mysteries. That would a fascination journey – we just need vehicle capable moving into it – Our imagination and curiosity. Today’s skilled testers are blessed with this imagination and curiosity. The journey has just begun …

Do you think post has horribly and outrageously simplified the seriousness and intent behind the post? Well, I can not be too serious …

Automation Dreams - Thinking about END in the begining ...

"Cherish your visions and your dreams as they are the children of your soul; the blue prints of your ultimate accomplishments."

- Napolean Hill

Thinking about “end” in the mind helps me most of the cases such as planning, checking and carrying out various day –today tasks like performing 4-5 interconnected tasks, leaving home for a week/month long journey, arranging my daughter’s birthday etc. Thinking about “END” while you are about to begin or at the time of beginning a task - can help you to visualize the entire stretch of journey from start to end, anticipate various milestones, problems and as many as details as possible. This can prove to be a powerful mind modeling concept if you train yourself in visualizing from the END and work backward from there to the beginning.

Let us apply this to Test Automation …..
- How do you describe a successful automation?
- How can one describe steady state automation in deployment?
- Can you trace a journey from the beginning of an automation initiative and the end where your automation has become “obsolete” or has reached a steady state?
- Is there any END for Test Automation as a software project?

Automation initiatives have end goal(s) to achieve – cut testing cost by certain percentage, increase test coverage by certain percentage etc. In a way every automation project has some dreams to realize. Can we describe those dreams related to automation?

Let me give it a try ….

I will have an automation suite that runs “unattended” for up to 8 hours. (Length of Execution)
My automation suites covers 40% of my regression tests for product X. (Automation coverage)
I will have an automation suite whose results I can trust the most or When my Automated Test fails – I am sure that is bug (Reliability and Trust)
I have 40% of my test cases automated. Hence my testing effort now on will be 40% less (You can dream and these are DREAMS …. )
My smoke test suite is automated – I can now make changes to the code more frequently even when there is time crunch.
And so on ..

What are you Automation Dreams? Can you describe them?

Physicians, Surgeons, X Ray Lab Technicians ....

Heard of this rant?
“We would like our business users, domain experts and subject matter experts write automated scripts (tests?)”
“Our next generation automation tool – allows persons with ZERO programming and testing knowledge generate automation scripts in MINUTES”

Think about it …. Will you ask a Physician (Medical Doctors) or Surgeon to work as XRay or Ultra Sound Lab Technician and vice versa? Or An Aneathesian to perform a surgical operation?
Each one in professional life are known to have some core competencies. Best bet will be to utilize each one to their strengths….

Business users know business domain while Automation engineers and Tester know programming and software testing.

IMHO, asking business users to do “main stream” testing and automation is sure recipe for “failure”.

Any views? Do you think the analogy presented here is logical?

More as I hear from you ….

Shrini

Inattentional Blindness ...

Sajjad has an interesting article on "inattentional blindness" and Exploratory Testing here


Sajjd mentions that - "Exploratory testing is supposed to be better at minimizing inattentional blindness". This kicks off a thread of thinking in my mind - Why it might be so? what elements in ET helps one to minimize inattentional blindness?"

Let me take a guess and answer --
May be it is "thinking between alternating polarities" while doing ET - doing vs explaining, fast vs slow, reading vs doing, focussing vs defocussing. In my opinion inattentional blindness happens due to "heavy" focus on one or more "atomic" aspects of bigger object under observation. one possible solution is to defocus as often as you can.

A related phrase (a kind of antonym) I use is "your eyes will see what you would like to see". This is more dominant theme in scripted testing where a tester is "pre-programmed" observe only expected results.

Here is a wikipedia page http://en.wikipedia.org/wiki/Inattentional_blindness

Do not forget to read James Bach's comments for this post -

"Testing is a lot like fishing. You try to create conditions that maximize your chances for catching something tasty. Exploratory testing is a little more like fishing, because just as with real life fishermen ......"

Shrini

Presenting a tutorial @ QAI STC 2007

I am pleased to announce that I will be presenting two half tutorials as a part of QAI's Software Testing conference at Delhi and Bangalore. The dates are 29th Aug 2007 at Delhi and 31st Aug 2007 (Bangalore).

The details of the conference tutorials are as follows:

http://www.qaiasia.com/Conferences/STC_conference_2007/tutorials_kulkarni.htm
http://www.qaiasia.com/Conferences/STC_conference_2007/Tutorials.htm


This half day tutorial has been created out of my last few years of "on the floor" experience of managing automation projects - right from conception till UAT and deployment. I hope to improve the material and other stuff to a day long program ..

See if you can make it to this and share your thoughts ...

Sorry for last minute announcement .. :(

Shrini

A mystery called Automated Testing ...

"No device has been invented which can match the human eye for accuracy and precision in many tasks; nor the human ear. Even the admittedly handicapped human is able to identify and distinguish among far more scents than any automated device. Human pattern recognition, language recognition, and language production ability is well beyond anything currently envisioned by automation engineers."
- Answers.com

Hope you have heard many times the word “Automated Testing” (Some places I even have heard “automatic Testing”)

Have you ever thought – what does that mean? Let us dig little and try to figure out what each term in above phrase *might* mean?

We have two words in above phrase – “Automated” or “Automatic” and “Testing”. Let us explore possible meanings of these terms ….

(A) Automated – something that happens without any human intervention – something that is done by a machine-software program or by self.

Answers.com says -
"automated" (adj )
Definition: made or done by a machine
Antonyms: by hand, manual

Automation (ancient Greek: = self dictated),

The word testing has many meanings – I present two contrasting definitions

(B) Testing is an act of technical investigation performed on behalf of stakeholders in order to reveal quality related information – it is an act of questioning – An infinite search for problems – problems that can annoy/irritate/destroy/damage a stakeholder.

(C) Testing an act of executing a test case derived from a specification and verifies whether the test case passes or fails.

(D) It is an act of confirmation that software behaves in a way as prescribed by the underlying specification/design.

(E) Testing proves that the software works as desired.

Now think about constructing the meaning of the word “Automated Testing” and we have these definitions (A), (B), (C), (D), (E) …

Automated Testing = (A) + (B) - Possible? I believe No!!!
Computer/Program can not question, improvise, learn, adapt, think, emphasize

Automated Testing = (A) + (C) – Seems possible – but in a narrow way. Using another program (automation tool) one can execute an automated test … but that is only a part of whole story – what about Test design? Defect investigation? And host of other activities that fall under testing umbrella …

Automated Testing = (A) + (D)
Automated Testing = (A) + (E)

Both of above seem to be “Not possible” as checking or proving that software works the way intended or as per spec (writing in English – hence can be interpreted in infinite number of ways) using “automation” – solely – is not practically possible.

So … think twice before using the term “Automated Testing” – as nothing really EXISTS in this world of testing called as such …

Challenge me … think of ways where you can justify the use of “automated testing” – come up with definitions of “automation” and “testing” and link them !!!

BTW what do you think about "automated tests" ??

Shrini

A Blogger Bug

A bug in blogger Create Post --

The title field is "non editable" - hence this post and my previous posts are going without any title for the post ...

I am investigating this -- any clues?

Update 13th July : The title field becomes editable when (this is one the ways I figured out) I edit (quick edit from the blog ) the post with the blank title.

Shrini

Regression Testing and Insanity ....

"Insanity is doing the same thing, over and over again, but expecting different results"
Albert Einstein (attributed to Ben Franklin too)

Does this sound similar to what we do in the name of "regression testing”? In regression testing (more so in that regression testing whose test execution is automated - note I am not using the term "automated regression testing") , we believe that by doing repetitions of some things that we did previously - we can find new results.

Thanks to inherent complexity in software and involvement of humans in development, testing and usage of the same - this regression insanity appears to be *working*

This one of those Francis Bacon's idols - Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

Read following to know more about Francis Bacon and how his idols are relevant in testing

Francis Bacon’s New Organon (James Bach)
Bacon and boundary testing (Mike Kelly)
The four Idols of Francis Bacon
The Four Idols of Sir Francis Bacon by Ben Chambers and Zeb Dahl.


More on this later...

Shrini

My acrobatics on the Boundaries ..

The boundary condition of the universe is that it has no boundary." The universe would be completely self-contained and not affected by anything outside itself. It would neither be created nor destroyed. It would just BE

Stephen W. Hawking - A BRIEF HISTORY OF TIME

Science is a differential equation. Religion is a boundary condition- Alan Turing

Boundaries are actually the main factor in space, just as the present, another boundary, is the main factor in time- Eduardo Chillida

Earth has its boundaries, but human stupidity is limitless – Gustave Flaubert

I would read above quote from French novelist of 18th century (Flaubert was regarded as the prime mover of the realist school of French literature and best known for his masterpiece) by replacing the word “Stupidity” with “intelligence” or “imaginative power”.

Mike Kelly has an interesting post related to boundary testing here

I blogged about this topic few months ago under the name BVE and I think it makes perfect sense to link these two posts and get a new perspective of boundary test design (part of domain testing)

Note following important points in Mike’s post (with my views/comments) ---

1. Understanding, Identifying and working with boundaries in software - is a Modeling problem. There can be multiple ways in which boundaries in a software system can be modeled.
2. Depending upon how *close* your model models the system behavior, your (boundary) testing can be incomplete or wrong to *that* extent. Since there can be boundaries out side your model – you will not notice them.
3. There can be (are) multiple boundaries - what you notice is limited by your model (ref. #1)
4. No boundary exists in isolation
5. All boundaries, those that are explicitly identified in model – hence are known to you and those laying outside your model, INTEREACT and ALTER/AFFECT system behavior in some *significant*. This introduces complexity in the model. In other words there exists some RELATIONSHIP between these boundaries.
6. Inputs in identifying (in fact for modeling) the boundaries come from various sources such as technical specifications, user expectations, requirement specifications, device/OS specifications. If you narrowly focus on any one source of information – you will miss modeling of others boundaries (outside the mode) hence you would miss bugs/system behaviors associated with them.
7. A weak boundary Analysis would start with boundary first without first having a thought /explicitly/ about my model.
8. A strong or useful boundary analysis would start with model and identifying boundaries resulting from all possible sources of information.

Don’t forget to check following ---

All models (those especially having predominant influence of domain testing) are APPEAR to be deterministic (algorithmic/mathematical representations - set, graph, state machines etc) but indeed are HEURISTIC models as act of modeling seem to follow (in most of the cases) a heuristic approach…


Shrini

Tester's Quiz: What is Equivalence Partitioning Technique?

Most of you have heard this term "Equivalence Partitioning Technique". It is one of the common techniques associated with black box test design techniques (under Domain Testing techniques). Today's Quiz on my blog is related to this...

Explain in 4-5 sentences or less about "Equivalence Partitioning Test design Technique" and provide an example that illustrates the technique (avoid examples like "login" screen)

Notes:
I would also appreciate lots of "context" specific questions like "Who is asking?", “Why 4-5 sentences” etc …

I will consolidate all comments and questions and make another post later....
I am also looking for "real life" experiences of applications of this technique....

Some background:
One motivation for me to present this to my blog readers is my own experiences of discussing in various job interviews (where I was the interviewer) and lots of informal discussions with people. In my experience, lots of testers (especially in India) seem to have a very narrow view of this technique. Most of them have admitted that "it is a theoretical technique - useful only in interviews... etc"

What would happen next?

I would like to de-mystify this stuff (with your views, questions and opinions) and present to you my version of EQ partitioning technique in a blog post....

Shrini

Skill Vs Technique ...

Mathew Heusser, wrote following lines for an article for Better software..

“…..instead of listing specific techniques, we might be better off listing skills …Once we have the skills, the techniques and terminology can help. After all, our community created test terminology as a servant …”

That is excellent point. In the industry like our where “terminologies and jargons” rule, focusing on skills would be greatly useful.

Skill - A skill is an ability, usually learned and acquired through training, to perform actions which achieve a desired outcome. It is more of a subjective attribute.

Example:

Writing
Speaking
Problem solving
Critical thinking and General system thinking
Modeling
Questioning
Creativity

Technique - A well-defined procedure used to accomplish a specific activity or task. More than one technique may be available for accomplishing a specific activity or task. Selection of the technique should be based on project approach, available tools, cost and time constraints etc.

Note that a skill denotes an “ability” where as Technique is a “procedure”. It becomes very easy for a technique to be lost in the jungle of jargons and terminologies – for skill it is less likely so …

Invest time in identifying and developing skills are important to your job …

Shrini

Tester's Tool kit ....

James Bach mentions about a tool kit - a small set of utility programs that every tester needs to have to help in his/her testing, here ...

Programs like perfmon, burp poxy, filemon, regmon etc can be really useful and check here for other windows platform utilities

So far I have been using ethereal as network monitoring tool ...

Here is an improved version from microsoft Netmon 3.0

Shrini

Michael Hunter Interviews me for DDJ ...

Michael Hunter aka Braidy Tester, interviewed me for DDJ. See the interview here

With this I have joined illustrious list of Whos who in software Testing arena...

Michael has interviewed following people earlier (few notable ones) ... When you have time, go through each of these interviews and figure out how each one of following thinkers and great visionaries in testing present their views and their world of Testing ...

Gerald M Weinberg
James Bach
Michael Bolton
Jon Bach
Brian Marick
James Whittaker

I interacted with Michael from my Microsoft days. My first ever blog (on msdn) mentions Michael as sole inspiration for me getting into blogging. I enjoy reading Michael's blog and I keeping checking it once in every 2-3 days to see if anything has come up new there...

Michael hunter represents a unique testing culture within Microsoft and it is good see him working with people in context driven testing community. That makes him more special as a context driven microsoftee ....

Thanks Michael for featuring me in your 5 questions interview series ... I hope that will inspire many more testers in India, especially, to embrace testing and grow this community

Shrini

Reliving TASO ...

Ben Simo and Sam Burgiss reminded me about "mother of all" paper on Test automation.

Yes, I am talking about James Bach's Test Automation Snake oil ... It is kind of bible for all Test automation professionals.

This is nearly a 10 years old article (original in Nov 1996 and a rewrite in June 1999) - I am surprised to see, like Nostradamus (a Famous a French apothecary), James, 1o years ago predicted and neatly articulated "wrong" ways of applying automation. The statements and observations made in the paper are mostly valid and applicable in Automation world even today.

I have started learning about Automation from this paper in a “real” way. Before I got into what I call it as "Holistic Approach to Automation" (for the lack of any better word) I did some automation here and there using some tools and later I realized how bad that was.

Today, I see so many young and upcoming testers falling into the trap and start learning Automation tools straight without knowing about Testing and how automation affects human testing. And there are tool vendors, training institutes and recruiters/head hunters – all just look for few words “automation" and names of some leading GUI automation tools - you got a job.

My advice to all who would like to pursue the career in Test automation...

1. Knowing tool is a very small part of you as automation Engineer.
2. Invest time in learning "Testing" especially human side of it and practice
3. Pick up a programming course - learn programming - C, C#, Java, VB, PERL can be good candidates.
4. Read anything by TASO by James and his other articles ... When I say read - don’t just go over them once and discard them ... Understand them deeply, discuss them, if you have question - share it with others in the community -- write to me I can help you with my views ...
Every time I read James' articles - I get new ideas, new thinking, new interpretations...

After doing all above - if you get time - use (I say use) an automation tool ...That is last thing that you require in the path to success in Test Automation...

You might also want to have a look at following noteworthy references related to Test automation...

Seven Steps to Automation success by Bret Pettichord
High Volume Test Automation Cem Kaner

James - you should seriously consider writing a sequel - TASO part II ...

Though you say Agile Automation is your TASO II, the plot appears to be not the same -- we are carving some more on the lines of TASO ....

Shrini

I will be presenting at STAR EAST ...

I am pleased to announce that I will be a track speaker for this year’s STAR EAST conference at Orlando, Florida May14-18. I am very excited to speak this biggest and most popular conference dedicated to software testing. I will be speaking on “10 classic Reasons why Test Automation projects fail” (Track T17- on May 17th) - one of my popular paper till date.

I also excited to get on stage for 2 other occasions other than my presentation –
Lightening talk for 5 minutes on “Traps in test estimations” - see the announcement regarding this by the moderator “Mathew Heusser”, here. It is really privilege to be in the line up for lightening talks that includes “who’s who” in “Software Testing” world – James Bach, Scott Barber, Michael Bolton, Mathew Heusser …

I will be chairing the session for “Test outsourcing track– Track F8” helping the speakers with introduction and time keeping.

This conference being my first STAR EAST conference (first ever) will also is a special one as I will be meeting some special people whose work has greatly inspired me …
I hope to see and chat with people like Cem Kaner, Jon Bach, and Ben Simo and off course James Bach , Michael Bolton and Scott Barber…

Drop a line to me if you are interested to meet me and discuss the common passion - software testing…

See you there …

Shrini

Story around a bug ....

I found this interesting, lucid and narrative story related to a software bug (or bugs) here. Something on the lines of Story of testcase that I wrote sometime back.

Happy bugging ...

On a different note, do check out Jonathan Kohl's post on "Getting started with Exploratory Testing Part 1" here

Shrini

Ten Software Testing Myths ...

I was reading Lidor Wyssocky’s blog and his post on 10 software development myths. I thought, let me re-do this list for software testing – replace words “development” by “testing” and other relevant words. Here is the list … Bingo … we, the software testers *nearly* share the list of myths (frustrations?) …

It is interesting to note that last 5 myths go unchanged … development and testing share the honor. I even doubt that Lidor might be a software tester or a developer having a strong tester like mind.

10. The tester’s task is easy: he should merely write and execute the test cases by translating requirements to test cases. Additionally log some bugs.

9. Every test case is documented. Otherwise, how on earth can we expect to do regression testing and in general repeat testing?

8. Test case Reviews are a one-time effort. All you have to do is take an artifact after it is completed, and verify that it is correct. Test case reviews, for example, should merely verify that *all* requirements are covered by test cases and EVERY REQUIREMENT is COVERED by AT LEAST ONE TEST CASE.

7. Software Testing should be like manufacturing. Each of us is a robot in an assembly line. Given a certain input, we should be able to come up automatically with the right output. Execute a set of test cases (should execute 100 test cases a day) and report pass/fail status.

6. Software Testing has nothing to do with creativity. Creativity – what? The only part which requires creativity is designing your assembly line of test case design. From that point on, everyone should just be obedient.

5. Creativity and discipline cannot live together. Creativity equals chaos. [This one remains unchanged from original list of software development myths]

4. The answer to every challenge we face in the software industry lies in defining a process. That process defines the assembly line without which we are doomed to work in a constant state of chaos. [BIG ONE …This one remains unchanged from original list of software development myths]

3. Processes have nothing to do with people. You are merely defining inputs and outputs for different parts of your machine.

2. If a process is not 100% repeatable, it is not a process. Letting people adapt the process and do “whatever they want” is just going back to chaos again.

1. Quality is all about serving the customer. Whatever the customer wants, he should get. Things that don’t concern your customer should not be of interest to you.

Like Lidor, can say "As I said, I guess we still have a long way to go…" ?

Shrini

this app can break ... Notepad Bug challenge ...

Jon Bach is another brilliant thinker, speaker and coach in the area of software testing. Along with his "illustrious" and "one and only" James Bach - Jon has been doing some pioneering work in the area of exploratory testing and human aspect of testing.

While I was reading Jon's interview by another famous "Michael", the braidy tester - Micheal Hunter , I bumped into this interesting bugs that Jon mentions. This happens to be Jon's one of the most interesting bugs.

In words of Jon "Run notepad.exe and type "this app can break". Save it as test.txt, close it, then open it again. You might see that it now shows all rectangles, as if the font couldn’t be loaded."

There are other mentions of this bug here ( "Bush hid the facts")

I bet as you read this, you will be trying see if bug appears --- Bingo ... what an interesting bug ...

I did some very initial investigations around "nailing down" the bug - I posting them as they are -- I can not wait until I completely nail down the bug. It is too tempting for me to blog about it....

Successful cases
===================

this pap can break
this app can creak
this app nan break
thin app can break
shin app can break
shis app can break

this app can breal

aaaa aaa aaa aaaaa
bbbb bbb bbb bbbbb
aaaa bbb ccc ddddd

1111 111 111 11111 (interesting)


Unsuccessful ones
=====================

1a11 2b2 3c3 4ddd4
1aaa 2bb 3cc 4dddd
this cap can break (interesting)
THIS APP CAN BREAK (interesting too)

Few other observations
======================

Does not happen second time in a same file
Does not happen if a file with garbage appears once and create a copy of the same and open and check.

Investigation continues …..

What are your takes?

Finally, when you give up and do not find any more ideas as what is happening -- See here , here (good ones) and here... there are some interesting discussions and paths of investigation …. But don’t read (cheat) until you give your best try … I would say using Google would also amount to cheating …

BTW -- have seen following testing challenges put up in blog world in recent days?

1. Elisabeth Hendrickson’s Triangle challenge
2. Mathew Heusser’s challenge


Shrini

Ways to make Test Automation difficult ...

One of the interesting combos in Test automation is “Test design and Extent of achievable Automation”. I have seen in several automation projects that I worked, involving commercial automation tools that test design is the single most important aspect that decides the extent of automation one can achieve in a automation initiative. Techincal feasibility which is decided by the tool or tool set is another important parameter that affects the reach of automation

The context here is that test cases are developed first with the intent of manual execution and later picked up for automation. This is a very common scenario that is prevalent today.

Let us consider two following situations or examples that indicate two distinct and extreme styles of test design

Design 1: “Understanding tests require domain knowledge” - Tests are designed and documented in a style that suitable for someone who is expert and has a familiarity with the application. Abbreviations and shortcuts etc are used very commonly. For a tester who does not have previous experience (they call it as domain knowledge), the tests does not make any sense and look like incomplete. Objective here is to write tests in quick time so that only a select few can understand. An analogy would be a prescription written by a doctor that only a pharmacist can read and understand (this is a regular practice for the Doctors in India). Test design of packaged software apps, ERP applications is seen to follow this approach


Design 2: “Even college students should be able to execute the tests” – This is other extreme of test design where, the tests are detailed to a greatest detail possible so that each documented test is complete unit. Some of these tests even start from “Start windows ….” The intent of test design is to reduce the time required for a new person to understand and execute the tests. The drivers for such a test design/documentation style are high resource turn over and high cost of testing resources. IT organizations with very small or no testing teams on their own seem to follow this approach of test design.

Now, which test design style would be suitable for “Automation”? Which test design style lends itself for automation?

I would say none. In my opinion, both of these test design styles are excellent examples or ways of making the automation “difficult”

Important questions in this case would be “What is testing mission” and “what are objectives of Automation”. Unfortunately, the proponents of both of these styles of test design tend to down play the importance of these questions.

An excellent example of “context free thinking” …

Quoting James Bach --- “What makes something not context-driven is partly that the methods applied don't solve the problems at hand. This is the most common situation I encounter with large projects: somebody thinks that documentation is important, but they don't think much about whether the documentation they produce is really helping the situation, or rather hurting it.”

-- Shrini

Boundary Value Exploration - BVE

Happy New Year and here is my first blog post in this New Year. I can not believe that I did not blog for nearly 3 months. I am thankful to the readers of my blog for posting comments on my previous posts and really keeping me on my toes. I am privileged to have likes of Cem Kaner to comment on few blog posts.

This is cricketing time (World Cup at West Indies starting from March 11). Being one the avid cricket watcher and enthusiast – I will be eagerly hooked on to idiot Box for all matches ….

When we are talking about cricket, boundaries, sixes and all excitement of cricket – How can I forget – testing? Thanks to Pradeep, a fellow Rapid/Context Driven Tester (oops I have started calling myself a Rapid Tester ….) I can find a blog post here talking about hitting boundaries and sixes in Testing.

You will find some interesting debates and discussions about Pradeep’s post at BJ Rollison’s Blog here, here and here

This is an interesting discussion about linking boundaries in testing to that in cricket. Most of us while learning the ropes of test design – have dealt with “Boundary Testing” or “Boundary Value Analysis”. Learning about boundaries for a new tester is like learning ABC.

Let me dig around and present some interesting stuff on this topic.

Introduction and the context

BVA or BVT is considered as Testing/Test design technique in black box testing methodology. Black box testing takes an external perspective of the test object to derive test cases.

What is a boundary?

• A separation, natural or artificial, which marks the division of two contiguous properties.
• The line or plane indicating the limit or extent of something

When applied to Test design – I would say a boundary for a variable is a periphery or a temporary limit at which some noticeable change in the application behavior happens. Some of these boundaries are created are due to application intended behavior or documented specifications – business rules (say “for getting a driver license, the applicant must be at least 18 years old”). Some are “implicitly” created by program/code implementation – A value greater than 4294967295 can not be stored in an unsigned integer variable. At each of these boundaries – some change in the behavior happens - that is what we would like observe. Boundaries are special things – strange things happen near and beyond them.

Additionally note that –

• It helps (by the way of simplification) to apply the term “boundary” to parameters, variables, attributes, properties, fields of a given test problem. Using the term “boundary” any bigger context as in “boundaries for the program or application or software” – can complicate the task of the thinking and modeling the test space. I would like to call anything that can be manipulated during the usage of a software application as a variable.

• During boundary testing – each variable is taken one at a time – meaning effects of more than one variable’s boundaries are not considered. Combination testing (pairwise, orthogonal arrays etc) can help in considering multiple variables with 2 or more considered at a time.

• There can be multiple boundaries for a given parameter – few imposed by application functionality or business rules and few others by programmatic implementation via temporary and permanent storage and processing of the information/data.

• A given parameter being analyzed for boundaries can be further modeled to be consisting of several subsets each having their own boundaries or edges. Identifying these subsets for a parameter can help greatly in identifying multiple boundaries. By the way these subsets are referred as “Equivalence classes”. The technique is also referred as “partitioning”

• If we can identify “n” equivalence classes (each having at least 2 boundaries) then we have identified “2n” boundaries for a parameter.

Why boundary?

It is believed that “bugs lurk around boundaries” – developers tend to be “lazy” or “careless” around boundaries. Note that this is belief and at times can be wrong. Hence, testing or designing tests “focusing on boundaries” is a “low hanging fruit”. How many times you have seen a tester jumping to enter a “huge – Stttttttttttttttttttttttttttttting” in an edit box that is designed to receive a string input. It is really tempting to check “stuff” at boundaries.
Given the scale of emphasis on boundaries, sometimes I get a feeling that “developers” have become intelligent and tend to “take care” of these.


What is a boundary Value Testing and Analysis?

Wikipedia: Boundary value analysis is software testing related technique to determine test cases covering known areas of frequent problems at the boundaries of software component input ranges.

FOLDOC (Free online dictionary of computing, UK): A test data selection technique in which values are chosen to lie along data extremes. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a systems works correctly for these special values then it will work correctly for all values in between.

Examples of boundary value tests:

• Maximum negative, maximum positive, and 0 inputs or outputs;
• Input or output strings at size limits, 1 character beyond size limits, the empty string, and strings of 1 character;
• Input or output numeric values at size limits and 1 beyond size limits;
• Empty input files and files with one character in them;
• For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1);
• If input conditions specify a number of values n, test with (n-1), n and (n+1) input values;
  Apply the above to output conditions (e.g., generate table of minimum and maximum size);
• If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.

Where applicable?
A test object or a problem that can be modeled in terms of variables and domains. An approach to Domain Testing.


How boundary testing is done?

Here is how most of the boundary testing I have seen, happens –

• Identify the fields (variables – mostly input type) for the feature to be tested
• Refer to an available specification (raise alarm or shout if you don’t have one) or ask developer and get the boundaries for the fields identified above in step. Optionally raise an issue or bug on specification if it does not “clearly” states the boundaries.
• Derive a matrix containing the fields, corresponding boundaries and a set of test inputs at each of these boundaries.
• Declare that boundary testing has been done.

What is the problem here?

Let us analyze above process of performing BVA and BVT (boundary value testing). Above process is built upon several (reckless) assumptions that are often not “thought thorough” and “incorrect”. If your assumptions are false – your BVA/BVT looses its value and credibility.

• BVA/BVT is a universal Technique. If you ask any “upcoming” tester about “test design techniques” – invariably he will start and mostly end with “BVA”. Not knowing when BVA can not be applied can be “dangerous and costly”.
• Specification is a must for doing boundary analysis.
• Developer can/will tell you what is the boundary – Just go by what he/she says.
• All the variables can be modeled to have a linear scale with an upper bound and lower bound – there are 2 fixed boundaries per variable.

Boundary value Exploration -

Traditional approach to boundary testing – the whole process appears to be very simple – a notion of few fixed/well known and documented boundaries and testing with values around these.

Instead of BVA – I propose to coin this term BVE – Boundary Value Exploration (I just checked Google to make sure that someone has not already used this term).
Let me give you few thoughts around this new term as I am thinking about it. In next few posts I will cover the details with examples.

Instead of looking at test variables as numbers on a linear scale, in BVE, we would be exploring the variables for interesting behaviors – our objective would be discover as many as boundaries as possible and then study them. These boundaries would then be subjected to all sorts of “tortures” so that they can reveal some “interesting information”

While modeling and exploring boundaries for a parameter of a testing problem or a feature, I suggest to approach it from two angles. Please note that all these parameters are explicit (meaning those can be traced to specifications or product code). I would cover “implicit” parameters later (next few posts)


BVE – Outsider’s View:

Anything that is visible to an end user – typically that is/can be manipulated during the program usage is covered while viewing like an outsider. This is a typical black box approach. I use the term “outsider” to indicate that details like actual programming implementations, programming language, data types etc are either not available or test designer takes a deliberate view to temporarily ignore them.

While performing BVE as an outsider, you would consider things like business rules, constraints imposed by the application behavior (black box), specifications, error messages, application logs etc.


BVE – Insider’s View:
This is all about “code and related information related to implementation”. When performing BVE from insiders view – you would be looking into various details like programming languages, data structures, data types, use of relational operators, loops etc. You will use the information gathered during “BVE –Outsiders view” and link them to various data storages used for data processing. In process you might discover new boundaries, corresponding values.

Let me conclude this part of BVE – by saying

“There are boundaries to be explored (followed by analysis) with respect boundary testing – some of them appear to be fixed (until proved otherwise), some are not clearly understood, some are not known but can exist."

After all this, one thing is *nearly* sure “There are no fixed boundaries” and there is a great deal of suspense, and mystery around these boundaries, waiting to be solved.

Shrini

Updates :

There are two other interesting references to boundary testing

1. Boundary Testing by Mike D Kelly and James Bach

2. Testing with numbers by Karen N Johnson (thanks BJ for pointing out this one)

Why counting is a bad idea

Let us consider a typical Test report - a report that is presented in a meeting for assessing the progress of testing, attended by key stakeholders and team members:

No of Test cases prepared: 1230
No of Test cases Executed: 345
No of Test cases Failed : 50
No of Bugs reported: 59
No of Requirements Analyzed : 45
No of requirements updated :50
No of Transactions covered in Performance Testing : 24
No of Use cases Tested : 233

Productivity

No of Test cases prepared Per Person Per hour = 5
No of Test cases executed per person per hour = 15


What do you see here?

Managers love numbers - Numbers give objective information, numbers quantify observations and help in taking decisions (??). Numbers simplify things, one can see trends in numbers.

You might have heard about one or more of above statements (mostly in review, progress meetings right?). When it comes to testing, followers of Factory approach testing, are comfortable in just counting things like test cases, requirements, use cases, Bugs, passed and failed test cases etc and take decisions about "quality" of the product.

Why counting (without qualifications) is bad idea in testing? What are disadvantages of such practice? Let us briefly take a look of few famous frequently *counted* things

Count Requirements (as in there are 350 requirements for this project)
Can we count?
How to count? Do we have a bulleted list of requirements? If not, what to do?
How to translate given requirements into "bulleted list"
How to account for Information loss, interpretation errors while counting requirements
Count Test cases ( as in test team has written (or designed or prepared) 450 test cases in last week)
Test cases are test ideas. Test case is only a vague, incomplete and shallow representation of actual intellectual engagement that happens in testers mind at the time of test execution (Michael Bolton, mentioned this in his recent Rapid software Testing workshop at Hyderabad)
How can we count ideas?
Test cases can be counted in multiple ways - more often than not, in a ways that are "manipulative" - count is likely to be misleading
When used for knowing or assessing Testing progress - likely to mislead the management
Count Bugs ( we have 45 bugs discovered in this cycle of testing so far)
The story or background about a bug is more interesting and valuable than the count of bugs ( this again I owe it to Micheal Bolton - "Tell me story about this sev 1 bug? would be more informative and revealing question than asking how many sev 1 bugs we have uncovered so far?
when tied to testers effectiveness - is likely cause testers to manipulate bug numbers ( as in Tester 1 is great tester as he always logs maximum number of bugs)
Let us face the fact of life in software testing - there are certain things in testing that can not be counted as we count no of cars in the parking lot, no of patients visited a dentist's clinic or No of Students in a school.

Certain artifacts like test cases, requirements and bugs are not countable things and any attempt to count them can only lead to manipulations and ill-informed decisions.

Wait --- Are there any things at all in testing that we can count without loss of effectiveness and usefulness of the information that a counted number might reveal?

Shrini

How can a software tester shoot on his/her own foot?

Would like to know the self destructive or suicidal type of notions of the today's tester? Would like to know how a tester can shoot his/her own foot?

There are many ways – one of them is by “declaring” or "asserting" that -

Software Testing is an act of (whole or Part) Software Quality Assurance.
Few variations of above –

Software Testing = Software QC (quality control) + Software QA
Software Testing = Verification + Validation.

As an ardent follower or disciple of Context Driven school of Testing – I swear by following definitions or views

• Quality – “Value” to someone (Jerry Weinberg)
• Bug (or defect or Issue) – “Something that threatens the Value” (I am not sure about the source)
OR “something that bugs somebody” (James Bach)
• Whatever is QA – that is not Testing – Cem Kaner
• Testing – Act of Questioning product with an intent of discovering Quality related information for the use of a stakeholder (James Bach /Cem Kaner – I attempted to combine definitions by both James and Cem)

OR
• An Act of Evaluation aiming at Exploring ways in which the Value to Stakeholders is under threat (this is my definition – which discovered quite recently – open for criticism)

• Stakeholder – is someone who will be affected in success or failure OR actions or actions of a product or Service (Cem Kaner)

• Management is the TRUE QA group in a organization (Cem Kaner)

Now let us see how notions that assert to Testing as QA or combination QA and QC roles are self destructive or similar to shooting on ones foot …

1. The terms like QA, QC were appear to have barrowed from Manufacturing Industry – Can you measure and assess the attributes of software in the same way as you do for a mechanical component like piston or Bolt?
2. You can not control or assure Quality in a software By Testing
3. It can be more dangerous or costly to claim as a Tester that “I assure or control Quality by Testing” as it can backfire when you don’t.
4. Unless your position in Organization hierarchy is very high – you as a tester can NOT TAKE decisions about
a. Resources and Cost that is allocated for the project (Budget)
b. Features that go into the product (Scope)
c. Time when product will be shipped out of your doors (Schedule)
d. Operations of all related Groups - Development, Business, Sales and Marketing etc.

When none or most of above not in your hands – How can you Assure or control quality?

5. When you claim that you assure or control quality – others can be relaxed – Developer can say – I can afford to leave bugs in my code – I have anyway some paid to do the policing job Or others will say “Let those Testers worry about Quality, we have work to do” – Cem Kaner

6. You will become scapegoat or Victim when you leak Bugs (or issues or defects) go past you. One of the stakeholders may ask – “you were paid to do the job of assuring or controlling Quality – how did you let this bug(s) to product”

An interesting and relevant is reference is mentioned in Cem Kaner’s Famous article
“The ongoing revolution in software testing”

Johanna Rothman Says (as quoted in Cem Kaner’s article) -

Testers can claim to do “QA” only if the answers to the following questions, is YES
• Do testers have the authority and cash to provide training for programmers who need it?
• Do testers have the authority to settle customer complaints? Or to drive the handling of customer complaints?
• Do testers have the ability and authority to fix bugs?
• Do testers have the ability and authority to either write or rewrite the user manuals?
• Do testers have the ability to study customer needs and design the product accordingly?

Clear enough?

What is the way out ---?

Treat software testing as a service to Stakeholder(s) to help them conceptualize, build and enhance the *value* of a product or a service.

Be a reporter or service provider – Don’t be Quality Police or Quality Inspector of an assembly line …

Launching Indian Software Testing bloggers community ...

Are you someone from India or of Indian origin?
Do you work in/for Software Industry?
Do you do or have interest in Software Testing?
Do you read blogs on software Testing?
Do you blog?
Is Software Testing is your passion?
Do you believe in sharing knowledge in software testing community in India?


Friends – if ,answers to one or more of above questions is “Yes” – please send me an email – I am launching “Indian software Testing bloggers” community – a Platform all passionate Indian Bloggers out there. I need your support, energy, passion to build this community.


I have thought of one or two ideas as how do I host the community on the web, what is the charter of the community etc … please share your comments ….

Let us start this with a small step – who knows one day might take the shape of big “Revolution” in Indian software Testing.

I already have notable people like Pradeep S (who blogged recently calling Indian testers to starting blogging here) with me and guidance and blessings of world renowned visionaries like - James Bach, Michael Bolton.

Here are my contact details –

Shrini Kulkarni
Email: shrink@gmail.com
Cell: 91-9945841931

“What topic in Testing you want to blog today?”

Story of a Test case ....

A Test case or Test is an important entity that we as testers create/use/work with as part of our testing activities. What if a test case were to come Live like a living thing or Ghost and were tell it's story or it's life cycle ---

Here is how it MIGHT go ...


• Born – in word document or excel or in some text file or HTML form of web based test tool – very rarely in the minds of a tester.
• Some form of Reference document (requirements, design or Functional Spec) is considered to be my one parent while my other parent is Application behavior that I am suppose to check and verify.
• I exist for proving that my both parents are one and they don’t have conflicts.
• My body structure is such that when one of my parents changes it’s shape or form – I need to change otherwise I temporarily get dumped and do no exist –get invalidated.
• I live in different types - manual, automated, semi automated, documented, un documented, versioned and non versioned, In test management tool, in informal documentation.
• I am a countable *thing* though I differ from my siblings, cousins, friends in may ways.
• Named by a tester and pushed into some repository I have an ID
• I am referred in many ways like test, test case, Test idea, Test spec, Test procedure, Test pack, Test set
• Sometimes I am too detailed that a school kid can execute me by following the steps and sometimes I can be very tricky. Some time I become very lengthy and some times just a one liner – “Verify this….”
• Sometimes I have hard coded data and some times I will not have expected results. Some times my internal parts contradict each other.
• I get classified as “simple, medium and complex” so that people can measure time for creating or modifying, automating or executing me
• I get a graduate degree when I people start calling me as ‘Regression test” – I need to pass every time I get executed.
• Developers hate me when I fail and managers would like see only “passed” tests.
• Some times I am called unit test or some time end to scenario
• Some one will review to confirm that I am indeed born to my logical parents
• I tend to loose my identity when some one automates me and forgets me that I ever existed.
• Some tester adopts me and uses me and abuses me and cruelly compares me against my parents and declares that I pass or Fail.
• I just go into hibernation mode every now and then (when either or both of my parents change their form and shape)
• I am at mercy of tester to look for me changes that might be required to bring me back to life (from hibernation)
• When one or both of my parents change their form and shape OR when one or both of them die (Feature deleted form the application and reference document) – That is the end of my life – I get deleted.

Interesting Right?

What is the story of your Test case?

Shrini

Some more interview tips - Questions that you should ask to the interviewer ...

Continuing from my last post on this topic, I would like to touch upon an interesting and important aspect of a job interview from job aspiraint's point of view.

1. Asking questions about Employer and his company
(Demonstrate that you have researched about the company and (already)know any publically available information)

i)Nature of business, Size, office locations, Company's history
ii)Company's achievements in the recent past
iii)Company's Financials
iv)Organizational hierarchy and the where the position for which you are being interviwed fits
v)Information about competators
vi) Ask about customers and company's standing in it's operating domain
vii) Company's future plans about consolidation, diversification, expansion etc


2. Other high impact questions

i) What are the immediate challenges that you [the manager doing the interview] are going to face in the next 3 months? Are there ways that someone in the position you're hiring for could help address those challenges?

(source Johanna Rothman's this Blog Post:

ii) A year from now, how will you evaluate if I have been successful in this position? (Source : Louise Fletcher's Bluesky Resumes Blog)

iii) what is next step? Where will we go from here?


Suggestions, views and comments Welcome

Shrini

Context Driven thinking in Testing ...

I have been discussion/arguing with BJ Rollison on the issues of "Schools of testing" here ...

http://blogs.msdn.com/imtesty/archive/2006/10/20/end-segregation.aspx

BJ is suggesting to end the seggration of four schools of testing - which I strongly disagree.

James Bach blogged on this here http://www.satisfice.com/blog/archives/74

and look at this simple explaination (by James again ) of equating adapting to a context to "parenting" here ...

http://www.satisfice.com/blog/archives/60#comments

"There’s only one context that matters– the one that you are in at the moment. If that changes, then you adjust accordingly. It’s like parenting. You don’t have to figure out how to parent every child, just the ones that belong to you. The context-specific attitude says adapt to your children and then stop adapting. The context-driven attitude, taken to its logical conclusion, is like a child psychologist’s approach. Child psychologists need to know how to adapt to any given child (normal and strange) who walks in the door.


For the same reason, if you figure out how to report coverage on your project in a way that works for you, you can’t assume that the same method will work for me, nor do you need to worry about whether it would work for me. You can’t tell me “James, I have discovered the right way and you should do it my way, too.” What you say, intead, is “James, would you like me to describe some experiences I’ve had with coverage reporting? I feel good about how I do it, over here in my project.
"

Expect more on this in coming days

Shrini

Hola SPAIN - QA&TS international conference and me ...

Hola SPAIN….

I was at Bilbao, SPAIN for a QA&Testing conference on embedded system – oct 18-20. I spoke about “Test case design and Automation” – a topic that I am working on since last 3-4 months. My talk and others in Test automation track were the ones that focused on “non embedded’ software systems. This paper was an initial attempt to explore the two big, complex and more of mis-understood concepts in software testing – “Test case design and Test automation”. I will be continuing to work on this topic and explore more about these. I was also given an opportunity to express my views about “Future of software testing and Challenges” at a round table discussion in the one the evenings at the conference.

I met few nice and interesting people in the conference – Paul Jorgenson, Scot Barber, Ray Arel, Doron. At lengths, I discussed with these people about various topics ranging from automation, Test design, Testing and automation in Embedded systems space.

Paul Jorgensen,(Author of the book "Software Testing A craftsman’s approach") - with his depth of experience in testing (both from Working Telecom industry and University experience), was a pleasure to listen and discuss. He gave me a patient hearing for my bugging questions on topics related to test design. His presentation on “All pairs testing” was one of thought provoking papers of the conference.

Ray Arell of Intel was very lively and quickly mixed up with group and I never felt that we were meeting for the first time. His presentation on “How to expand and improve your Test capabilities” was another great presentation of the conference. Ray is a highly experienced professional with about 21 years of exp and has to his credit a book on the topic “Change based Test Management”. Ray with his witty comments, kept the participants hooked up to him all the time. My discussions with him were related to testing and related practices in microprocessor/Semiconductor industries. We traveled together right from Bilbao to Bangalore.

Scott Barber (www.perftestplus.com) managed to be at the conference on the second day and was another interesting person to meet. Scott’s proximity with people like James Bach, Michael, Bolton, Cem Kaner, especially made me to spend more time with him and understand his current areas of interest. He is performance Testing Guru – gave lots of good tips and hints to me on the topics ranging from Performance testing, Test automation, Challenges in Independent Test consulting and future trends in Test automation – which prefers to call as “computer assisted Testing” (Which I agree with him). Thank you Scott - for all those valuable suggestions that you gave me.

One very pleasant side effect of my presence in this conference was ‘exploring” the beauty of city Bilbao. I am nature freak and love to hang around greenery, water bodies, lakes etc. Few locations in Bilbao provided me a perfect opportunity to be with the nature. I struggled with language at few occasions and that with the food – it was difficult to find a 100% veg food that I need. Conference organizers at every dinner and lunch – made special effort to some closest Veggy food for. They did great job at making this conference. In all, it was a very enjoyable and knowledgeable trip for me. I definitely look forward to participate in QA&Test Conference 2007 … bye Bilbao till then.

Bug or a Feature ?

Differentiating between bug and feature - more often than not is a result of someone (typically a developer or a tester) trying to prove some other person (typically again a developer or a tester)wrong. Somebody says "See this seems to be a bug to me - I trying to be like typical end user" where as somebody yells back "Look this is as per this document and no where it is mentioned that the feature should work like this".

In simple words, the difference between bug and a feature OR "Desirable" and "Undesirable" OR "Expected" an "Not Expected” - is with respect to some REFERENCE. What is that reference? Who defined it? How credible is it? The moment all the concerned parties and entities involved in Bug-or-Feature conflict agree upon this Reference - the distinction becomes very very clear.

In most of the cases - requirements document, market survey or some expert opinion is considered to be the reference. The confusion in most of times is because of lack of reference (Oracle) and still worse the lack of knowledge that “there is no Oracle".

If I find myself in such situation - I simply say "This is my observation on this feature. I think we should analyze and explore to see if anything wrong here. I would be happy to participate in this exploration. I would not get into issue of whether it is a bug or feature - I am not the right person to decide that. I only report on observations that I make with respect the features I test.

so next time when you hear an argument like this --- Just ask this question - "Can all of us first agree upon the reference?". I bet, you would sound lot intelligent in the crowd ....

Shrini

Are you making most of Test automation?

Test automation is a beautiful and a very handy concept. Making computer doing things while you watch all those things that you would otherwise don’t observe – can be a very powerful thing in Testing. Deploying Automation can magnify the reach of manual tests in lots of ways. Unfortunately, Automation in lots of places has been deployed with intention to replace (not supplement, enhance or extend) human Testing.

If you are using Test automation for following cases – you are making the BEST use the investment made in creating and maintaining automation suites …

1. Those tests that have a high chance of human errors, those features that under go minute changes frequently that a human eye is likely to miss.
2. Another extreme of (1) – Routine and mundane tasks like installation and smoke tests.
3. Frequently repeated, High volume repetitive Tests
4. For covering multiple platforms
5. Those cases where it would impossible to perform a test manually
a. Simulating some behavior that exposes a risk e.g. memory leakage


Less desirable or efficient Usages of Automation

1. Automating a set of low power manual tests just to cut down the cycle time of testing of those tests manually – without evaluation of those tests.
2. Automating every test that is possible to automate and aim at covering maximum of testing by automation – a notion of “more is better”

Shrini

Top burning issues in Software Testing ...

Top 10 Burning issues in Software testing ….

While thinking about asking a thought provoking question to make James Bach - write about it, I got this idea. Here is a post that lists top 7 (want to reach 10) burning issues in software testing today. I solicit other issues and thoughts about how to address them … Write to me, I will consolidate them post back on this blog …

1. External Issues

a. Business pressure (cost reduction, quick time to market, proliferation of computers –hence growing complexity of software systems hence growing complexity in Testing)
b. Tester’s place in overall Software Engineering eco-system (conflicting roles and responsibilities)
i. Developers
ii. Business analysts
iii. Sales and Marketing
iv. Stakeholders
v. End users
2. Project management
a. Predictability - questions like - when you will be done? How far to go? How much time it will take and how much does it cost?
b. Dependency on Business Analysts (specs) and Developers (code delivery) and Project manager expectations (deadlines)
c. Accountability issue – What if a bug is missed from testing?
d. Impossibility of 100% Testing
e. Testing resources are limited, added late in the project cycle
f. When development delays – Testing time gets chopped – Deadline remains – when the testing misses a bug – heads are rolled in testing.

3. Justification for Existence issue

a. Objective of testing
b. “Anyone can do” Notion
c. Testing as quality Gatekeeper
d. Awareness issue – how make others especially stakeholders understand value of testing.
e. Outsourcing

4. Hiring and Managing testers (Performance monitoring)

a. Skill issue: what is important skill to look for in a good tester – Technical knowledge, business domain knowledge, Test process knowledge (all folklore and legacy), Formal Techniques and methodologies in testing, Good learning and thinking capabilities and so …
b. Objective goals – How do you know tester has done a good job
c. Measuring the performance of testers by # of bugs logged
d. Notion of “Tester needs to have deep technical knowledge (not mere process stuff) so that he can get respect from development
e. Notion of “Testers needs to be a business domain expert “– Modern day interview question – “What all domains you have worked on “? Do you have any experience on “Telecom (that too billing) domain, health care, Do you have knowledge in Capital Markets? And so on…
f. Notion of “We need testers who can code”

5. Tools and mechanical Part of testing

a. Automated Testing (not Automated Test execution)
b. Regression Testing ( Repeatability Argument)
c. Testing is a branch of Computer science (yes or no?)


6. Philosophy of Testing Issue – what do you think a tester should do?

a. Handling Process and Metrics fanatics
b. Testing as Factor assembly line (test cases IN, Results/metrics OUT)
c. Find bugs – more bugs – better Testing
d. Prove that software works
e. Quality Assurance/Quality

7. Lack of Education and Skill development programs

a. General Awareness among the community
b. Problems with certifications
c. Formal university Programs
d. Research

Shrini

Ask James Bach - a question on Testing...

James Bach has a post on his blog inviting questions on Testing. This open invitation comes with a rider ... only *interesting* questions will be answered, rest will ignored and best ones will be *awarded* with James writing a whole new blog post on it. I am still thinking on coming up with a question that will make James to write a seperate Blog post - that will be a real question.

Just to remind you the commenting policies on James' blog --- Any comments that makes to his blog post (after moderation) are considered to be useful to the readers of the blog (as endorsed by James himself).In my opinion that is a like "treat" to me when my comment makes it to comments list.

So what are you waiting for ... Just grab the opportunity ... Ask James a nice question on Testing ...

A good bug report ...

Read this bug report for Mozilla Browser ---

https://bugzilla.mozilla.org/show_bug.cgi?id=154589

What makes this bug report a special example ...

1. Clearly explains the background of a bug - with example
2. Makes a strong case for Why bug is important from user perspective .
3. Persuasive enough for a stakeholder press for fix.
4. Examples of other sources that give references.

As Cem Kaner puts it -- a good bug always makes developers to fix it. If you have managed to draw attention of developers, PM and other stake holders - you have made a strong beginning - make the bug report look appealing

BTW, bugzilla.mozilla.org is a best place to learn

0. It is an open Bug database -- A huge knowledge repository.
1. Good bug reports - look for bug patterns - learn from them.
2. Know about security Vulnerabilities and Brower issues.
3. Learn and brush fundamentals of Web and standards that make "Internet"


Shrini

Test Automation - Takes toll of Microsoft Testers ....

I read this old story (Reported by Seattle times in and around Jan 2004) about Microsoft Laying off 62 Testers in Windows group.

http://seattletimes.nwsource.com/html/microsoft/2002155249_mslayoffs20.html

Because (as reported by Seattle times - www.seatletimes.com)

1. They had automation so testers not required.
2. They need to cut cost - either send jobs to India (low cost option) or aggressively automate...

It is pretty sad to note that a company like Microsoft (I am an ex-Microsoftee) is taking step like this. Conventional wisdom and all classical/contemporary literature on Test automation makes it clear that "automation cannot replace human beings and human part of testing". I am at loss to understand why Microsoft (some groups in MS) thought that automation can replace Testers.

This is a story published about more than year ago and is not an official communication from the Redmond based Software giant.

But The Seattle Times, being the largest daily newspaper in Washington state and the largest Sunday newspaper in the Northwest. Well respected for its comprehensive local coverage, The Seattle Times, winner of seven Pulitzer Prizes, is also recognized nationally and internationally for in-depth, quality reporting and award-winning photography and design.

I am afraid it sends wrong signal -- Microsoft should have (might have) done something to set this right...

Anyone listening?

Shrini

James Bach on automation ...

James bach has posted following blog post on Manual Tests and automation.

http://www.satisfice.com/blog/archives/58

It is prety interesting stuff. Read the post and my comments on that post.

Especially Rules of Automation as per James

Test Automation Rule #1: A good manual test cannot be automated.

Rule #1B: If you can truly automate a manual test, it couldn’t have been a good manual test.

Rule #1C: If you have a great automated test, it’s not the same as the manual test that you believe you were automating.

Note the comments chain for that blog post --- Really thought provoking

More on this later ...

Shrini

W3C Markup Validator ...

Michael Bolton (www.devlopsense.com) mentioned about this tool while we were having lunch together at Toronto. I was looking for ways to assess, report and hence improve the Testability of Web applications and hence improve automatability. I found this to be something that is close to what I was looking for.

http://validator.w3.org/

Using this tool we can verify the web pages of the application against W3C and other Web standards.In my opinion, confirmane of web applications to standards like W3c are useful from following aspects

1. Application upgrades and future enhancements in platform and core technologies - will become less painful and provide cost advantages. For example - using new web/application server, supporting new mobile platform, Technology upgrades in J2EE and .NET.

2. Improve Testability - This is a big issue in Automation. If the applications that are candidates for automation are not built for testability ( simple things like having unique IDs for gui controls and Windows so that automation tool can recognize them) - automation will be difficult and will cause lots of custom code to be written. At the end in terms of both development and maintenance of automation solutions for web applications.

In addition you can also find other free tools at

http://www.w3.org/QA/Tools/

shrini

Some web security related stuff

Information about 3rd party cookies

http://www.mvps.org/winhelp2002/cookies.htm


Using hosts file to block third party cookies

http://www.mvps.org/winhelp2002/hosts.htm


General Secutiy Issues in windows and IE

http://www.mvps.org/winhelp2002/security.htm#Firewall