10 Commandments for Test Automation Outsourcing

Most of my test automation experience and learning comes from working with IT groups and GUI based automation using COTS tools. Outsourcing of testing is a top agenda on today’s IT manager and automation happens to be one of the popular and “most frequently discussed” item.

I am regularly asked to formulate, present and consult automation projects in IT space. Based on this experience, I am attempting to formulate commandments - about 10 of them. I believe these are few considerations that an IT manager who is thinking about outsourcing test automation work.

Automation is not an answer to your testing problems such as limited testing bandwidth, limited time to test, poor application quality etc. These are true testing problems – one way to address them is to set your “manual testing” right then think of Automation. If your manual testing is poor, throwing automation in that will only cause that “poor” testing be completed quickly.

Automation is White Elephant While automation has clear benefits only when it is treated carefully. Automation is not a turnkey, as your application undergoes changes; automation solution needs to be maintained. Plus there are recurring tool costs, training costs and test case/project management over heads. Is your vendor telling about this sufficiently? Is your vendor downplaying this aspect? Watch out …

Anything that is quickly creatable – is quickly perishable Do not believe claims of “script creation in minutes”, “Automated script generation” etc. Any enterprise level serious automation is software development and requires clear attention and methodical approach to design/create and maintain.

Your business users can not create and own Automation solutions Let your business users/SME’s and domain experts do what they are best at – business support. Do not believe claims of business users creating and maintaining automation solutions. It requires good amount of testing and automation knowledge.

Judge your vendor by the questions they ask about automation Suggesting an automation solution requires thorough study of context of project, release cycles, current state of project, future expected life of application, business expectations, state of automation practice etc. Vendors who propose a “standard”, “gold plated” automation solution without asking questions to probe the context are “simply” selling you something – be skeptical about such vendors.
Higher % of offshoring in automation higher investments to make While moving work to offshore will give cost benefits, when it comes to automation there are number of factors to be considered while deciding how much of automation can happen at offshore. Key thing is about creation of application environment at offshore. If a local application environment can be created at offshore with tool licenses – higher degree of automation work can happen at offshore. It is important to note that for GUI centric/COTS automation tool based automation, both automation tool and application should reside on same machine. Hence pay attention to your application environment and feasibility of creating local environment at vendors offshore location before deciding the amount of work that can move to offshore. Even when such local environment is available at offshore, certain activities like acceptance testing, demo to users, and certain types of test cases will need to be done only at onsite. Be sensitive to these factors. Make sure your vendor asks about this and suggests suitable alternatives.

Pay attention to dependencies and Quality of current test Artifacts A successful outsourced automation requires that project dependencies are well understood by all stakeholders. Access of application from offshore, access to testers/developers/SMEs on test cases, Access to reviewers and code acceptance people are few dependencies that one needs to track as part of project. If automation is on the basis of existing manual test cases- make sure that these are detailed enough and available in a form that can be sent across to vendor offshore team.

Decide Acceptance criteria Formulation of Acceptance criteria is not item that is often given least importance while planning outsourced automation projects. Identify an internal owner in your organization who will accept the code/solution delivered by the vendor. Make sure that this person gets engaged in the project from the beginning and formulates the acceptance criteria along with vendor technical team. Failure to identify automation acceptance person from your end and getting a formal agreement on acceptance criteria can leave you “High and Dry” and leaves an open space for the vendor to deliver any “working” automation code but not necessarily the one that stays for long time.

Avoid linking an automation project (and its deliverables) to application release dates one common mistake committed by IT groups outsourcing automation is to link the automation deliverables to immediate ensuing product/application release and cut down time/effort for manual testing. One IT manager said to his project team “We are having a major release for application XXX in January and we have automation solution coming from a vendor by December. Since Vendor has promised that 90% of manual testing will be automated – let us plan to allocate 2 weeks of testing instead of 10 weeks planned earlier. As promised by vendor we can deploy automation and cut down testing by more than 50%”. What is the problem here? What if Automation delivery slips for the reasons beyond control of everyone? What if development delays? What if due to poor test case quality and fast changing application, automation scripts give inconsistent results? This will result in conflict - “believe automation” or “get some good manual testing done”? Such conflicts can severely impact your releases and hence business plans.

Record and Playback (RP) Automation is for Kids This one point can never be overemphasized. Many IT managers still feel that record playback features of industry standard automation tools can help them to create automation quickly and thereafter their own resources should be able to record and create/maintain automation scripts. However, the experience has shown time and again that RP approach is not beyond learning about how automation tool works with application and can not be used for real time, sustainable automation. If vendor proposes this as a part of the solution – you should be alert and suspect the abilities of this vendor to deliver automation solution

Do you agree with these commandments? Any different experience?

[update]
Some additional tips :
What a Vendor should ask you
- questions about your manual testing practice
- Your objectives of Automation and expectations
- Readiness to take up automation in terms of test cases, application state and environment
- Ask about your expectations on ROI

What vendor should suggest you
- Plan for automation maintenance when supplier is gone.
- Automation may or may not reduce cycle time – that depends upon nature of tests, application technology stack, nature of tool etc
- Automation may or may not reduce the cost of testing.

What to look for in an automation proposal
- Acceptance criteria
- Automation design details – how tests will be structured
- Environment related assumptions
- Pre-requisites about tool licenses, test cases, test data, access to developers, testers and business users (for clarifications about test cases)

You can not automate testing – all of testing as testing is intrinsically a human thinking and investigation activity. What you in reality claim to automate is some portions of testing – namely “test execution” of some select test cases. Note that activities like test design, bug investigation and logging, Test results verification etc are still to be done by a human tester. Automation will take you some places but not all. Unfortunately, the places where it does not take you – are the ones where real problems lie – only a skilled human tester can help.

Shrini

Automation reduces cycle time Part III

I wrote about it earlier here and here

“If automation can not reduce cycle time, save testing cost and reduce manual testing errors, why do automation at all” said a colleague of mine in a discussion over mail chain yesterday.

After all, automation and test cycle time (whatever may be the meaning and definition) are so “inseparable” – the theme often evokes deep emotional, business, technological thoughts. While I agree with my colleague on the last item he mentioned “Reduce manual testing errors”, for others I would say they are true under a highly restrictive and idealistic conditions. This is similar to the practice in automobile engineering of specifying mileage of an automobile under “standard road/test conditions”. Such claims are made by automobile manufacturers to push their brand as the fuel efficient vehicle. There also goes a disclaimer that says “Actual mileage may vary depending upon the prevailing conditions on the road, driving habits etc”.

This is very true with respect to test automation too. While in ideal conditions and specific types of testing, automation can help in reducing “test execution” (not testing) time, in reality actual benefits from automation may vary depending upon the project context, state of development and testing practice and others.

I see a future where Automation tool vendors are forced to add a disclaimer to protect them from getting sued by someone who just purchased the tool expecting return as claimed in sales and marketing materials of the automation software.

As I mentioned here, testing cycle time is a complex variable that depends upon several parameters and many of these parameters are out off control of automation and even testing. Establishing a straight linear relation between automation and testing cycle time, according to me is a “horrible” and “unrealistic” simplification.

What is (are) the (fundamental) problem(s) that in testing, the automation is expected to solve?

I am afraid, there are very few testers or test mangers out there that begin their exploration into automation with this question. For many, the notion of automation appears to be well understood I am still struggling to answer this question in a context free sense. How can there be a universal solution to a vaguely or incompletely specified problem?

Dear reader, what do you think are the problems that automation is expected solve?

Shorten test cycle time? Reduce manual errors? Help to kill boredom and fatigue due to repeatability? Save dollars of testing cost? Bring consistent testing results? To supplement human testing capabilities?

To be continued …

Software - A game of questions and answers

"The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question."
— Peter Drucker

"Testing is a questioning process in order to evaluate software" - James Bach

"Computers are useless. They can only give you answers." - Pablo Picasso

"One who asks a question is a fool for five minutes; one who does not ask a question remains a fool forever." - A Chinese proverb

Other day I was discussing with one of my colleague … somehow our discussion went about interpreting in simple terms the whole “game” of software development and testing. Here is what and how ended up in agreeing on the “simple” model to describe software and software lifecycle …(not SDLC but SLC)


Software development is about coming up with answers (and demonstrating those answers with an example) to the questions raised by testers, end users and other stakeholders.

Software testers ask questions about claims and capabilities of what software is supposed to do, take the questions to developers and project mangers and ask for answers.

Project manager or project sponsors scan these questions and pick up those that they think are worth “answering”, prioritize them and pass on the developers for providing answers and ways to demonstrate the answers. Before releasing the software to testers, developer do some question-answer session with buddy developers and leads (peer testing, unit testing and code reviews)

Developers then get on mission to analyze questions and develop/construct answers in the form of capabilities in the software and “release” to testers to check to see the answers are “satisfactory”. When developers do not get answers or feel that it takes relatively long time to find the answers – they turn to project manager with their analysis as why answer can not be made available immediately. Project manager then takes the decision of “deferring” those “unanswered” question to be taken up in future releases.
At times Testers, act on behalf of end users and other

Testers verify those answers and check to see they are OK … some times there will be follow-up questions or new questions (regression bugs/issues) and they are routed to developers via the project manager. This cycle repeats until there are new questions to be answered by the developers.

So … as long as there are questions to be answered about software … there will be the need of developers (who will provide answers) and there will be need of Project managers (to prioritize and check which questions need to be answered) and hence a software development project …

Guess what … it is software testers who drive the whole thing by asking relevant and important questions about software – about it’s claims and capabilities …

So … as important trait of a tester is to practice asking “good” questions …

Shrini

Software - A machine or an organism or ?

“The first obvious difference between machines and organisms is the fact that machines are constructed, whereas organisms grow. …

Whereas the activities of a machine are determined by its structure, the relation is reversed in organisms - organic structure is determined by processes”

This is how Fritjof Capra opens up chapter 8 of his celebrated book “Turning point”
http://www.mountainman.com.au/capra_1.html

How do we understand software …? As a machine or an organism? Does software grow? How do we understand software by it structure or by observing (!!!) it behavior? What are useful models of software that help us to understand?

What kind of thing is this software? One thing is sure … it is not just code …

A general System …!!!! It is code and about various connected systems … can you think of all those systems connected with software system or systems?

Shrini

Are all best practices "worthless"? Testing Best Practices

Other day I was quoting following from Jerry’s new book on testing to one of my colleague who is a “best practice” proponent.

…..The risks in these two situations are vastly different, so do you think I recommended the same testing process I used for finding a personal web-writing application? Do you imagine I recommended that my client install random freeware pacemakers into live patients until he found something he liked, or didn't dislike? Why not?

I took above sentences as reference and told him.. “Can you use software testing strategy that one uses for web application writing to that of an embedded software in a heart pace maker? Hence best practices are such a junk thing ...”

To that he was silent for a while answered --- I agree with your point that test strategy or approach used for web application cannot be applied for embedded software in pace maker … How about picking the practice from a same field/domain – will that not save the time, energy and effort for my client ? Let us say I develop a list of practices for a given field (embedded software used in human bodies) and keep “selling” them as best practices (jump start kit) for those clients who deal with such software? What is your opinion? Would you still say … best practices (in a context) are junk?

I did not have a good answer for him …. Then we discussed about “universal best practices” (I am not sure if such phrase exists as all best practices are universal in nature by default and context less??) such as “walking is good for health”,”Test considering end user scenarios”, “Do unit testing” “Do code review”, “Aspirin is good for heart”, “Drunken driving leads to accidents”, “Do meditation to calm your mind” etc. I told him about at least 3 contexts for each of these best practices where following best practices can lead to harmful effects.

After listening to me … he said … Shrini … you appear to be "making up" all these contexts to prove your point …I want you to answer my question – Are all generic best practices recommendations are worthless or fake? When customers want something readymade that will help them to jumpstart the work, they would like to see if I, as a consultant, can bring some “best practices” from my previous similar experiments. Is that expectation unreasonable?

I am thinking ... I don’t have a good answer for him … do you? I hope Jerry would have some answer …

Are there any "universal best practices" or by default all best practices are universal and context free? Will a best practice cease to remain as bet practice once it comes with a context?

[update] Quoting from Jerry's book again - "As humans - we are not perfect thinkers, we are affected by emotions and we are not clones. We are Imperfect, irrational, value driven,diverse humans - hence we test software and test our
testing AND hence test "best practices" that sales and marketing folks associate with software testing.

Shrini

Exploratory Testing SHOCK ....

A colleague of mine other day expressed his struggle to make exploratory testing work for his time (scalability and making it as a best practice !!!) . He said "Exploratory testing is HIGHLY person DEPENDENT - that is the biggest problem for me ... Do you have any process document for doing best exploratory testing. I will have that included in our testing process framework. BTW that will help us in earning some kudos from our CMMI level 5 assessment team."

I said --" ... that is true ...why exploratory testing ... any good sapient testing is "person - human" dependent. A good testing requires thinking and inquisitive human mind. Are you planning to get testing done by machines, robots - so that your person dependency goes away? If yes .. kindly ask the CMMI team to order few robots for a Testing POC (proof of concept) "

He could not answer for a while ... then responded with low voice "I know you will say something like this only ... I have an answer… Automation!!!! I have raised a request for buying 10 licenses of this # 1 tool in the test tools market .. Howzzzzzzzzzzzzzzzt ?"

Now it is my turn to faint ....

Side Effects of Metrics/Statistics

Jamie Dobson writes this piece of "reality" with respect to statistics/metrics and numbers.

“... that human beings will always work toward their defined success criteria."

True and very revealing for all metric enthusiasts. Just let people know what they will be measured on - they will modify their work pattern and output to suite positively on the measurement criteria. For example, a test team is measured on number of bugs the team logs, you will see more and more bugs and if a test team is measured on number of test cases they execute, you will see testers executing increasing number of test cases.

One thing that happens is what I refer as "goal displacement" - Goal of doing “required” work getting displaced by doing *that* work in *that* way as described/interpreted by measurement criteria. Can you see a problem here?

When working with social/cultural setup involving human beings, introduction of "monitoring/measuring", typically causes a "shift" in a overall behavior of the group towards "what is being measured" instead of "what is required". We tend to believe that people behave the same with or without a measurement system in place – we are wrong

This is the side effect that I am referring when a metrics program is introduced in a software project setup....

Are you aware of this? What steps can be taken to address the side effects?

Shrini

Software Testing certifications Part II

Dr Cem Kaner posted a note to Software testing yahoo group on the topic “software testing certifications”. I thought, it would really make lots of sense and value to a discussion about software testing certifications to share those views here.

Dr Kaner quotes following in his note.
http://www.channelinsider.com/c/a/Careers/VARs-IT-Certs-More-About-Marketing-Less-About-Skills/

Continuing my thoughts on certifications, here is something that I would like to add … on the basis of Dr Kaner’s notes.

1. Certifications have value as “marketing” aid and most confuse them to be as means of getting knowledge or experience or learning.
2. IT organizations and service providers use their “certified staff” as “proof” of their well trained staff to their clients.
3. Certifications do have place in hiring. Whether you like it or not, organizations still use certifications as main filtering mechanisms in hiring just like a college engineering degree.
4. Certifications matter for those who are in the initial stages of their career especially those who are looking to get their foot in testing field. Most of the time, certifications get them a call to the interview.
5. Certifications can get you an interview call, might even get you a job but there after it is your skill and work that “keeps” you on job. Do not mistake certification for life time warranty for the job.
6. One very common argument in favor of certification is that “certification help in knowing the testing vocabulary” – This is true to some extent. But while going for certification with this objective, keep in mind that - there are no universally accepted authorities that define and mandate testing terminologies and terms and practices vary across the board.
7. Certification enthusiasts claim that certifications are means of learning and gaining knowledge in the subject. WRONG … there are better ways of studying and learning than going for certification
8. With the help of internet, thanks for Google and other search engines, today the information is everywhere, just look around you can learn lot and gain knowledge by effectively searching the web, reading blogs, writing blog and engaging in conversation with other in the community.

Now about let me talk about those certifications that hold value in today’s world

• CISSP (Certified Information Systems Security Professional] To earn a CISSP, candidates must have five years of experience and endorsement from any professional certified by (ISC)2, the organization that awards CISSP certifications.
• CCNA (Cisco Certified Internetwork Expert]) – this one especially my favorite as it requires the candidate to demonstrate the knowledge as part of the exam in a lab environment – e.g fixing a faulty router.

To summarize, certifications tend to be of value for some (hiring managers and new entrants) and there are some examples of good certifications that test the skill of the candidate. Do not confuse certifications to “learning” and “knowledge” – most of the current software testing certifications are to be used as “marketing” tools.

What if Automation finds bugs ....? Good thing or bad thing?

Ryan in response to my post on "cycle time reduction and automation", mentions that "It is an accurate statement that automation will not improve cycle time if it finds bugs, where bugs would have otherwise gone undetected. However, I believe the claims automation vendors make, is based on the fact that manual testing is also going to uncover those bugs."

So, when automation finds bugs, your cycle time increases. When people claim cycle time reduction, they "overlook" this fact. why? Those bugs will be found by manual testing too. This may or may not be the case. Bugs discovered by human testing and Automation tend to be of different types.

Now, let us track that trail of what happens a bug is discovered -
In automation - situation could be bit tricky especially when the automation tests, logs are bigger. An error/bug reported by an automation bug needs to be checked to see if it is a bug in automation code or a bug in application or bug in data setup or some timing or synchronization related problem (in GUI automation scenario). Let us say you have 5-7 pages of log file - you will have to scan/read through the log file an locate the bug. You might have to do execute failed automated test manually (and corresponding data setup etc).
In manual testing, human tester can easily trace and follow the bug trail and document the bug. At a high level, bug investigation and isolation tasks tend to become relatively low.

Hence, when automation discovers a bug - things get really problematic.

If one were to cut down cycle time by automation or otherwise, they HAVE to make sure either "no bugs are discovered" or "any discovered bugs are IGNORED" or "bugs that are discovered, if fixed, not tested again and other regression testing is done ....

Can automation control or influence any of above events - prevents bugs being discovered or igonore the bugs if accidently discovered or mandate that bugs fixes will not be subsequently tested?

For the sake of argument, let us suppose that both human test cycle and Automation find same number of bugs ... and take out "bugs" portion of test cycle, how automation can save test cycle time? On what parameters this cycle time reduction by Automation depends ?

Type of test - nature of interactions between "test execution agent" (human or an automated script) and nature of verifications (during and post execution).

• GUI Forms with lots of data input fields - can result in quick form fill tests when automated (zero think time).
• Tests that require longer processing time can not gain from automation as automation can not speed up the processing time.
• Tests that require visual inspection - window titles, error messages, color, tool tip and other GUI centric elements - are better tested manunally as programmatic tests would mean lots of investment. Human testes are quicker and cheaper in such cases.
• Result verification that requires detailed analysis, text file processing, database check etc are good candidates for gaining cycle time.

Thus, there are parameters that are beyond the reach of automation ... hence the notion of cycle time reduction has to be really, really taken with "caution".

Shrini

A catalogue of Test Automation Benefits ...

Other day, some one asked me “in your opinion, what are the real benefits of automation”. That triggered ideas for me to write this post… Let me attempt to consolidate and list all the benefits people claim around “automation”

Real: In my opinion, these are “real” and achievable benefits.

- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency
- Quick – No think time while running tests. (Computer program does not think – So not useful in those cases where you need to think as you execute. How does automation program help you to gain speed when thinking is required?)
- Supplements human ability to spot software bugs
- Helps to run big number of data combinations – can test robustness
- Helps in multi platform combinations (OS/browser/database and other application setups)
- Repeat the testing (test execution) done for configuration A in Configuration B also.
- Hence increased Test coverage
- Following non test execution tasks
- Generate and manage special set of test data
- Large Volume Test comparisons
- Automated workflows and alerts
- Environment setup

Fake: These benefits are on transition point – where the focus slowly starts drifting from “real” to “imaginary” and hence “fake”. These are false promises that are realistically not possible.

- Improvement in application quality (automation can not improve application quality – even testing can not … only developers and business analysts can)
- Improvement in Test process (Test process is pre-requisite for automation)
- Non technical people can do automation – no programming language required
Improved test planning (not sure how)
- By executing a set of test cases for a specified number of times – ROI from automation can be realized. Say in 10 executions –automation test pays for itself after that is “cost saving” all along


Conditional: These are the benefits that are realizable and could be reasonable but under a very strict set of conditions – “once in a while” cases. When these benefits are stated, mostly the conditions that must be fulfilled to realize these benefits, are not stated. This makes these benefits look as though there are real and universally applicable

- Simply – saves time.
- Test effort reduction
- Improved Time to market
- Improved test productivity
- 24x7 testing possible
- Knowledge retention


Totally outrageous: There are some really outrageous and are more or less like a typical sales pitch. People, who make and believe such claims, do not seem to understand human side of testing, testing itself and automation. They are just believe that machines are better than human testers.

- An automation test execution is equal to an hour of skilled human testing
- Automation can replicate human interactions
- Reduced dependency on human testers
- Solves problem of resource crunch and less time available for testing
- Reduced defect rate – fewer defects


Anything I missed?

What is a bug ... A new meaning ...

Let me give a try to this one-liner ...(short post)

"A software bug is a reflection of the mind of a confused human user"

Analyse this statement .....

[Some updates]
Above one-liner proposed by me, seem to have generated interest in some ... Let me clarify further ....
This one liner of mine is a beginning of an effort to understand human thinking process while he/she sees a bug. A human goes through a series of emotions while dealing with a software bug. A dominant emotion among these is "confusion" - state of perplexity, chaos, uncertainty.
Let us say you are running a test, observing what is happening and doing a quick comparison with what were expecting ... Suddenly something "unexpected" happens, your mind starts remodelling all that is happening, you were expecting "x" to happen, where as you are seeing "a", "b" and "c" are happening - that is contradicting your model .... your heart beat goes up, blood pressure goes up ... lots of physiological changes happen ... your brain tries to stabilize, create a new model on the fly, comprehend a things on the ground and after a while, you calm down and begin to put up an explanation of what is happening and why? A bug report comes out at the end ....

So Bhargavi --- the life cycle of a bug begins with a "confused" mind of a tester. A confusion is a mental state where you are not able to comprehend and reason to the situations and problems that you are subjected to (imagine a traffic cop at traffic jam on road, a kid on first day in a new school etc). It is the confusion that triggers the thinking in the mind of tester, tester follows the trail of the thinking and builds up a new model and finally when is mind is settled, comes up with a explanation. So when a bugs gets nailed down, mind becomes calm.

Even an obvious bug - when you look at it deeply is a result of thinking process triggered by "confusion" what is expected and what is observed. As a normal human reaction to a state of confusion, you reason out the things and make them clear - then bug becomes obvious ...
If an obvious bug is at level "0", confused state is at "-1" level. Also the "sense" of confidence is always followed by a subtle at times quick state of confusion. When try to open your car door in wrong way, open your cupboard keys in wrong position, when you are trying to open a door by pulling that opens only on "push" mode ... our mind goes through a quick state of confusion ... you quickly notice the situation and gain your calm.

It is deep cognitive process .... Psychologists will be able to explain better ... "Psychology of a BUG" ?

Shrini

Goal of Testing and a quotable quote

Michael Bolton in response to this post from Steve Rowe mentioned this gem …

“Do our automated tests take into account the notion that different people might value different things, and that one of the tester's primary goals is to recognize different constituencies and the ways in which their values might be threatened? “

Many people in our community think that as testers our goal is:

1. To find bugs
2. To prove that the application "works" as per the specifications.
3. Run a bunch of tests and report the results,
4. Develop some automation to speed up the execution of tests

But to “analyze”, “investigate” the value systems/perceptions of different stakeholders of a software product and explore various possibilities where stakeholder’s value is threatened. This involves among other things – finding some bugs, running some tests, writing some automation etc. Note that End user or customer of the software is an important stakeholder.

This is statement from Michael is an extension or probably logical conclusion of the famous statement that is often associated with testing “Be customer’s advocate” (or “think like customer”). Michael seems to suggest that it is not only customer whom we should consider, as testers we think about all stakeholders and explore what each of these stake holders value.

A stakeholder is a person is who is affected by the success or failure of a project or the Actions and inactions of a product or the effects of service – Cem Kaner

In one of the comments for the same post, I found another worth quoting statement from Ben Walther …

“There will always be inputs into your system that violate the assumptions made in building it. A computer generated test will not be able to violate such assumptions.”

Very true … that is why testing is so challenging and exciting

Shrini

Mission - Test Estimation model

It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.—Fred Brooks

Steve McConnell opens his famous book "Software Estimation: Demystifying the Black Art " with above statement ... Development community is fortunate to have some one like Steve McConnell to help them with solving this puzzle called "Estimation"

I am working on developing a test estimation model. I think in today’s testing world, this is a biggest and the most complex testing problem to be solved. As I am reading and researching about this topic and formulating my initial thoughts, I am thinking about testing, testing models, questioning, thinking, modeling, bug investigation, test-stop criteria, non-linearity, size of testing work and so on.

Following is a list of challenges/questions that I am searching answers for …

1. Model of Testing: Testing is an evaluation activity as against Development that is a construction activity. How to think about estimating a “evaluation/Analysis” based activity?

2. Sapient Testing model that requires critical thinking, questioning, modeling is essentially a non linear activity where as development can be a relatively bounded activity of starting from a spec to a working code. For example if a particular testing task takes x hours, 10 such units will take 10x hours … we can not make such extrapolation right?

3. Sizing testing activity is still a big problem – what is the unit of testing task? Atom? Or molecule? What is the building block?

4. Tasks like bug finding, investigation, test design are difficult to quantify?

5. When do we stop testing? What are exit criteria for test cycle? This will set upper limit for testing scope?

6. How many test cycles we need? How do we estimate?

7. What are the human factors involved the process of test estimation?

8. How do we address problem of "slipped bugs" in case of plain and straight forward "scripted testing" (write test cases and execute them word by word and log the bugs and end of the cycle). What about bugs or tests that come out of specs, exploratory tests? (Something that we worry heavily in IT services Industry – how much testing to do? There are penalty clauses for missed bugs)

9. What about productivity numbers used. In estimation -> we first talk about sizing the work, then apply productivity figures to arrive at effort then split the effort into schedule. How do you deal with numbers like – 10 test cases executed per person per day? Should you believe them or question what a test case is?

10. Is estimation a guess work?

11. Is estimation similar to the work of a fortune teller, weather forecaster, Election analyst, Stock Market analyst, A punter in horse race, or a gambler? After all in test estimation we tend to predict the future right?

Any views? Are these questions important while arriving at a test estimation model?

Interesting quotes on Prediction:

"Those who have knowledge, don't predict. Those who predict, don't have knowledge.” - Lao Tzu
“Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” - Peter Drucker

Shrini

Can Automation reduce cycle time or improve time to market?

Continuing on this discussion on test effort and manual testing – there is another popular variation in which Automation tool vendors claim “improved time to market” and/or “reduced cycle time”. In this post, let me dissect these claims and see how true and credible is this claim.

First of all let freeze what one mean by “Time to market” and “Cycle time”. Let me define the terms as bellow in the context of a traditional/Waterfall model type of software development.

Time to market is a time window between the time your start development (requirements and so on) till you ship the product in market for general public consumption. Depending upon whether you are doing a major release or minor release, this window may span from few months to few years (as in the case of Windows VISTA)

Cycle (when used without any qualification indicates a cycle of development and Testing) time is a time window for a software product under development. The cycle time can be divided into Development time and testing time. A development cycle starts with deliberation of requirements, design of features to be implemented. A test cycle starts with a development team “releasing” the code for testing to begin and test cycle ends with test team completing the planned testing for that cycle. During this period development team can fix the bugs reported. Hence for all practical purposes, a cycle time implies a time window between start of design/requirements until test team completing the testing/development team is ready to release next build.

So it is apparent from above definitions that cycle time is a subset of time to market.


Automation can reduce cycle time (less of a problem) only if (check how many items or situations that “automation” can control)

• Automated tests run without reporting ANY bugs on the software (A bug reported by automation means – some investigation and confirmation by manual execution that the bug reported by automation is indeed a bug with the application)
• Automated tests DO NOT report any run time errors (A runtime error by a automated test means some investigation and re-run.)
• Development team INSTANTLY fixes any bugs reported by Automation and these fixes are so well done that there will be no further verification (manual or automated) required
  Manual testing (a very small portion indeed) that happens in the cycle does not report any bugs. All the manual tests pass.
• If manual testing reports some bugs those bugs will be fixed INSTANTLY without requiring any further verificationBug reporting time, triage, investigation (if any) is so small that they are negligible.
  

Automation can reduce/Improve time to Market only if –

• All the items mentioned under “Cycle time” and
• Business stakeholders do not worry about outstanding bugs. They take the decision to ship the product as soon as Automation test cycle is completed (because automation cycle is NOT expected to report any bugs). So the end of the automation test cycle, shipping the product is a logical thing to follow.

If you analyze these situations, you would notice that many of the factors that influence cycle time or time to market are not under the control of “test automation”. These factors are to do with development team, quality of the code, quality of requirements, number and nature of the bugs reported by both manual and automated test execution and above stakeholder’s decision about those bugs reported. One can not claim that there will be cycle time reduction or improved time to marked JUST because x% of test cases are automated. A big and unrealistic generalization - only an automation tool vendor can afford to make.

So next time when someone says automation reduces time (either cycle time or time to market) – do quiz them and say “What do you mean”?

Bonus Question : Can automation accelerate your Testing? If yes, under what circumstances?

Next Post : Can automation address the IT problem of Limited (human) resoures and tight deadlines?

Shrini

When method dictates Goal – See Goal displacement in action - II

I wrote about Goal displacement part I here. Surprisingly .. no comments yet ... :(

Here is example # 2

Software Testing Certifications: A conversation between a certification enthusiast (CE) and a critique (CC – that is me)

CE : Do you know about this certification for software testing, this is very popular in Europe and US?
CC: Yes, I do but I am not sure if this really helps in evaluating skills of our testers.

CE: I think it does. I have heard about exam it seems pretty exhaustive and coves all aspects of testing.
CC: Umm … what do you think the certification is testing? What and how does it evaluate testing skill of the candidate?

CE: Certification tests the “knowledge” of the tester, familiarity to terms, definitions and experience of the candidate…
CC : Really? How? How a certification tests knowledge and experience of the tester?

CE : By carefully selected questions and evaluation by testing experts. The certification exam is based on a body of knowledge and questions – mostly objective type. I think one can rely on the exam.
CC : So … you are saying an objective (yes/no and multiple choice) type question paper is used to test the skill and experience of a tester. Does the exam allows theory type of questions? Can the tester debate on an issue? Does the exam involve any kind of putting the tester into real testing situation? Does it observe the tester in action ?

CE : Come’on, how it is possible? Certification bodies have their limitations, they can not set up “practical exams” to watch tester doing testing and then rate. Do you want certification body to set up a audio/video facility to allow for debates, questioning and real time situation simulation?
CC : Don’t you think, that you would right and reasonable way to assess the skill of a tester not using a fixed set of questions that emphasizes on memory recall, reproduction of text of study material?

CE: That is right … Look at feasibility of having such exams … what about cost of administering such tests? What about evaluation? It would be costly. That is why the certification bodies might have created a scheme of tests that are easier to evaluate and conduct on mass scale. This will enable a relative cheaper exam and allow more people to take the exam Right?
CC: Well …good point. But what is the goal of certification exams? What they attempt to achieve?

CE: Evaluate and Assess skills and experience of a software tester.
CC: But your current exam seems to be structured in such way that it is easier to administer and evaluate.

CE : Ummm… that is correct.
CC: See this is what I call as Goal displacement. Certification bodies wanted to design, administer and evaluate a system of exam to evaluate and assess skill of a tester. But they seem to have taken the path of designing an exam system that is easier to administer.
The goal of software testing certifications is to act as a mechanism of evaluating skills in software testing and provide a benchmark for the talent in that space. As the popularity of such certifications grow, so grows the need for system for mass administration and evaluation. This causes the change in the examination pattern and evaluation mechanism to facilitate the mass administration. Any part of the exam that is good from skill evaluation perspective will be dumped if that does not lend itself to easier evaluation of exam results.

Side Note:

Software estimation models (especially test estimation models) often suffer from Goal displacement problems. Other day I told my colleage " The reason why we struggle with test estimates is that we use simplified model of testing while estimating but reality is different hence estimates are typically go wrong". For that he said -- "OK .... we know that actual testing models are complex (non linear and involve critical thinking/Questioning etc). That is why we use simplified models of testing ".... How strange ..!!!!

This is another example of goal displacement right away ... To make estimation possible - change the testing model itself ... use a simple one. So the goal of using a good and resonable testing model that happens to be complex (and does not lend itself that easily to estimation) is replaced by a goal of finding a testing model that is simple and easier to estimate ...

what should be your goal - good testing or estimation?

Shrini

Can Automation reduce human testing effort ?

This post is for all those IT managers who are considering outsourcing testing and listening/talking to various vendor presentations on testing and automation.

I was in a conversation with a client other day. He asked me “Can you reduce the number of resources that you have deployed currently for regression testing of this application to say by half in next six months or so by using “Automation”?

I was not shocked to hear this since for past few years, I have seen many clients in IT application testing space have had similar expectations out of outsourced testing and hence on automation. In an Outsourced IT application testing domain, this is how a service provider positions Automation – A means to cut down the cost of testing (resources).

Client’s expectation of reduction in testing resources (with no reference to scope of testing) using automation in a specified time frame stands on following assumptions or beliefs.

- Regression testing is simply executing a set of test cases and reporting results nothing more than that.
- Regression testing gives the confidence that nothing is broken – all that was working previously is “intact”
- Automated test cycle means zero human involvement hence percentage of automation should directly result in proportionately reduction in human effort of testing. For example if 50% of test cases are automated, effectively manual test cycle time can be reduced to that percentage.
- Automation is turn-key. Once you have an automation solution developed, one can just keep using it unlimited number of times without any extra cost and effort
- Operationally, automated test cycle ALWAYS takes less (fraction of manual test cycle version) time – there would be no additional effort required to investigate results and chase script failures.

Let us look at costs of creating and owning automation (note that majority of these are of recurring types)

- Tool evaluation, proof of concept and other initial investigation costs
- Automation tool cost and recurring licenses/upgrade costs
- Resource Training costs
- Costs associated with automation environment (development and execution) – servers, applications, connectivity etc
- Costs associate with manual test cases cleanup, rework, test data, clarifications and any other effort to facilitate automation
- Automation development/ testing/review and acceptance costs
Costs of setting up and maintenance of automation execution environment
- Costs of investigation of automation results, failures
- Costs of re-runs (testing automation code fixes) and any required additional human testing
- Costs of maintaining automation code – design and structure of automation extent and frequency of application changes, automation tool changes (new versions etc) and change in application platform determine this cost component

How do these cost components compare with costs associated with human testing? Will an automated test cycle always be quicker than human testing cycle? Is speed of excution all that matters to you? Can these two versions of testing cycle be compared? Can one hour of automated testing (test execution) be compared with 1 hour of skilled human testing?

Test automation if not “appropriately” applied can be highly expensive and painful. If you are dependent on automation for achieving project GoLive timelines – be aware you could be risking the release. When someone says they can reduce testing effort by automation – be sure they are selling you something and they do not understand fully what is testing and what automation is.

Test automation is High risk item in business. Before you invest money or even before you make any business decision that based on automation capability – make sure you, as an IT manager is aware of darker side of automation.

If you are getting a sense that I am discouraging Automation and regression testing – Yes, I am. Considering the approach that today’s IT World is taking automation – I am against such approaches to Automation – but NOT to all other forms of automation that can happen.

So next time you sit in a vendor presentation and hear this "Test automation reduces manual test cycle time and effort" - Please ask "HOW" and "UNDER WHAT CONDITIONS". I am sure the presenter will have tough time in answering your question as these are not a common questions to these "automation snake oil" seller. In case if you get an answer, make sure to check relevance of those parameters to YOUR project and application context.

Shrini

I am a sapient Tester …

Here is my first and a very crude attempt write a poem (if it can be called as a poem) on sapient tester ...

I understand that testing is a sapient process *
(* A sapient process is any process that relies on skilled humans)
I question things around me
I model things to understand their behavior
I learn from related fields in testing (ones that are close require higher levels of human cognition)
I use my brain and I constantly try to improve my thinking
I don’t believe in rote processes that mechanize human thinking
I don’t use documentation, KA/KT* as essentials for getting started with testing
I can figure out my way in understanding the software I am testing.
I use automation as aid in manual testing not as a replacement
I use techniques as heuristics
I just don’t simply follow the written instructions and procedures - I question them
I collaborate with developers, other testers, users, support personnel and other stakeholders to understand wide range of perspectives.
I understand depth and value of human testing
I understand that by automating a test, I might be loosing some thing.
I am sapient tester... Are you?

Association of software testing (AST) publishes a magazine called "sapient testing"

James Bach wrote about sapient processes here

* KA - knowledge Acquisition and KT - knowledge Transfer

Shrini

Thinking about Zero defect software - Attaining software Nirvana ...

“Humans have always proved impossible is possible … Dreaming, Desire and passion is key to achieve “impossible”. Wright Brothers dreamt about “flying” – they achieved it. Once, reaching moon was impossible – now? You need to give your life to it …Zero defect software is possible …” You need to aim high”.

This is how, a colleague of mine argued with me when we were discussing about Zero defect software. He believed that many software like IBM mainframe apps are running with near zero defects … (there could be defects but of cosmetic nature)

The notion of defects, number…

No defect is bigger or smaller … Let us say there is a 0.5 second delay in system response … Will this matter? Is this a bug … depends upon which software we are talking about? Context is important …

When people talk about software processes, discipline and making an attempt to achieve zero defect … they are often of the opinion that …
Human beings make mistakes deliberately … many or all can be avoided if there is a second eye or a watch dog
Human do sloppy work unless controlled – given a choice no one would do a good job (if no one is observing)
Humans require constrained and regulated environment
Humans - overlook

It is this possibility that people require discipline makes others (mangers especially) to think that zero defect is possible. People who vociferously argue in favor of zero defect software – think so because they feel that it is because of human laziness and other related aspects – defects are introduced. Put a system of governance, policing – you will achieve zero defect. How true is this?

Chasing the definitions of Quality ....

Why do we have so many meanings/interpretations/definitions of quality - A colleague of mine asked recently. I did not have any ready answer for him ... I said ... after a pause ... "There are so many ways to see the quality". Because, I think every one, whoever attempted to define quality (from Juran to Weinberg, ISO to CMMI), did so wearing a "user's" hat while they were not real users. It is like doing a role play temporarily - a third party view.

Michael Bolton mentioned this elegantly...

Quality is not an attribute of a thing or a thing (most of us believe that is in a thing). It is a relationship between the user and the product.

So extending Michael’s view point - quality is a two way and intimate relationship between the user and product. There is no room for third party.

So as a third party everyone can come up with "possible" meanings of the term quality.

The moment we understand that it is relationship on a timescale (perception of quality does change over time) not something built into the thing ... we will be good. Beware a notion of good quality does not alway remain so .... as time changes so does the perception about quality in the minds of the beholder.

What does this mean to we, testers?

- As testers we do a third party assessment of (to best of our abilities) the relation ship between a product and it's anticipated user base.
- Things like fast, great UI, cheap, reliable are only manifestations of relationship between user and product - not real things or attributes about quality.

- Quality is observer dependent - quality can be viewed always and only viewed from the eyes of the user's frame of reference
- User's taste, perception of quality for a product changes. Hence there can not be any "one-time", "fits all size" kind of testing/assessment for a product (more so .. if it a software product)

Who is next in the line with a definition of quality?

Shrini

When method dictates outcome and Goal – See Goal displacement in action - I

I first heard (came to know) about this word from James Bach and this immediately caught my attention. I started seeing it everywhere. The phrase "goal displacement" is generally referred to a situation where method of achieving something dictates the out come and goal. The process by which the means used to achieve a goal becomes more important than the goal itself.

Here is an example that I think illustrates what I mean by “Goal Displacement”.

A model of Test Automation

Manager: I am quite impressed with overall progress this automation project. But I would like see some metrics and measurements to quantitatively monitor future projects in this area – like number of scripts produced per day, individual automation engineer’s productivity etc
Me: Ummm … right now, our model of automation development does not itself to such measurements. We don’t treat overall work in terms of scripts and measure engineer’s productivity. I am afraid; I don’t have an easy answer to your question

Manager: I understand your concerns. But I need those metrics.
Me: Here is a deal, how about modifying the whole process of automation such that it is easier to measure and exactly built according to metrics that we would like to measure (regardless of what outcome we would like produce)?

Manager: I did not understand that … Can you not just introduce them to existing model without affecting anything else - especially the output?
Me: To accommodate the kinds of things that you are asking, I will have to modify the way we develop things – which I am afraid will affect the quality of our deliverables. I don’t see any way to make those measurements in current model.

Manager: Metrics and measurements are not suppose to change the process instead I heard that metrics help to control and improve the process.
Me: In this case they do. Do you want me go ahead and change? I see a serious, negative side effect of changing the process/model to suite metrics.

Manager: Ummmm … Let us talk about it some other day (walked away)

What is happening here? Manager wanted to see some metrics to see how the automation team is performing. I simply said "You can have those metrics but at the cost of changing a process that is apparently working fine and producing the results that make you happy". So here, the introduction of metrics (or a measurement process) causes a shift in the goal of producing good automation code to goal of having a process that measurable in some way. It is like saying if current process (way of doing things) does not cater to a new related expectation, possibly a conflicting one with original goal (a displaced goal) - then forget about goal - do whatever change required to meet new goal or expectation.

Stay tuned for two more examples of goal displacement ...
Shrini

Software Testing certification - To be or not to be ..

I am yet open up my thoughts on certification ... also I have not been very critical of certification programs. Here is an attempt to publish my personal views on this topic ..

In 2003 I took CSTE exam ( A link from QAI is here) - as asked my manager and cleared with flying colors - 90% above ...My preparation for the exam was painful ... I had scratched the CBOK (reference material) all over as I did not agree with most of material - which seemed to copy paste from a software engg text book and about 20 years old stuff. I could memorize lots of stuff without sufficiently challenging it. I passed the exam ... did I learn anything new that helped my work --- NONE other than some terms and their meanings (questionable).

I also constantly watched the kinds of people who passed such exams .... I must the quality was bad. Passing was easy ....In my opinion, most of certifications (I am a certified Java programmer too) suffer from this problem. There is so much focus on memorizing things not about learning.... There is little space for debates and questioning... That is you questions do not have "comments" field .. or "I don't agree with premise of the question because...." kind of open ended response from the candidate .. Who has the time to site and evaluate all responses ...?

Discussions on certifications often take "Emotional" angle - people say exams like BE and certifications help one to improve skill and knowledge .. I would say they only provide reading material .. rest is in your hand (rather in mind) . I strongly feel that skill comes first and gets developed by practicing, doing, spending real good time with dedication not by passing an exam that tests your memory retention capabilities.

The problem I see with testing certifications is that - they don't test testers skill to do testing ... (a practical exam like our science lab in schools might help) instead they test tester's memory retention skills.

Are there any certification exams that watch tester testing stuff (with some video etc) and then give "good to go" Tag? No .. that would be expensive and tough exam to administer right ...

An example of goal displacement ... certification exams appear to be designed for easier and large scale (yes/No or choice) kind of assessment than complex/tougher assessment of watching tester in action and rating his/her level of skill.

Why goal displacement -- Some group of people in testing thought - we need an exam to test our testers .. so what kind of exam should we have? ... something that is easier to administer or something that tests tester in action... They prefer to choose the first one ... So the goal of having a good certification got shifted to having a certification exam that is easier to evaluate ...why ? Good certification exam requires complex models and involve lengthy/costly administration. It costs lots of money...

Having said all this ... its your call .. I am neither in favor or against to certifications in testing (some people 's daily bread comes by running those exams -- let us not take away that ...). It is a big business in testing industry today. Being testers, let us evaluate what is good for each one of us (considering each ones technical and educational background) and take the decision.

Any alternative to certification ....?
Focus on skill ... how many of us can go to an interview and challenge the interviewer "Give me a software to test ... give me an hour ... come back I will have a defect/issue list ready". How many of us can take this challenge of "testing anything, anywhere, under any time frame" ? Managers in IT companies need to see evidence that this person can test ... he has tester attitude...

In absence of such evidence, managers resort to something that easy to check (do you have this certification?). If you are fresher or beginner in software testing field ... and struggling to get a break or good job ... You have two options ... Get certified and get a job .. (there after it is your skill that keeps you on job not the certification) other option ... practice testing, read stuff, read books, learn programming language, debate, sharpen your analytical skills, write blog, discuss with others in the field .. build a credible profile for your self (Google should be able to find you) and go confidently to any interview and say "give me stuff to test ..."

First one is easy path ... some money and about few weeks of reading, take the exam your are done... second one is tough one.. Requires you to commit to the profession ... requires your learn things, may take years to be good at testing, many years to build a credible profile that speaks for itself ...

People choose first one mostly as it is easy ... So I would not say all who support certification are not skilled people and take short cut to success neither all those who oppose certification are good skilled testers …

You want to be certified? it is your call -- What ever you do, be a tester by heart - test everything/question everything until you are convinced

Read James Bach on Certification here

Michael Bolton on why I am not certified

Ben Simo on his favorite certification

Balancing on the other side -- AST certification debate

Shrini

18 Myths Associated with Exploratory Testing ...

Myths are public dreams; dreams are private myths – Joseph Campbell
Science must begin with myths, and with the criticism of myths - Karl Popper

Instead of writing on what is exploratory testing, to me, it looks easier to start of explaining what is not exploratory testing. For those who are hearing this word for the first time – let me remind you this is a testing approach that involves simultaneous test design and execution with an emphasize on learning. The term exploratory testing is coined by Dr Cem Kaner around 1983.

Exploratory testing as approach has a legacy. Its predecessor “Ad-hoc” testing is a perceived as spoilt kid. Until recently, I was under the impression that adhoc testing is quick and monkey type of testing (often connected with sloppy testing) where you just play with application in order to find some bugs that are out of specification. I was proved wrong during a rapid software testing workshop that I attended at Toronto. James Bach mentioned in that class that the word “Adhoc” means “to the purpose”. Most people believe (as I did initially) that Adhoc testing is undirected, random clicking of mouse and navigation. This is probably the reason, why Dr Kaner decided to name the baby differently so that the “stigma” associated with the approach of testing goes off.

I think Dr Kaner did not stop at just naming the baby, he along with James Bach, Michael Bolton, Jonathan Kohl, Jonathan Bach and others did lots of research and practice related to “exploratory testing”. Today, if many of us talk so confidently about this – it is due to the pioneering work done by these people.

Let me also remind that exploratory testing is not a testing technique but an approach and opposite of exploratory testing is “scripted testing”.

There so much of confusion, skepticism and disbelief associated with this approach to testing. Based on my interactions with people from all walks of testing world, I have drawn a partial list of myths that are associated with exploratory testing.

Let us go straight into the list of myths …

Why bother about ET?
1. ET -- every one does ET while doing testing - why bother? why have a special name?
2. ET is some kind of snake oil - people use this term to indicate some mystic thing to make money
3. ET will not work in my context - I am not even willing to give it a try as I am not convinced we should give a try. I know it for sure.
4. ET does not seem to have come under the radar of Gartner or Forrester's -- it might not be that popular.

About the form of ET
5. ET - is an unstructured, ad-hoc (meaning sloppy) process.
6. ET is nothing but doing testing without any formal test cases.
7. ET is instant bug hunting process.


About ways of doing ET
8. ET should always be done after completion of all planned scripted testing (if time permits) If you have time (while you wait for a new build), do ET to use your time productively.
9. ET is not predictable and repeatable (was it meant to be repeatable?)
10. ET shows scant respect for well established Test techniques - how can you do testing without using any of those time tested techniques

About skills required for ET
11. ET requires in depth domain expertise hence only domain experts can do it - it is not every ones cup of tea
12. ET seems to require special skills (domain, quick learning etc) - hence we do not do it.
13. ET is highly person dependent, unacceptable to us - we are resource starved industry.

About suitability of ET in a context
14. Our quality process standards require that every test effort be substantiated by a detailed report of what test cases are executed, on what platform, what data is used, how many test cases passed, how many failed etc. ET is not so good in producing such report. ET does not provide enough evidence/proof of having executed testing.

About perceived Value of ET
15. ET is not process oriented and is not methodical (requires highly skilled disciplined, responsible testers). Anything that is so “person/Skill” oriented is strict no-no in our environment (process driven)
16. ET can not be outsourced - even if it is we can not assess the progress - it is an uncontrollable process.
17. ET can not be automated - I am looking to reduce spend on testing - ET can not help me there.
18. ET seems to be useful in only in those environments where there are no requirements but ours is a very structured process. Why to use ET?

Why these are myths and not the truths – that is an exercise to the readers. I welcome each one you, the reader to challenge my claim that these are myths and lets debate …

Shrini

Here comes my 100th post ...

4 Years, 100 Posts, 200+ Comments, 15+ Commentators, about 5000+ page views (statistics since Dec 2007)

It all started in July 2004, on 23rd I posted my first blog post. Why did I start blogging? I just wanted to join the bandwagon of my colleagues and make presence in blog-o-sphere. . I could not blog much in my first year. I struggled to decide what to write and often felt choked after writing few paragraphs or sentences. Mostly my initial posts referred another post or some news which I announced or made reference in my blog. I did not have any significant stuff to offer to the readers. I think this is what people call as “writers block”. I suffered with that in my initial blogging days.

Things started to change as I wrote more, attracted few best minds in our industry to my blog. As I started receiving more comments, I, all of a sudden, started to see lot more ideas to write and debate on. Many times I asked myself, why do I blog? Got only one solid voice from inside … “I wish to communicate …I have something to tell to testing world”. So, all these months and years, the notion of “idea or urge to communicate” drove my blogging. I felt very happy and satisfied after a posted a blog post. It was like winning a game, or scoring good marks in an exam. A good (in my own terms) blog post always energized me.

After posting, I would monitor my blog for comments. Depending upon the content of the post, I expect 2-3 comments in 24 hours of the post going live. Sometimes I loose sleep if I don’t comments on a post (having a content that I believe is thought provoking and interesting). I then, shamelessly send mails to my frequent readers (l have a list) asking them (explicitly) to comment on the specific post. As far as possible I avoid doing this as I feel this is similar to a well dressed girl going round and asking “how do I look” - a desperate attempt to grab attention. Usually best responses come without asking for it.

In my opinion, a success of blog can be judged in terms of the number and quality of comments. Number – because it is an indicative of number of readers of the blog who not only spent time with the blog also went further to write a comment. Most of the blog readers today use blog as one way communication tool – it is not. Some well moderated blogs like the ones from James Bach, are effective 2 way communication mechanism.

Many times, I think of getting on with “Google Adsense” …in current shape and size, my blog even may qualify for it. I am resisting … I personally do not like the blogs with “google adsense” as it distracts from reading. I see a blog as a library of interesting topics often focused on a specific area. Anything else is distraction for me. I don’t want to loose that appeal from my blog by opening it up to Google.


As a blogger I must avoid doing

without leaving any clue, stop blogging for months and then come back and say “ sorry It has been long time since my last post” … Readers tend to develop some inertia when then see a lull … So keep blogging (one in a month at least .. this time can vary from individual to individual).

Discourage or block comments or make the process of commenting too tedious. To me comments are very important as they are only available mechanism of getting feedback from my blog readers.

As blog reader of other blogs, I would not like if some one does not respond to my comment (especially where I challenge the post content). So if a reader of my blogs posts a comment asking me to clarify – that would be my first priority.

Post reference to other blogs without having your own thoughts of ideas

Short posts like “I am back or here is an interesting thing” … No short posts unless there is real need.

These are my own rules of blogging – I plan to follow them religiously and make sure my readers get best experience of reading my blog.

Benefits of blogging (at least to me)
This blog helped me in connecting many unknown faces. I have few students asking for some help on a topic on testing, I got someone asking me to do a software product review for me. I got several job offers from the readers (that were looking for hiring me), I made new friends via my blog. I have great names in testing World like James Bach, Cem Kaner, Jerry Weinberg – commenting on the posts of my blog.

My message to my blog readers –

Keep checking my blog and generously comment on the posts. You can also use my blog as a platform for debate on something other commentator said about a post. For example you might want to react to a comment posted by Pradeep on my post. I welcome that. This will make my blog as multi fold communication – a discussion forum. As far as possible avoid giving very general comments like “Great post or I liked this blog etc …” As I welcome the appreciation (at times I need it) but would like those items to be in 1:1 mail to me (shrinik@gmail.com) – for a reader, comments like “ I like this post or this is a great blog” not very useful. I encourage my readers to challenge me more often – critically analyze the views presented here … write about it and leave a link. I think such challenging comments are quite useful to me and other readers as it presents a different view point. Do write to me about the list of topics that you would like me to blog.

There is one contradiction that I am not able to solve. If I am blogging for myself (for my satisfaction) should I bother about my readers – their choices, likes/dislikes? Should I worry about responding to the comments?

Should you bother about me blogging or not …. What do you say – Dear Reader?

Software Requirement Heuristic ...All requirements are Vague

Here is a heuristic - "All formal/written requirements are vague – only exception is when requirements are collected using a process that does not involve any human”

(Wait … if at all such process exists, where it came from? There must have been a human being such a process needs to be designed by a human … …hence there is a human element here too … Is above heuristic, an axiom or hypothesis or law?)

So there will be always gaps between "Customers view", "end user view" and programmers view". ET can help you to explore these gaps.

Jerry Weinberg in his book on Exploring requirements.. - gives the example of a nursery rhyme "Mary had a little lamb" and goes on to prove that there can be at least more than a dozen ways of interpreting above sentence ... Remember we typically deal with requirements documents of 50-60 pages ... Imagine the amount of misinterpretation possibilities....

I was discussing with this with Michael Bolton, recently. I said “I can say for sure that as long as requirements are written in English, they will be ambiguous. English is a funny language". He immediately shot back and said "you can say same thing with Hind, Tamil, French or any other human language"... I laughed and said “well, you are right, how about this ... One way to get unambiguous requirements is to express them formal languages like computer programming languages ... say Java. "Class MyClass" in java is nearly unambiguous". Michael continued "Here, the ambiguities come, not from written requirement but from the process of translating what is there in one’s mind to formal language".

That brought us to the real truth about software requirements ... "As long as one human tries convert what is there in her mind to some thing that written form; mind to document or mind to mind data transfer .... There will be ambiguities"

People can say a thing in different words but might mean one thing or they might say one thing and might mean differently....

Welcome to the world of software ... a discipline with high human content ....hence we need skilled software testers to keep world "lighted"

First thing you need to acknowledge is Requirements are fallible - they can be wrong - horribly incorrect (so will be corresponding program). If you base your entire testing only on requirement - you can go wrong by miles ... beware of this requirement based testing trap...

BTW there is a type of testing called requirement based testing ... can you imagine how narrow it can be?

You might have heard of people counting requirements … Can we count requirements as we count apples and pizzas?

What do you say about this concept of Requirements Traceability Matrix (RTM) – a holy thing that most swear by? Fundamental principle on which the concept is that “requirements and test cases can be counted”

Please… for God’s sake understand requirements, bugs, test cases are not real things and can not be counted in a way you count physical things, they are mere concepts, ideas …

Reductionism and Test Techniques - who, what?

Scientific reductionism is an undeniably powerful tool, but it can mislead us too, especially when applied to something as complex as, on the one side, a food, and on the other, a human eater. It encourages us to take a mechanistic view of that transaction: put in this nutrient; get out that physiological result.
- Michael Pollan

I have heard statements like the ones below several times … I am sure many of you have.

- Orthogonal Array or pair wise technique reduces the number of test cases and hence can optimize the test effort

- Use of orthogonal array based testing (Test design technique) has demonstrated to produce superior test plans that improve testing productivity by a factor of 2.

- Equivalence class partitioning (ECP) is a functional testing technique that systematically reduces the number of tests from all the possible data inputs and/or outputs, and it provides a high degree of confidence that any other data in one particular subset will repeat the same result.

- Boundary value technique catches the bugs occurring near boundaries. Use of boundary value technique increases the effectiveness of test coverage.


What do you notice common in all these statements? Faith and reductionism. When you say “Technique XXX does this or help in this [reduce the number of test cases], you are forgetting that it is not the technique but it is YOU as someone who is making an assumption/assertion.

For example, when you say “Orthogonal array technique reduces the number test cases” – you actually saying, I as a tester using this technique making assumption that only pair wise test cases generated by this technique matter – rest I ignore.
Or
When some one says “ECP technique systematically reduces the number of test cases” – that is a view of reductionism. It is tester's hypothesis, judgment and assertion that certain sets of data values CAN be treated equivalently. ECP does not "reduce" ANYTHING by itself systematically or otherwise. ECP's principle per se is that there are groups of data that can be modeled to be treated identically by the AUT

I can say that, this is a real difference. This shifts the focus from the technique to the person who is using it. You then can not shift the blame to the technique if it fails.

As testers we use lots of tools, techniques and heuristics to understand, operate and observe the test objects. Every time I use a technique or a heuristic, It is my explicit choice and I make a set of assumptions and assertions. The goodness and applicability of results of the test or technique, directly depends upon these assumptions and assertions. Technique/tool has no role in it.

Pradeep Soundararajan and Michael Bolton have it here – A focus on tester not on test or a tool or technique…

http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html

If you take a gun and kill a person – what do you say? Gun killed the person?

Shrini

Percipient Subject and Software Quality

Jerry Weinberg in his famous book “An Introduction to General Systems Thinking” mentions quoting Albert Einstein…

“Belief in external world independent of percipient subject is foundation of all science”

Let me extend this to Quality (or Software Quality) …

“Belief in external world independent of percipient subject is essential for the notion of Quality in software world”

All definitions of Quality (except that of Jerry) from Juran, Deming and others somehow assumed the existence of “external world independent of percipient subject”. Hence Quality has been more or less defined in absolute terms.

Let me remind you -
- Quality is value to some person (one who matters)
- Quality is not an attribute of a *thing” – we can not measure Quality but can only assess it in “relative terms”
- Quality is in beholder’s perceptions.
- Quality is not an intrinsic attribute of any “thing”
(Quotes from Jerry, Michael Bolton and others…)

Bonus Tip:

“Belief in external world independent of percipient subject is essential for the success of a metrics program”

Most of Metrics programs that are claimed to be successful stand on this belief.

Howzzzzzzzat?

Shrini

7 Habits of successful Testers

Here comes the last post for the year 2007 - to be expanded later ...

• Self Driven or high levels of Inner drive for learning new things – No fear of unknown.
• Spontaneous – Thinks on the feet – Good in emergency response.
• Agile and adaptable.
• Love for Science (Physics/Chemistry), Mathematics and Philosophy.
• Love for problem, Puzzles.
• Hunger for self Expression – Writing, speaking.
• Organized Skepticism and constantly challenge their own thoughts

Shrini

Exploratory Testing challenged - Part I

Here is a conversation that I had recently with one of my manager. We were discussing about merits and demerits of “Exploratory Testing” (ET) as a testing approach. This manager is a die-hard fan of “scripted testing” and apparently swears by “Quality/Factory” school of Testing.

Me: We should propose ET for this
Manager: Why? What are the benefits?

Me: ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
Manager: Why our test scripts fail to find those bugs? I trust that our specifications are exhaustive and our scripts are thoroughly reviewed and signed off by the business experts. Test scripts should be able to find nearly all bugs that I consider important to be fixed.

Me: Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”. Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts. There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space. So, there is good possibility that scripts will not find all bugs that potentially be discovered.

Manager: What are other benefits of ET?
Me: ET helps the test coverage and provide enhanced bug finding capabilities over scripted testing – especially to nullify “Pesticide Paradox” associated with scripts.

Manager: How? What is this pesticide paradox?
Me: Just as pests in soil over a period of repeated application of a specific pesticide acquire immunity to it and fail to die or show up – software bugs become immune to repeated application of specific Test cases. Over the period of time developers are become aware of test cases that are executed and *specially* test to make sure that new build of the application good just enough to pass those tests. As result of this, there is a “false” sense of stability and quality of the application.

Manager: So... Test cases wear out … why so?
Me: Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment. Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.

Manager: what else …?
Me: Quoting James Bach – “The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper. There's a lot of value in that way of testing. But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.”

Manager: What are other attributes of ET?
Me: Cem Kaner states that exploratory tests provide
- Interactive
- Concurrence of cognition and execution
- Creativity
- Drive towards fast results
- De-emphasize archived testing materials

Manager: I heard another related term “adhoc testing” is this similar to ET?
Me: Yes and no … Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing. Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.

Manager: What is specific about ET vis-à-vis scripted testing?
Me: ET is more of investigative approach where as scripted testing is more of “validation” or “conformance” oriented. In scripted Testing, the tests, the sequences, data, variations etc is pre-defined where as in ET, test design/execution and learning all happen more or less at the same time.

Manager: I heard that ET requires “experience” and “Domain knowledge”. Can an average tester do a good ET?
Me: I am not sure how you define “average tester”, “experience” and “domain knowledge”. I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others. Domain knowledge certainly helps in ET but I do not consider it as mandatory.

Manager: Fair enough… What types of bugs ET can discover?
Me: It depends upon what kind of bugs you want to discover. ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application. ET can be configured to cater to specific investigative missions. You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.

Manager: I notice all along you argued like a “purist” in Testing. I am more of a Business owner; I would need to relate every dollar I spent to the return or the benefit that it gives.

Me: No … I would not call myself as purist, not at least here. I bring in lots of business considerations in my recommendations related to testing. ET provides a way of optimizing testing efforts by time-boxed sessions with charters. Depending upon the nature of the information stakeholders are looking for, ET sessions can be accurately planned.

Manager: Yes …

Me: Let us say you have 5000 scripts for an application and they pass all the time. Would you be worried?
Manager: Ummmm… It depends upon the context. But mostly I would not worry about it. I interpret that as a sign of enhanced maturity of those specific application areas. It is quite possible that there are no bugs that I should worry about - the “scripts passing” is a “confirmation” of that fact.

Me: What if this trend continues and next 5 cycles of testing also do not produce any bugs? Would you be worried then?
Manager: No …not at all In fact, I would reduce the size of scripts being executed to say half – 2500 as the application has become stable. This is an indication for me to "cut down" testing effort, I can possibly look at automation as well.

Me: Here is a twist, what if your ALL scripts are passing but your are seeing bugs (either detected by other means or by the customers) .. Would not you doubt your test cases?
Manager: Depends upon the kinds of bugs I see … If I were doubt something or someone at all … I would doubt test results, testers integrity and project management in general. Test scripts are less likely to be at “falult”. That would process issue - we would need to tighten the process.

Me: OK … What corrective action you would take then? What you steps will you take?
Manager: I would immediately order a thorough Root Cause analysis of the defects and identify what is causing them in the first place. Tighten the development, configuration and deployment process. I would strictly enforce the processes (improved) in Testing and mandate the testers to correctly and meticulously execute the scripts and report the relevant results correctly.

Me: What if you still find bugs outside your scripts?
Manager: That would be a “hypothetical question” – not likely to happen. In any case my focus would be to improve the testing process and strengthen Test scripts. Again, if you are finding still finding bugs – probably those bugs would be “obscure” type – I might not have to bother about them…

Manager: Good… I am still not convinced that ET can give the bang for the buck. As someone who is interested in predictability and repeatability of testing, I am interested in a test process that can scale.
Me: Ummmmm … OK… what is a testing process? Is this something that “actually happens” or “something that is intended”? Is repeatability and predictability - all you care?
Manager: You are too much … there is a limit to asking questions …I don’t think this discussion is leading to any good … Let us talk about it some other time [walks out of the room]

I am continuing my discussion with this manager and post the views and continued discussions in Part 2....

A very happy new year to all …

Shrini

Advantages of "highly repeatable tests" ...

I was reading Ben simo's post on "what is software testing" - a meticulously created list of quotes about software testing. The beauty of this post is that it traces the history of software testing”. One good way to read this post is to evalauate each statement or the quote mentioned, with respect to it's relevance to software testing.

I stumbled upon this GEM from James Bach.

“Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.”

- James Bach,Test Automation Snake Oil, 1996

So, if you have an excellent set of "highly" repeatable tests, in terms of execution (automatable sequence of actions) and in terms of results (pass of fail) - congratulations, you have successfully managed to find a set of test cases or scenarios where the software is least likely to fail - meaning you will not (or do not expect to) see bugs/problems

But ... wait .. is that your testing mission?

I heard someone yelling from my back… “Yes .... that is what we expect in regression testing. But sometimes occasionally we do find bug as developer made a mistake that was caught or tester [by mistake] deviated from scripted test sequence [a process issue or discipline issue]"

What do you say?

Shrini