Saturday, May 17, 2008

Can Automation reduce human testing effort ?

This post is for all those IT managers who are considering outsourcing testing and listening/talking to various vendor presentations on testing and automation.

I was in a conversation with a client other day. He asked me “Can you reduce the number of resources that you have deployed currently for regression testing of this application to say by half in next six months or so by using “Automation”?

I was not shocked to hear this since for past few years, I have seen many clients in IT application testing space have had similar expectations out of outsourced testing and hence on automation. In an Outsourced IT application testing domain, this is how a service provider positions Automation – A means to cut down the cost of testing (resources).

Client’s expectation of reduction in testing resources (with no reference to scope of testing) using automation in a specified time frame stands on following assumptions or beliefs.

- Regression testing is simply executing a set of test cases and reporting results nothing more than that.
- Regression testing gives the confidence that nothing is broken – all that was working previously is “intact”
- Automated test cycle means zero human involvement hence percentage of automation should directly result in proportionately reduction in human effort of testing. For example if 50% of test cases are automated, effectively manual test cycle time can be reduced to that percentage.
- Automation is turn-key. Once you have an automation solution developed, one can just keep using it unlimited number of times without any extra cost and effort
- Operationally, automated test cycle ALWAYS takes less (fraction of manual test cycle version) time – there would be no additional effort required to investigate results and chase script failures.

Let us look at costs of creating and owning automation (note that majority of these are of recurring types)

- Tool evaluation, proof of concept and other initial investigation costs
- Automation tool cost and recurring licenses/upgrade costs
- Resource Training costs
- Costs associated with automation environment (development and execution) – servers, applications, connectivity etc
- Costs associate with manual test cases cleanup, rework, test data, clarifications and any other effort to facilitate automation
- Automation development/ testing/review and acceptance costs
Costs of setting up and maintenance of automation execution environment
- Costs of investigation of automation results, failures
- Costs of re-runs (testing automation code fixes) and any required additional human testing
- Costs of maintaining automation code – design and structure of automation extent and frequency of application changes, automation tool changes (new versions etc) and change in application platform determine this cost component

How do these cost components compare with costs associated with human testing? Will an automated test cycle always be quicker than human testing cycle? Is speed of excution all that matters to you? Can these two versions of testing cycle be compared? Can one hour of automated testing (test execution) be compared with 1 hour of skilled human testing?

Test automation if not “appropriately” applied can be highly expensive and painful. If you are dependent on automation for achieving project GoLive timelines – be aware you could be risking the release. When someone says they can reduce testing effort by automation – be sure they are selling you something and they do not understand fully what is testing and what automation is.

Test automation is High risk item in business. Before you invest money or even before you make any business decision that based on automation capability – make sure you, as an IT manager is aware of darker side of automation.

If you are getting a sense that I am discouraging Automation and regression testing – Yes, I am. Considering the approach that today’s IT World is taking automation – I am against such approaches to Automation – but NOT to all other forms of automation that can happen.

So next time you sit in a vendor presentation and hear this "Test automation reduces manual test cycle time and effort" - Please ask "HOW" and "UNDER WHAT CONDITIONS". I am sure the presenter will have tough time in answering your question as these are not a common questions to these "automation snake oil" seller. In case if you get an answer, make sure to check relevance of those parameters to YOUR project and application context.



Sharath Byregowda said...

Hi Shrini,

I asked the same question "how would the tool reduce manual test effort and time" in one of the training sessions, the speaker could not convince me nor the audience, when the argument heated up, these were his final words on the discussion - "We are here to learn the tool and let's concentrate on it".

This makes me wonder, why managers are so obsessed with automation, have they really come across any project wherein automation was able to meet all their requirements? Or have the companies made the word “automation” mandatory, to be used by Managers.

Don’t you feel if Managers continue to use the word “automation” the same way, they might also mislead even the clients? – may be this effect has already started.


Shrini Kulkarni said...

Sharath --

>>> "We are here to learn the tool and let's concentrate on it".

Rightly said .. probably that was a session on learning a tool not apply mind about how to use the tool. Pass it ...

>>why managers are so obsessed with automation

Simply because of the promise that tool vendors give - easy to use, cycle time reduction, less hassle to manage, is obdient and so on...
The promises seem so tempting that managers stop thinking ...

>>have they really come across any project wherein automation was able to meet all their requirements

Probably not ... but there can be other reasons to blame. When an automation project fails, you can typically assign the cause to anything other than approach to automation/testing.

>>>they might also mislead even the clients?

Remember at client side also there are managers who award projects to those show and make fancy claims about automation. At the end of everything, everyone has been taken for a ride, except the tool vendor and few sapient testers who understands the trap ...


stanleyxu (2nd) said...

Most issues are found manually, but most time is spent on running those tests, that can be automated. I think /good/ automation do help test engineers, because they can save a lot of time and investigate more potential defects.

If you ask me "HOW", I would say:
1. Judge, whether your application is worth for automated testing.
2. Judge, whether your team is able to make tests automated.

If you have a positive answer, then you can start considering of automation.
1. Search or create a /good/ automation framework
2. Create a check list and score each item with a difficulty
3. Write test cases from easy to difficult. (Difficult items can rather always be tested manually)

Not only the development requires agile, but also the testing.

Anonymous said...

Hi Shrini,

This is a very good blog I came across. I did face the same issues in explaining the same to my managers and clients.


Sofia Hunt said...

Thanks for sharing such an informative post . I also wrote the same lines on my blog -