For example : A typical test cycle:
Before Automation -- (AS IS statistics - documented)
Number of Manual test cases = 300
Time taken to execute these test cases = 10 Person days
Tester's productivity = 30 Test cases per day (derived data)
After Automation -- (Assume 150 of these test cases are automated)
How do you think test cycle would look like?
Number of test cases Automated = 150
Number of test cases requiring manual effort = 150
Time taken for 300 test cases = Time for Automated execution + Time for Manual test execution
= (how much time will 150 automated test cases will take to complete) + 5 days (with 30 test cases per day productivity)
= 1 + 5 (assuming that automated tests run at 1/5th of time of manual execution)
= 6 days
Hence a business case will be developed to justify automation investment
The business case would say - with automation, we can save 4 person days of testing effort per cycle. Depending upon how many cycles of testing are lined up - we can break even the automation investment.
Oh... What a great feeling ... This is what I see all day in and out..
Let me tell you one thing .. executives, managers love this stuff ... so neat - objectively stated in numbers, justifying in terms of monetary terms why we need automation.
But some discomfort and terrible feeling in me ... what you people think - we are missing here?
12 comments:
What's the effort to create the automated tests, and to keep them up-to-update as the software changes? (Of course, I know that!) :)
How automation friendly application is. Only during pre analysis it is found that testing hooks are missing for automation and dev is to rework to assist functional test automation.
Well, generally automation is just an aid and not a replacement for manual efforts. In my opinion, points which really drives the automation of testing efforts are -
1. Size of the project.
2. How stable the project is
3. Appropriate tool for automation
4. What , when and how much is to be automated
Correctly put by Tarun and Amit above. There is no doubt that automation will reduce repetitive manual testing efforts and shorten testing time. But the real question to ask is how much time will it take to automate those 150 test cases. How many experts will be required? How much training will be required to mobilize resources? What is the true cost of automation? How easily can automated tests be maintained? And how can this be scaled without any hiccups?
Apart form what marcus said(which will normally be Hidden), One thing that will make you even more unconfortable is ,The testing time is only 5 days as opposed to 6 days(Automation scripts run parallely).
:-p
Hi,
2 emotions come to mind.
1 --- As an employee, I would be very happy with the above. If there's a formula which keeps my directors and managers happy, why bother with change? :)
2 --- As a product owner, I would be "not so happy". I might want to look at priority/usage of automated test cases, creating the automated cases and keeping them update. And about Automation result analysis, in which we tend to spend more time than on automation itself.
Besides taking into account the effort needed to set and maintain these automated tests (great point), what I see is:
Before automation, testers needed / had 10 days to do their job. After automation, same tasks are completed in 6 days. Why not to invest these 4 remaining days in testing more? Maybe taking new approaches or even, if possible, doing more automation?
Automation must a add on to manual part of testing more of aid ....If the point is in business justification ,fact and figures it would not be actually beneficiary as the whole point of the automation is lost in the process and the quality of the scripts would also be a question mark hmmm...
All good comments here. Off the top of my head of items which may not be included in this assessment:
-Maintenance (always the ugly duckling that almost no one cares to pay notice of)
-definitely the effort to create the scripts
-Test Data preparation - More than likely, the manual execution assessment takes into this account, but is the automation execution piece doing the same?
Glad I could drop by Shrini - hope you are well. I am very interested in what you feel is missing here.
Ryan G.
Thanks Ryan G (Green Right?) for dropping in. Good points. Noted.
Shrini
Two things always to remember with deriving ROI for test automation:
1. Automation script development and framework approach
2. Maintenance of the scripts as we run through the multiple test execution cycles.
Dedicated automation team, Customized Framework, Continuous Optimization of framework, Right decision on tools, Right decisions on when and what to automate and most Importantly a Complimentary development process and standards would definitely create wonders in terms of "Time Saved + sophistication + Re-usability" in any kind of software project.
Post a Comment