I wrote about it earlier here and here
“If automation can not reduce cycle time, save testing cost and reduce manual testing errors, why do automation at all” said a colleague of mine in a discussion over mail chain yesterday.
After all, automation and test cycle time (whatever may be the meaning and definition) are so “inseparable” – the theme often evokes deep emotional, business, technological thoughts. While I agree with my colleague on the last item he mentioned “Reduce manual testing errors”, for others I would say they are true under a highly restrictive and idealistic conditions. This is similar to the practice in automobile engineering of specifying mileage of an automobile under “standard road/test conditions”. Such claims are made by automobile manufacturers to push their brand as the fuel efficient vehicle. There also goes a disclaimer that says “Actual mileage may vary depending upon the prevailing conditions on the road, driving habits etc”.
This is very true with respect to test automation too. While in ideal conditions and specific types of testing, automation can help in reducing “test execution” (not testing) time, in reality actual benefits from automation may vary depending upon the project context, state of development and testing practice and others.
I see a future where Automation tool vendors are forced to add a disclaimer to protect them from getting sued by someone who just purchased the tool expecting return as claimed in sales and marketing materials of the automation software.
As I mentioned here, testing cycle time is a complex variable that depends upon several parameters and many of these parameters are out off control of automation and even testing. Establishing a straight linear relation between automation and testing cycle time, according to me is a “horrible” and “unrealistic” simplification.
What is (are) the (fundamental) problem(s) that in testing, the automation is expected to solve?
I am afraid, there are very few testers or test mangers out there that begin their exploration into automation with this question. For many, the notion of automation appears to be well understood I am still struggling to answer this question in a context free sense. How can there be a universal solution to a vaguely or incompletely specified problem?
Dear reader, what do you think are the problems that automation is expected solve?
Shorten test cycle time? Reduce manual errors? Help to kill boredom and fatigue due to repeatability? Save dollars of testing cost? Bring consistent testing results? To supplement human testing capabilities?
To be continued …
4 comments:
On my current Project, the automation testing is being developed to solve the problems of testing the application in different language (potential looking at a total of 18 languages), using the same test script for all languages (update one test script, therefore updated for all languages).
Also it is used for regression testing of daily builds which are automatically installed and tested each day. This allows the Software Engineers to correct the code while it is still fresh in their minds (only one day old, not days or weeks).
>>>>What is (are) the (fundamental) problem(s) that in testing, the automation is expected to solve?
Shrini,
In my team we use a classification system for test automation. I've explained this in my blogpost here
In my experience, we normally get our automations to level-0 or level-1
Considering the context of the levels, each level may have a specific objective that we are trying to solve. In our context, level-0 tests ensures the 'testers can take the build for furthur testing'. level-0 tests may ensure that the 'coding process' has not introduced any major defects 'the most commonly occuring user scenarios'
I always ask my automation testers to spefically mention their objective as part of the automation plan they create. With a clearly defined objective, we can always check the end result of the automation to the objective defined.
1. Increase stability of environment by means of everyday execution
2. Extend regression testing that can't be executed manually due to time limit
3. Reduce tests' execution time during test cycle
4. Avoid fatigue and boredom of testers
Your colleague only mentioned a couple of reasons to invest in automation. I can think of others, mostly about leverage (Archimedes said: 'Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.').
On the project I'm managing, we're using test automation to allow us to do two things that we can't readily do manually:
The first is to simulate hundreds to thousands of concurrent operations being monitored by a monitoring software reporting back to a single server (scale and performance testing). There's no way we could do that simulation manually.
The second is testing new versions of the operating systems that we support with our monitoring software. Every time a Linux vendor comes out with an update to their distribution, we need to support it. With automation, we can test those distro updates in a less than a day, where it used to take us about two weeks. I suppose this may have something to do with "cycle time" but someone needs to give that term a clear definition before I'll agree with them.
Of course almost all of our automation is home grown, we've found that the commercial tools are mostly useless. (Although for configuring the distros, we're using cfengine.)
As you noted, the generic goal of automated tests is often assumed to be the holy grail of software testing. Good test automation requires significant upfront and ongoing investment; most companies are not willing to make that level of investment. Unfortunately, many companies assume that test automation will enable them to skip the investment in well trained observant software testers.
Post a Comment