"No device has been invented which can match the human eye for accuracy and precision in many tasks; nor the human ear. Even the admittedly handicapped human is able to identify and distinguish among far more scents than any automated device. Human pattern recognition, language recognition, and language production ability is well beyond anything currently envisioned by automation engineers."
- Answers.com
Hope you have heard many times the word “Automated Testing” (Some places I even have heard “automatic Testing”)
Have you ever thought – what does that mean? Let us dig little and try to figure out what each term in above phrase *might* mean?
We have two words in above phrase – “Automated” or “Automatic” and “Testing”. Let us explore possible meanings of these terms ….
(A) Automated – something that happens without any human intervention – something that is done by a machine-software program or by self.
Answers.com says -
"automated" (adj )
Definition: made or done by a machine
Antonyms: by hand, manual
Automation (ancient Greek: = self dictated),
The word testing has many meanings – I present two contrasting definitions
(B) Testing is an act of technical investigation performed on behalf of stakeholders in order to reveal quality related information – it is an act of questioning – An infinite search for problems – problems that can annoy/irritate/destroy/damage a stakeholder.
(C) Testing an act of executing a test case derived from a specification and verifies whether the test case passes or fails.
(D) It is an act of confirmation that software behaves in a way as prescribed by the underlying specification/design.
(E) Testing proves that the software works as desired.
Now think about constructing the meaning of the word “Automated Testing” and we have these definitions (A), (B), (C), (D), (E) …
Automated Testing = (A) + (B) - Possible? I believe No!!!
Computer/Program can not question, improvise, learn, adapt, think, emphasize
Automated Testing = (A) + (C) – Seems possible – but in a narrow way. Using another program (automation tool) one can execute an automated test … but that is only a part of whole story – what about Test design? Defect investigation? And host of other activities that fall under testing umbrella …
Automated Testing = (A) + (D)
Automated Testing = (A) + (E)
Both of above seem to be “Not possible” as checking or proving that software works the way intended or as per spec (writing in English – hence can be interpreted in infinite number of ways) using “automation” – solely – is not practically possible.
So … think twice before using the term “Automated Testing” – as nothing really EXISTS in this world of testing called as such …
Challenge me … think of ways where you can justify the use of “automated testing” – come up with definitions of “automation” and “testing” and link them !!!
BTW what do you think about "automated tests" ??
Shrini
13 comments:
"It's nonsensical to talk about automated tests as if they were automated human testing."
- Lee Copeland, 2007 CAST Keynote
When I refer to "test automation" or "automated testing", I am not referring to completely automating any software testing. Instead, I refer to automating some testing-related activities.
Software is automation. Therefore, I think it would be impossible to test software without automation. :)
Just as the software we test is unlikely to be able to completely automate whatever its users do, test automation is unlikely to be able to completely automate what we testers do. Software -- including test automation -- is a tool to be used by people.
The act of testing -- questioning software -- requires a thinking questioner. The questioner can benefit in using tools but cannot be replaced by tools. (See Automating Outside The Box.)
I've encountered many people that limit "test automation" to test execution even though they use automation for things other than execution. Any time we use software to help us test, we are using automation.
In spite of the common execution centric view of automation, most of the testers I know use "test automation" to refer to automation used in testing and do not think of automation as something that replaces thinking human beings.
Those that do put too much trust in automation are also those that place too much trust in scripted manual test cases. When one has the view that test execution is nothing more than following pre-scripted test steps, then it's not surprising that they may think they can create "automated tests".
When I (and many others I know) refer to an "automated test", it usually means a set of test steps (based on a script or a behavior model) that have been codified in a way that a computer can execute the control and/or validation of whatever is under test. I view these automated tests to be tests like a lab test that a doctor might order. The "test" does not do the thinking for the doctor or tester. Theses tests provide information used by the doctor or tester to test the subject of their inquiry.
I fully understand that any automated test step execution -- whether based on scripts or explicit models -- is unable to perform the sapient tasks of a human tester. However, automation can provide useful information to human testers.
The question to be considered when implementing any automation is whether the benefits automation brings outweighs the costs of the implementation AND the costs of removing interactive sapience from whatever the automation does.
Ben Simo
QualityFrog.com
Hi Shrini,
Nice to read your post.
How I think about automation in testing is - automating certain areas of my testing activities to help in overall testing process. As Ben correctly pointed out, automation in testing is not just for test execution, we can use it in several other ways.
Talking practically, testing can not be completely automated, but there are certain areas in our day to day job as a tester which should better be automated to save time.
Automation is an aid to testing. When a tester develops certain piece of code to test an application, he can put certain intelligence in the code as well. He can code in such a way, that the outcome is a detailed report which he can analyze, which will employ his analysis skills. A tester should use automation in addition to his manual analysis of the application he is testing. This also aids in improving upon the existing automation framework.
Where we fail is that we automate certain areas of testing, and then forget to constantly review the same. We take it as a master piece which should not be touched. The automated testing framework like anything else needs constant reviews especially when it is to be used in an iterative manner for every new build of the application.
So, all I have to say is that I am not against automated testing but it should be used judicially. I guess that you will agree that automation is not bad, but it's employed badly most of the times. I look forward to writings from you to improve the automated testing process - the do's and don'ts and some precious lessons from your experience.
Regards,
Rahul Verma
Challenge me … think of ways where you can justify the use of “automated testing” – come up with definitions of “automation” and “testing” and link them !!!
I've been thinking about new terms for some time but haven't really been able to come up with something that makes sense.
On one hand, all software testing (other than possibly requirements and code inspection) is automated to some extent.
On another hand, all software testing requires thinking people -- even bad testing with all the thinking prior to execution.
The common uses for "automated tests" and "test automation" are usually limited to unit, release acceptance, and regression test execution. This is a very small view of automation.
I have no problem using the term "automation" to refer to software tools used in testing.
My dictionary lists trying something to find out about it and "determine the presence or properties of (a substance)" as a couple of the many definitions for "test". These definitions somewhat match what automated test execution can do for a tester.
This leaves out all the other forms of automation in testing; but then, I don't call those things "automated tests".
So, what else could we call automation used in testing? I'll have to think about that some more.
For now, I'll continue to encourage wise (sapient) use of automation in testing.
Rahul,
thanks for droping in and commenting ...
I urge you re-read this post ... This NOT at all a debate about Automation vs No automation/less Automation.
This is about usage of (wrong) Terminology "Automated Testing".
As you and Ben rightly pointed out ... All Testing can not be automated ..
then why use something (automated testing) that remotely indicates that such thing is possible.
My aim is to debate about some Ambigious and confusing terminologies in our community and keep stregthening our communication in the mass.
I take a stand that use of the word "Automated Testing" is MISLEADING and WRONG hence be avoided by those who understand testing and automation.
This term will still have supporters in groups like "Tool Vendors" "Quality-Process Groups", "Some development communities (such as agile)", "PM and others" ....
These people still think that "automated (software) testing is possible" and it exists ...
They need education ...
Shrini
Hi Shrini,
My aim is to debate about some Ambigious and confusing terminologies in our community and keep stregthening our communication in the mass.
I got your point and I very well agree with you.
My comment started with how I interpret automated testing by saying - "How I think about automation in testing is - automating certain areas of my testing activities to help in overall testing process." Rest of the comment was more of a general expression on automation and my views, rather than answering your question.
When I hear the term Automated testing, I interpret - "Automation aided testing" - because that's what the person who is using this term is practically doing.
You are quite good at coining/improving terminologies ( as seen in your post on boundary value exploration), so I will wait till you do similar justice to the area of automated testing as well. I hope that the same triggers next level of discussion.
Regards,
Rahul Verma
Hi Shrini,
Nice to see this post and the comments which follow.
I would agree to Bin Simo's point of view. In addition to that I think A+C option which you pointed out most closely fits the bill. If we think "automation" in terms of doing everything for us(starting from plan-->design-->defect analysis) then it would be a mistake on our part.
I simply define it as a process where some manual tests performed by us are done automatically that for me is "automated testing". We just CANNOT rule out the manual intervention at any place of our testing cycle.
I very well agree with the current usage of this terminology(though I am not linked with any vendor :))
Shrini,
I've been thinking about another term for "automated test". Then it hit me that there is a term that I've heard Harry Robinson use many times that applies to how I believe automation should be used in testing.
Think of the Six-Million Dollar Man. Think of Dr. Octopus from Spiderman. Think of Inspector Gadget -- err, maybe not the best example of a cognitive being.
Think of heart pacemakers. Think of cochlear implants. Think of today's robotic artificial limbs.
Think of word processors. Think of integrated development environments. Think of spreadsheets. Think of software.
All of these things blend man and machine into a working system that is better than the sum of the parts.
The machine part of the system enables the human part of the system to accomplish tasks in ways that could not be done without the machine. The human part of the system provides the intelligence.
What is this called in science fiction stories?
A cybernetic organism. A cyborg.
Robots may be able to follow a script, but the combined power of man and machine in a cyborg can explore.
Automation-assisted testing is cyborg testing.
Now I've got to go finish coding the machine part of a cyborg test.
:)
Ben Simo
QualityFrog.com
I think instead of changing the name or coming up with new terminology, it is easier to change the way people think about terms..If I compare development in the past few years, automation testing was equivalent to replacing human testers.. we have made progress and now you will rarely see this notion attached to automation or automated testing.
I feel that people have begun to
understand that automated testing should really mean effective use of tools in testing.. test automation is also a software that you need to use effectively.. Personaly I use automation to a great extent wherever I see any value.. but I never use them to replace testing performed by humans..
I use automation to make sure that focus can be on the real tasks and not on supplementary tasks.. I also use automated testing along with my build process to make sure that time is not wasted on bad builds..
I still remember struggling with the bad builds couple of years ago.. Now I can use tools like FitNesse / CruiseControl etc to make sure that automation is effective and productivity is improved.. Let automation testing remain automation testing and change what comes in mind when you use term automation testing :)
Geek
WWW.TestingGeek.Com
>>I think instead of changing the name or coming up with new terminology, it is easier to change the way people think about terms..
I think it is other way round ... I would like to change the way people think by using new terminologies and consistently using them. At the same to putting existing terminologies to test....
What ever terminolgies we are going adapt has to survive the scrutiny of the community I represent ...
>>> I use automation to make sure that focus can be on the real tasks and not on supplementary tasks.
I don't understand fully your usage of the words as real and supplimentary ...
>>> change what comes in mind when you use term automation testing :)
I think the change comes by continuously allowing our terminologies to debate and test and updating and discovering new ones .... Not by suppressing the need for new ones, related debates and finally
an analysis of existing (abused) terminologies) ...
Shrini
Doug Hoffman and I adopted the phrase Computer-Assisted Testing several years ago (1990's) when we were developing a course on test automation architecture for UC Berkeley Extension.
I wrote a paper out of that work, at http://www.kaner.com/pdfs/testarch.pdf
One of the most useful things in that analysis is that once we got past the notion of "automating testing" and got to the task analysis, we were able to see a wide range of tasks and a wide variety of tools that were available to help with specific tasks. Rather than looking for automatic testing, we found tools for efficient, computer-assisted testing.
Well said Srini .. Human is Human !
Anyways, today automation really requires excellent tools to support the automation as well.I know the nights Ive worked out to find out a good pattern recognition hardware , audio recognition hardware for testing embedded television software..
I have seen the hard days of having done an automation in the bad ways and taking the learnings into making automation the right way .. a good automation should help the test engineer rather be a burden to work with .. I agree everything cannot be automated .. Thats why we have semi automation atleast a controlled automation ..
automation is software I agree , but when we develop software for testing we can call it test software ,but test software I dont think will become test automation .. from my view point .. its auto execution of the testcases known as automation and the software used for this should be called as automation software ...
what is your thoughts on this ?
Automated testing may not be rightly used by people to understand what they mean ... giving various view points to the same ..
testing cannot be automated until it is meant to be automated in all favourable factors..
when testcases A , B , C , D can be automated, it means by auto running the sequence of these tests we can ensure regression stability ..
auto running of testcases reduce human intervention using tools to facilitate the same which can be automated testing ..
test automation can be the process of making the whole system of automated testing ..
automated testing should never be defined from the test perspective of pass of fail .. it should be used just to mean that the test sequences are now executed by machine rather than man.
Nice topic to talk about....i am currently working for a s/w dev. company. We are trying to build a 'Testing Automation' team.
What we have found is that anything that needs to be done more than three times is worth getting automated. Also if you need to work on large amounts of data then also automation is a candidate.By automating the testing the testing time can be reduced to about 25%.Also, you can run the automation scripts over night unattended.
These are still in our dream and we are hoping that the mystery will get solved.
Post a Comment