A mystery called Automated Testing ...

"No device has been invented which can match the human eye for accuracy and precision in many tasks; nor the human ear. Even the admittedly handicapped human is able to identify and distinguish among far more scents than any automated device. Human pattern recognition, language recognition, and language production ability is well beyond anything currently envisioned by automation engineers."
- Answers.com

Hope you have heard many times the word “Automated Testing” (Some places I even have heard “automatic Testing”)

Have you ever thought – what does that mean? Let us dig little and try to figure out what each term in above phrase *might* mean?

We have two words in above phrase – “Automated” or “Automatic” and “Testing”. Let us explore possible meanings of these terms ….

(A) Automated – something that happens without any human intervention – something that is done by a machine-software program or by self.

Answers.com says -
"automated" (adj )
Definition: made or done by a machine
Antonyms: by hand, manual

Automation (ancient Greek: = self dictated),

The word testing has many meanings – I present two contrasting definitions

(B) Testing is an act of technical investigation performed on behalf of stakeholders in order to reveal quality related information – it is an act of questioning – An infinite search for problems – problems that can annoy/irritate/destroy/damage a stakeholder.

(C) Testing an act of executing a test case derived from a specification and verifies whether the test case passes or fails.

(D) It is an act of confirmation that software behaves in a way as prescribed by the underlying specification/design.

(E) Testing proves that the software works as desired.

Now think about constructing the meaning of the word “Automated Testing” and we have these definitions (A), (B), (C), (D), (E) …

Automated Testing = (A) + (B) - Possible? I believe No!!!
Computer/Program can not question, improvise, learn, adapt, think, emphasize

Automated Testing = (A) + (C) – Seems possible – but in a narrow way. Using another program (automation tool) one can execute an automated test … but that is only a part of whole story – what about Test design? Defect investigation? And host of other activities that fall under testing umbrella …

Automated Testing = (A) + (D)
Automated Testing = (A) + (E)

Both of above seem to be “Not possible” as checking or proving that software works the way intended or as per spec (writing in English – hence can be interpreted in infinite number of ways) using “automation” – solely – is not practically possible.

So … think twice before using the term “Automated Testing” – as nothing really EXISTS in this world of testing called as such …

Challenge me … think of ways where you can justify the use of “automated testing” – come up with definitions of “automation” and “testing” and link them !!!

BTW what do you think about "automated tests" ??

Shrini

A Blogger Bug

A bug in blogger Create Post --

The title field is "non editable" - hence this post and my previous posts are going without any title for the post ...

I am investigating this -- any clues?

Update 13th July : The title field becomes editable when (this is one the ways I figured out) I edit (quick edit from the blog ) the post with the blank title.

Shrini

Regression Testing and Insanity ....

"Insanity is doing the same thing, over and over again, but expecting different results"
Albert Einstein (attributed to Ben Franklin too)

Does this sound similar to what we do in the name of "regression testing”? In regression testing (more so in that regression testing whose test execution is automated - note I am not using the term "automated regression testing") , we believe that by doing repetitions of some things that we did previously - we can find new results.

Thanks to inherent complexity in software and involvement of humans in development, testing and usage of the same - this regression insanity appears to be *working*

This one of those Francis Bacon's idols - Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

Read following to know more about Francis Bacon and how his idols are relevant in testing

Francis Bacon’s New Organon (James Bach)
Bacon and boundary testing (Mike Kelly)
The four Idols of Francis Bacon
The Four Idols of Sir Francis Bacon by Ben Chambers and Zeb Dahl.


More on this later...

Shrini

My acrobatics on the Boundaries ..

The boundary condition of the universe is that it has no boundary." The universe would be completely self-contained and not affected by anything outside itself. It would neither be created nor destroyed. It would just BE

Stephen W. Hawking - A BRIEF HISTORY OF TIME

Science is a differential equation. Religion is a boundary condition- Alan Turing

Boundaries are actually the main factor in space, just as the present, another boundary, is the main factor in time- Eduardo Chillida

Earth has its boundaries, but human stupidity is limitless – Gustave Flaubert

I would read above quote from French novelist of 18th century (Flaubert was regarded as the prime mover of the realist school of French literature and best known for his masterpiece) by replacing the word “Stupidity” with “intelligence” or “imaginative power”.

Mike Kelly has an interesting post related to boundary testing here

I blogged about this topic few months ago under the name BVE and I think it makes perfect sense to link these two posts and get a new perspective of boundary test design (part of domain testing)

Note following important points in Mike’s post (with my views/comments) ---

1. Understanding, Identifying and working with boundaries in software - is a Modeling problem. There can be multiple ways in which boundaries in a software system can be modeled.
2. Depending upon how *close* your model models the system behavior, your (boundary) testing can be incomplete or wrong to *that* extent. Since there can be boundaries out side your model – you will not notice them.
3. There can be (are) multiple boundaries - what you notice is limited by your model (ref. #1)
4. No boundary exists in isolation
5. All boundaries, those that are explicitly identified in model – hence are known to you and those laying outside your model, INTEREACT and ALTER/AFFECT system behavior in some *significant*. This introduces complexity in the model. In other words there exists some RELATIONSHIP between these boundaries.
6. Inputs in identifying (in fact for modeling) the boundaries come from various sources such as technical specifications, user expectations, requirement specifications, device/OS specifications. If you narrowly focus on any one source of information – you will miss modeling of others boundaries (outside the mode) hence you would miss bugs/system behaviors associated with them.
7. A weak boundary Analysis would start with boundary first without first having a thought /explicitly/ about my model.
8. A strong or useful boundary analysis would start with model and identifying boundaries resulting from all possible sources of information.

Don’t forget to check following ---

All models (those especially having predominant influence of domain testing) are APPEAR to be deterministic (algorithmic/mathematical representations - set, graph, state machines etc) but indeed are HEURISTIC models as act of modeling seem to follow (in most of the cases) a heuristic approach…


Shrini