Sunday, September 30, 2007

More on definition of Test Automation ...

While discussing about "definition" of "test automation", with Ben Simo I left a comment at Ben's blog post. I am sharing that here for the benefit of my blog readers ...

(You would also find some debate about "model based testing" in comments)

Test automation involves - Automation Design and Execution of automated tests ...

Design: An act of translation of some question that one asks regarding a feature of an software application into some programmable instructions so that the question being asked is modeled with reasonable accuracy (courtesy: Michael Bolton) In simple words translating a testing question into a program

Execution: Use of a machine (hence a set of computer programs) to support any aspect of Testing with Testing at the center of whole scheme of things.
(derived from the definitions by James Bach and Michael B)

As I continue to write about definitions and terminologies around test automation, let me reiterate - Nothing called "Automated Testing" exists in this world....



Saturday, September 29, 2007

Simple things and me ...

World likes simplicity – Pictures, Colors, Sound, Odor are used to abstract some complex behaviors.

Consider following –
Software Project Status – Red, Green and Yellow
Project Size – Small, Big, Medium
Test cases – Simple, Medium, complex
Testing – Gather requirements, write test cases (some times even prepare them), execute them, check if the pass, if pass report and go home and if fail log a bug and go home

Automation – Create an automated script using the state of the art Automation tool. Store the script in test management tool and execute it from it. See the result logged automatically. If script fails (I mean reports failure), automatically make another script to log the bug

Knowledge Transfer – As simple as Fund transfer in bank. Transfer the relevant documents to the subject – transfer. If there are clarifications there always our best friend “issue list”, “action item tracker”, “clarification list” – many names but all is one.

Knowledge acquisition – As simple as one country invades and acquires another country. Just acquire – by force

What is common between each one of these – simplicity.

I become restless when

- People tell me about testing project status in terms of Red, green or yellow.
- People start approaching testing as sequence of actions – test design, execution, bug logging and regression
- People start describing a bunch of test cases or software features as Simple, medium and complex – SMC model.
- Testers getting classified as great, lousy based on the number of test cases executed or number (not the type) of bugs logged.

To me, all of these are simplifications of some complex things that we are trying to understand. My frustration is “Why things can be so simple”? “There must be something that is hidden behind this simple thing”

Imagine, if Newton were to think – when apple fell on his head – “No big deal – it had to fall so it fell down – Gosh – my head is aching. Why did sit bellow this tree?” Same thing applies to Archimedes. Jump out bath tub and run …?

While thinking about simplicity about some things that we deal with helps us to get started off – important to understand that it was just a beginning.

While testers and most of the managers are happy about such simplifications - a skilled tester is always worried about such simplifications and often attempts to find the loop wholes in simplified models, notions, beliefs, description etc…

Nature’s simplest things have hidden inside then greatest and deepest mysteries. That would a fascination journey – we just need vehicle capable moving into it – Our imagination and curiosity. Today’s skilled testers are blessed with this imagination and curiosity. The journey has just begun …

Do you think post has horribly and outrageously simplified the seriousness and intent behind the post? Well, I can not be too serious …

Thursday, September 27, 2007

Automation Dreams - Thinking about END in the begining ...

"Cherish your visions and your dreams as they are the children of your soul; the blue prints of your ultimate accomplishments."

- Napolean Hill

Thinking about “end” in the mind helps me most of the cases such as planning, checking and carrying out various day –today tasks like performing 4-5 interconnected tasks, leaving home for a week/month long journey, arranging my daughter’s birthday etc. Thinking about “END” while you are about to begin or at the time of beginning a task - can help you to visualize the entire stretch of journey from start to end, anticipate various milestones, problems and as many as details as possible. This can prove to be a powerful mind modeling concept if you train yourself in visualizing from the END and work backward from there to the beginning.

Let us apply this to Test Automation …..
- How do you describe a successful automation?
- How can one describe steady state automation in deployment?
- Can you trace a journey from the beginning of an automation initiative and the end where your automation has become “obsolete” or has reached a steady state?
- Is there any END for Test Automation as a software project?

Automation initiatives have end goal(s) to achieve – cut testing cost by certain percentage, increase test coverage by certain percentage etc. In a way every automation project has some dreams to realize. Can we describe those dreams related to automation?

Let me give it a try ….

I will have an automation suite that runs “unattended” for up to 8 hours. (Length of Execution)
My automation suites covers 40% of my regression tests for product X. (Automation coverage)
I will have an automation suite whose results I can trust the most or When my Automated Test fails – I am sure that is bug (Reliability and Trust)
I have 40% of my test cases automated. Hence my testing effort now on will be 40% less (You can dream and these are DREAMS …. )
My smoke test suite is automated – I can now make changes to the code more frequently even when there is time crunch.
And so on ..

What are you Automation Dreams? Can you describe them?

Thursday, September 13, 2007

Physicians, Surgeons, X Ray Lab Technicians ....

Heard of this rant?
“We would like our business users, domain experts and subject matter experts write automated scripts (tests?)”
“Our next generation automation tool – allows persons with ZERO programming and testing knowledge generate automation scripts in MINUTES”

Think about it …. Will you ask a Physician (Medical Doctors) or Surgeon to work as XRay or Ultra Sound Lab Technician and vice versa? Or An Aneathesian to perform a surgical operation?
Each one in professional life are known to have some core competencies. Best bet will be to utilize each one to their strengths….

Business users know business domain while Automation engineers and Tester know programming and software testing.

IMHO, asking business users to do “main stream” testing and automation is sure recipe for “failure”.

Any views? Do you think the analogy presented here is logical?

More as I hear from you ….


Tuesday, September 11, 2007

Inattentional Blindness ...

Sajjad has an interesting article on "inattentional blindness" and Exploratory Testing here

Sajjd mentions that - "Exploratory testing is supposed to be better at minimizing inattentional blindness". This kicks off a thread of thinking in my mind - Why it might be so? what elements in ET helps one to minimize inattentional blindness?"

Let me take a guess and answer --
May be it is "thinking between alternating polarities" while doing ET - doing vs explaining, fast vs slow, reading vs doing, focussing vs defocussing. In my opinion inattentional blindness happens due to "heavy" focus on one or more "atomic" aspects of bigger object under observation. one possible solution is to defocus as often as you can.

A related phrase (a kind of antonym) I use is "your eyes will see what you would like to see". This is more dominant theme in scripted testing where a tester is "pre-programmed" observe only expected results.

Here is a wikipedia page

Do not forget to read James Bach's comments for this post -

"Testing is a lot like fishing. You try to create conditions that maximize your chances for catching something tasty. Exploratory testing is a little more like fishing, because just as with real life fishermen ......"