Friday, April 13, 2007

Ways to make Test Automation difficult ...

One of the interesting combos in Test automation is “Test design and Extent of achievable Automation”. I have seen in several automation projects that I worked, involving commercial automation tools that test design is the single most important aspect that decides the extent of automation one can achieve in a automation initiative. Techincal feasibility which is decided by the tool or tool set is another important parameter that affects the reach of automation

The context here is that test cases are developed first with the intent of manual execution and later picked up for automation. This is a very common scenario that is prevalent today.

Let us consider two following situations or examples that indicate two distinct and extreme styles of test design

Design 1: “Understanding tests require domain knowledge” - Tests are designed and documented in a style that suitable for someone who is expert and has a familiarity with the application. Abbreviations and shortcuts etc are used very commonly. For a tester who does not have previous experience (they call it as domain knowledge), the tests does not make any sense and look like incomplete. Objective here is to write tests in quick time so that only a select few can understand. An analogy would be a prescription written by a doctor that only a pharmacist can read and understand (this is a regular practice for the Doctors in India). Test design of packaged software apps, ERP applications is seen to follow this approach

Design 2: “Even college students should be able to execute the tests” – This is other extreme of test design where, the tests are detailed to a greatest detail possible so that each documented test is complete unit. Some of these tests even start from “Start windows ….” The intent of test design is to reduce the time required for a new person to understand and execute the tests. The drivers for such a test design/documentation style are high resource turn over and high cost of testing resources. IT organizations with very small or no testing teams on their own seem to follow this approach of test design.

Now, which test design style would be suitable for “Automation”? Which test design style lends itself for automation?

I would say none. In my opinion, both of these test design styles are excellent examples or ways of making the automation “difficult”

Important questions in this case would be “What is testing mission” and “what are objectives of Automation”. Unfortunately, the proponents of both of these styles of test design tend to down play the importance of these questions.

An excellent example of “context free thinking” …

Quoting James Bach --- “What makes something not context-driven is partly that the methods applied don't solve the problems at hand. This is the most common situation I encounter with large projects: somebody thinks that documentation is important, but they don't think much about whether the documentation they produce is really helping the situation, or rather hurting it.”

-- Shrini


Ben Simo said...

The context here is that test cases are developed first with the intent of manual execution and later picked up for automation.

I believe that this is a big reason for many test automation project failures. As James Bach wrote in his blog last summer, a good manual test cannot be automated. I would then argue that a great automated test is one that cannot be performed manually.

Automation for the sake of automation is destined to fail. Automation that does not ask -- and then respond to -- the context questions is destined to fail.

Test automation can work when applied in ways that assist testers -- and that varies depending on the context.

Rahul Verma said...

Hi Shrini,

This is an interesting post.

It would be great if you extend it further to discuss what test design you suggest instead.

I have been a performance tester for the last of couple of years and we used to have a "Traversal document" for business flows, with screen snapshots and notes on required test data. We never used traditional test cases. Most of the times it was prepared as a part of walkthrough of the application and sent along with MOM for client approval. This helped because in performance testing, we are mainly concerned with the request/respoinse format and not GUI operations.

Looking forward to your response,
Rahul Verma.

Dinesh Karmankar said...

Hi Shrini,

Well thought and real time test design examples which are difficult to automate.