Ten Software Testing Myths ...

I was reading Lidor Wyssocky’s blog and his post on 10 software development myths. I thought, let me re-do this list for software testing – replace words “development” by “testing” and other relevant words. Here is the list … Bingo … we, the software testers *nearly* share the list of myths (frustrations?) …

It is interesting to note that last 5 myths go unchanged … development and testing share the honor. I even doubt that Lidor might be a software tester or a developer having a strong tester like mind.

10. The tester’s task is easy: he should merely write and execute the test cases by translating requirements to test cases. Additionally log some bugs.

9. Every test case is documented. Otherwise, how on earth can we expect to do regression testing and in general repeat testing?

8. Test case Reviews are a one-time effort. All you have to do is take an artifact after it is completed, and verify that it is correct. Test case reviews, for example, should merely verify that *all* requirements are covered by test cases and EVERY REQUIREMENT is COVERED by AT LEAST ONE TEST CASE.

7. Software Testing should be like manufacturing. Each of us is a robot in an assembly line. Given a certain input, we should be able to come up automatically with the right output. Execute a set of test cases (should execute 100 test cases a day) and report pass/fail status.

6. Software Testing has nothing to do with creativity. Creativity – what? The only part which requires creativity is designing your assembly line of test case design. From that point on, everyone should just be obedient.

5. Creativity and discipline cannot live together. Creativity equals chaos. [This one remains unchanged from original list of software development myths]

4. The answer to every challenge we face in the software industry lies in defining a process. That process defines the assembly line without which we are doomed to work in a constant state of chaos. [BIG ONE …This one remains unchanged from original list of software development myths]

3. Processes have nothing to do with people. You are merely defining inputs and outputs for different parts of your machine.

2. If a process is not 100% repeatable, it is not a process. Letting people adapt the process and do “whatever they want” is just going back to chaos again.

1. Quality is all about serving the customer. Whatever the customer wants, he should get. Things that don’t concern your customer should not be of interest to you.

Like Lidor, can say "As I said, I guess we still have a long way to go…" ?

Shrini

this app can break ... Notepad Bug challenge ...

Jon Bach is another brilliant thinker, speaker and coach in the area of software testing. Along with his "illustrious" and "one and only" James Bach - Jon has been doing some pioneering work in the area of exploratory testing and human aspect of testing.

While I was reading Jon's interview by another famous "Michael", the braidy tester - Micheal Hunter , I bumped into this interesting bugs that Jon mentions. This happens to be Jon's one of the most interesting bugs.

In words of Jon "Run notepad.exe and type "this app can break". Save it as test.txt, close it, then open it again. You might see that it now shows all rectangles, as if the font couldn’t be loaded."

There are other mentions of this bug here ( "Bush hid the facts")

I bet as you read this, you will be trying see if bug appears --- Bingo ... what an interesting bug ...

I did some very initial investigations around "nailing down" the bug - I posting them as they are -- I can not wait until I completely nail down the bug. It is too tempting for me to blog about it....

Successful cases
===================

this pap can break
this app can creak
this app nan break
thin app can break
shin app can break
shis app can break

this app can breal

aaaa aaa aaa aaaaa
bbbb bbb bbb bbbbb
aaaa bbb ccc ddddd

1111 111 111 11111 (interesting)


Unsuccessful ones
=====================

1a11 2b2 3c3 4ddd4
1aaa 2bb 3cc 4dddd
this cap can break (interesting)
THIS APP CAN BREAK (interesting too)

Few other observations
======================

Does not happen second time in a same file
Does not happen if a file with garbage appears once and create a copy of the same and open and check.

Investigation continues …..

What are your takes?

Finally, when you give up and do not find any more ideas as what is happening -- See here , here (good ones) and here... there are some interesting discussions and paths of investigation …. But don’t read (cheat) until you give your best try … I would say using Google would also amount to cheating …

BTW -- have seen following testing challenges put up in blog world in recent days?

1. Elisabeth Hendrickson’s Triangle challenge
2. Mathew Heusser’s challenge


Shrini

Ways to make Test Automation difficult ...

One of the interesting combos in Test automation is “Test design and Extent of achievable Automation”. I have seen in several automation projects that I worked, involving commercial automation tools that test design is the single most important aspect that decides the extent of automation one can achieve in a automation initiative. Techincal feasibility which is decided by the tool or tool set is another important parameter that affects the reach of automation

The context here is that test cases are developed first with the intent of manual execution and later picked up for automation. This is a very common scenario that is prevalent today.

Let us consider two following situations or examples that indicate two distinct and extreme styles of test design

Design 1: “Understanding tests require domain knowledge” - Tests are designed and documented in a style that suitable for someone who is expert and has a familiarity with the application. Abbreviations and shortcuts etc are used very commonly. For a tester who does not have previous experience (they call it as domain knowledge), the tests does not make any sense and look like incomplete. Objective here is to write tests in quick time so that only a select few can understand. An analogy would be a prescription written by a doctor that only a pharmacist can read and understand (this is a regular practice for the Doctors in India). Test design of packaged software apps, ERP applications is seen to follow this approach


Design 2: “Even college students should be able to execute the tests” – This is other extreme of test design where, the tests are detailed to a greatest detail possible so that each documented test is complete unit. Some of these tests even start from “Start windows ….” The intent of test design is to reduce the time required for a new person to understand and execute the tests. The drivers for such a test design/documentation style are high resource turn over and high cost of testing resources. IT organizations with very small or no testing teams on their own seem to follow this approach of test design.

Now, which test design style would be suitable for “Automation”? Which test design style lends itself for automation?

I would say none. In my opinion, both of these test design styles are excellent examples or ways of making the automation “difficult”

Important questions in this case would be “What is testing mission” and “what are objectives of Automation”. Unfortunately, the proponents of both of these styles of test design tend to down play the importance of these questions.

An excellent example of “context free thinking” …

Quoting James Bach --- “What makes something not context-driven is partly that the methods applied don't solve the problems at hand. This is the most common situation I encounter with large projects: somebody thinks that documentation is important, but they don't think much about whether the documentation they produce is really helping the situation, or rather hurting it.”

-- Shrini