Sunday, April 22, 2007

Ten Software Testing Myths ...

I was reading Lidor Wyssocky’s blog and his post on 10 software development myths. I thought, let me re-do this list for software testing – replace words “development” by “testing” and other relevant words. Here is the list … Bingo … we, the software testers *nearly* share the list of myths (frustrations?) …

It is interesting to note that last 5 myths go unchanged … development and testing share the honor. I even doubt that Lidor might be a software tester or a developer having a strong tester like mind.

10. The tester’s task is easy: he should merely write and execute the test cases by translating requirements to test cases. Additionally log some bugs.

9. Every test case is documented. Otherwise, how on earth can we expect to do regression testing and in general repeat testing?

8. Test case Reviews are a one-time effort. All you have to do is take an artifact after it is completed, and verify that it is correct. Test case reviews, for example, should merely verify that *all* requirements are covered by test cases and EVERY REQUIREMENT is COVERED by AT LEAST ONE TEST CASE.

7. Software Testing should be like manufacturing. Each of us is a robot in an assembly line. Given a certain input, we should be able to come up automatically with the right output. Execute a set of test cases (should execute 100 test cases a day) and report pass/fail status.

6. Software Testing has nothing to do with creativity. Creativity – what? The only part which requires creativity is designing your assembly line of test case design. From that point on, everyone should just be obedient.

5. Creativity and discipline cannot live together. Creativity equals chaos. [This one remains unchanged from original list of software development myths]

4. The answer to every challenge we face in the software industry lies in defining a process. That process defines the assembly line without which we are doomed to work in a constant state of chaos. [BIG ONE …This one remains unchanged from original list of software development myths]

3. Processes have nothing to do with people. You are merely defining inputs and outputs for different parts of your machine.

2. If a process is not 100% repeatable, it is not a process. Letting people adapt the process and do “whatever they want” is just going back to chaos again.

1. Quality is all about serving the customer. Whatever the customer wants, he should get. Things that don’t concern your customer should not be of interest to you.

Like Lidor, can say "As I said, I guess we still have a long way to go…" ?

Shrini

12 comments:

alan said...

Could you include some examples of companies, or papers or blog entries which demonstrate any of these myths?
I lead an admittedly sheltered life, but I haven't seen or heard of any testers doing these things in years (if not decades).

Shrini Kulkarni said...

Alan,

Thanks for being so quick (first one) to comment on this blog.

Examples of companies - I do not want to go public on my blog to give the names. But I am pretty sure many of the myths are still very active. Naming the companies could lead me into a legal problem.
As such, no software company of any real reputation would admit about existence of such myths in their rank and file.

Papers? Do you mean papers on the web that support the existence of myths? or the ones that support their non existence?

Blogs ? same as above.

I am not sure about blogs that refute the existence of such myths however the blogs and writings of James Bach, Cem Kaner, Michael Bolton and others in Context driven Testing community, at several instances have highlighted the existence of such myths. Examples are so many that I just can not pin point to one


Shrini

Michael said...

I've heard these myths and misunderstandings at conferences, in my consulting, in classes, in chats with other bloggers and mailing list subscribers. Worse, years ago, I even used to believe in some of them.

The most common expression of a number of the myths is the notion of test automation as an unquestioned good; that whatever amount we have now, more would be better; that all testers have to be programmers. (That's a special case of the general problem: if X solved a problem for someone at some time in the past, we should always do more of X without asking why, and without asking questions about cost vs. value.)

Without naming names, how are things done at your company, Alan? Shrini, do you have any contrasting experience? Again, not names but specific stories might be enlightening.

---Michael B.

Anonymous said...

Just among the companies that hire me-- and those companies are probably more progressive than average-- I would say these myths are commonly held beliefs.

Much of the arguing that I do online and in conferences is related to such myths. I argue a lot.

I associate most of these myths with what I call the Factory School of software testing.

However, I think the post would be improved, Shrini, if you said more about what's wrong with these myths. What makes them myths?

-- James Bach

Debasis said...

Hi Shrini,

I am a silent regular reader of your blog. :-) This is a very very enlightening post on the common myths/misconceptions, prevalent in testing world. Thanks for this excellent post. Recently I have blogged about a similar topic here: http://software-testing-zone.blogspot.com/2007/04/software-testing-from-testers-eye.html . Please do come up with your valuable comments/suggestions/feedbacks. Thanks in Advance.

Regards,
-Debasis.

Alan said...

As I mentioned, I lead a sheltered life. I used to hear some of these myths, but haven't heard most of these in years. I do believe you when you say you see this, I just wanted to confirm the myths were viewed recently, and weren't something you experienced years in the past. (btw - I doubt anyone would consider legal action if you said their approach to testing sucked, but I guess you never know).

Just re-read the comments, and agree with James that these myths line up well with the Factory school as described by Petichord. I talk to testers at a lot of software companies (but only occassionally in IT shops) but haven't run across Factory testers (again, I guess I'm either sheltered or lucky). At my company, for example, I'm fortunate to have none of those myths be true.

The myth that Michael brought up (more automation == more goodness) is one I see _all_the_time_, and seems to be a hot topic these days. Ithink that testers should automate what makes sense to automate (yes, I know - bad answer). Automation choices are risk based (and even context based) decisions that will be different for every single project.
(and no, all testers don't need to be programmers, but there are advantages to having testers that can program. More here.

I'm kind of inspired to create my own ten myths list based on what I see. If I get it done, I'll run it by you for some comments.

Thanks for letting me sound off.

Shrini Kulkarni said...

Thanks Alan,

I will look forward for your list of 10 ...

Like I said, I wrote these 10, just like that from Lidor's list on Dev 10 myths. I see most of these day in day out around me. Trust me this is my practical experience.

Shrini

Shrini Kulkarni said...

James,

Thanks for posting your comment here.

I was initially thinking, that this list of 10 as myths would down easily in the community and did not think that someone would question them and their existence ...

Why these are myths and what is wrong with them ... that would definitely fill another blog post in itself. I will start writing on that ...

Shrini

Ben Simo said...

These myths still exist today. This week, I have heard statements based on numbers 9, 7, 6, and 4 from management; and, I've heard #1 from testers.

Ben Simo
http://QualityFrog.com

TestingTimes said...

Hi !
I am Yuvraj Singh and have just started my first blog.Would sincerely appreciate your feedback , criticism or comments ! I need your support and of course your comments !!!

Please visit -->www.questest.blogspot.com

Looking forward to your "posts"

Regards,
Yuvraj Singh

QA Tester said...

Hi Shrini,
You have showed us the real time scenario testers are facing that they need to follow the disciple blindly most of the times.

Sally Sen said...

Hey Shrini,
It's a wonderful blog you got out here.
Thanks! for all the sharing of and for testers information and also, the blogs that you read. It just get more interesting when browsing throughout the blog.
Again, thanks & keep up the good work.

Regards,
Sally Sen
Software Testing Companies