Sunday, April 19, 2009

10 ways to reduce cost of software testing

In current economic situations, IT folks worry about one thing – “reduce cost”. I have been frequently asked “how to reduce testing cost”. A no brainer answer would “do not do testing … at all”. How many buy this idea … can current breed of IT applications sustain with less or no testing at all? When I use the term testing –I am referring to “non-programmer” testing.

Here is my draft list of suggestions ...

1. Closely work with developers, do some parallel testing with them as the product/feature is getting developed
2. Identify and eliminate non-testing activities that occur in the name of process, documentation, management, metrics etc.
3. Analyze and profile every application under the portfolio to determine “stable” and “well tested” areas of the application. These areas should receive the least or no testing effort.
4. Analyze the test scripts suite and remove redundant, worn out ones. Aim to reduce scripted test repository as small as you can.
5. Review and reduce “regression testing” on the basis of “well tested/stable areas” of the application
6. Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
7. Plan testing in small but frequent cycles (Session based exploratory testing approach) – reduce planning and management overheads
8. Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
9. Cut down on lengthy test plans, testing reports, dashboards – switch to simple but frequent test reporting.
10. Simplify defect management process – reduce defect life cycle – resort to informal/quick defect communication.


Some this advice might look like a simple common sense (eliminate waste, focus on tasks that impact end result DIRECTLY). With so much selling happening about “testing tools”, “factory models”, “cheap and best testing services” – any common sense is difficult to come by.

How would IT community react to these suggestions – most likely response would “This would not work, how can we reduce testing, those test cases, processes, metrics, management practice?”. These suggestions would be most likely to be rejected on the grounds that testing cost needs to be reduced without “compromising quality”. Many IT folks think that quality comes from test scripts, processes, metrics, testing tools, automation etc. I am afraid quality is not such a simple thing.
Again, there are no free lunches here … if you are thinking about reducing cost of testing, there are always risks of impacting quality (roughly goodness or confidence in the product) in one or other way. If you approach the problem (cost vs quality) from a quality side (improve testing - test better, deeper and wider), then the chances of achieving good quality and also “some” cost benefits are more likely. However, if you approach it from “cost” side of the equation – you might do achieve that albeit some impact on overall goodness/quality of delivered product.

Note that some suggestions mentioned in this list call for some smart testers who can think on their feet, work with least supervision, least (optimum) documentation and processes and so on. I think, the focus should shift from process, tools, management, documentation to Skill. There can be problems in getting such resources in IT scenario (especially in outsourced/offshored world)

You have choice … which side you would like to approach the problem ..?

[update 20/Apr/2009] A colleague of mine reacted to this list saying "These are too risky suggestions and he would not recommend any of these. Business prudence is totally missing".

I think he was expecting to see some "low risk and high return" type of suggestions - like those "cheap" and "best" items. I still do not understand - there can be no risk free ways of reducing (testing) cost -unless you are totally spending like crazy without any thinking. We do not seem to have such risk free - free lunches - why fool ourselves and the client in believing such "non existent" things?

Another suggestion that came up was "Let us use standardized processes". How standardization can reduce cost? what is the cost bringing in standardization itself?
May be the expectation is that standardization will make each tester behave identical to another like robots. Are robots cheaper? may be? may be not .... They at least do not whine about working on week ends :)

Shrini

24 comments:

Hebbar said...

Shrini,

I think "housekeeping" is essential in both war time and peace time. It just so happens that the downturn and the likes pushes people to look heavily inside and internalize so called best practices. Best housekeeping generally ensures that the system is healthy, wealthy and meaningful. List you included go in this line very well and I think is another mirror important at these times....for sure!

Hebbar

Ashwin Palaparthi said...

Shrini,

You have hit the nail on the head. Focus must shift from "everything else" to the skill.

I am taking the opportunity to extend your point #2 in the below lines. My goal is not challenge the point but to "include" how cost can be saved by extending the same point.

Let us say, a tester has to test a text field with the below cases:

1. It has to be tested with multiple languages like English, Hebrew, Arabic and Kannada.

2. It has an upper limit for its length, say 40 characters, and the tester wants to prepare strings with 40 characters and 41 characters (and also need to know, if a string gets truncated, where it got truncated).

Naturally the tester may ask for a lot of time to prepare the inputs that drive these the tests.

It could be a tester who does everything manually, or a tech-savvy tester who want to develop scripts to accomplish the same, it takes time; and I consider that as "non testing time".

So, there is a desperate need for a tool that can aid the tester to do that.

Today, www.TestersDesk.com has a toolkit named "Common Test Data Generators" which help them on that aspect. It is free for the entire testing community.

As you can see, we named it as "common" test data generators instead of hyping the simple things it does.

What I am tasked in my life is to create an environment for testers where the time they spend is on the really needed stuff and off-load the rest to the tools.

Re-reading your 10 points, they make direct sense, and I will read them again (just to test what I am doing :-)).

Cheers!
Ashwin Palaparthi,
apalaparthi.blogspot.com

Markus Gärtner said...

From my experience Continuous Integration used in the development process also helps to cut down testings costs. Testing and development form a symbiosis in many areas here. Bringing together developers and testers is just the first step to realize doubled actions and decide whether or not to consider them a waste.

Your point on skill raised the again the question for the software testing craft inside me. We need to do a good job and work on our personal practices and abilities in a community around the globe.

Well written, mind provoking thoughts.

Michael Kelly said...

My particular favorite in the list:

"8. Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)"

I don't know what the fascination is with big expensive tools, but people love to spend money on them. They make sense sometimes, but most companies I've worked with barely use them.

Good post man.
Mike

Shrini Kulkarni said...

@Hebbar (not sure about your first name and I know 2 Hebbars, personally)

I am not sure how the word "house keeping" fits in the context of reducing the cost of testing.

I hope you are not equating testing to house keeping

Shrini

Shrini Kulkarni said...

@Ashwin,

I have been always fan of testersdeck and the tools there in. Due to the stuff I have been busy with I never got chance to lay my hands on the tool. I will surely do one day soon..

Coming to your suggestion of data coverage and usage of appropriate tools - Many are comfortable with COTS (costly) tools that claim to do magic and reduce testing. Most of us have experience what they can do and can not.

My #2 is about eliminating anything that is not Testing. All process, metrics, management etc. Let testers manage themselves - produce report that is just enough to serve its basic purpose.

It goes without saying that tools such as test data generator are pretty useful as they ease test data creation.

Again the caution is that do that minimum testing with or without tools that you can get away with.
Extraordinary situations call for extraordinary measures.

Doing minimum testing keeping all non testing away is one of several ways to reduce cost of testing.

Thanks for dropping in and sharing your views.

Shrini

Shrini Kulkarni said...

@Markus,

The problem with IT (especially outsourced, multi-vendor environment) is that development is done by more than one vendor, mostly these development groups do not talk to each other (conflicting business interests). Lots of governance and hand-off procedures are put in place to make sure that one development work can be given to multiple vendors with testing done by another vendor.

IT shops hope to setup a competitive environment by this and also reduce the risk of dependency on one vendor. The flip side is that there are many disparate development and testing groups doing development and testing - coordinating between them is a big time consuming task.

It is due to this dynamic scenario - traditional "good" practices like continuous integration etc can not be deployed in their entirety.

Again, in IT outsourced world, testers and developers (among many vendors) work with wall between them.

If they want to reduce cost --- I think they have to think about ways to break the walls and allow dev and test to work together.

Shrini

Ukkuru said...

Good Topic in the present context of economic slow down. Testers should try to write effective test cases. I have seen testers were the same test scenario is covered in multiple test cases. We should be able to prioritize test cases based on business risk and usage which is known as Risk based testing. Automation using commercial tools is not going to work out as clients will not be ready to purchase tools. But open source tools like selenium will be a good choice.Testers should use tools like the one available in testersdesk.com. Also try to develop a repository of frequently used test cases which can be readily used in any project....

Unknown said...

Hi Srini,
A very good blog.

Comments like "Let us use standardized processes" are counter productive and inhibits innovative ideas. While companies across industries are trying to find new ways of doing things to manage cost, I feel allowing people to innovate will only help the companies to reduce it.

I think one very good way to reduce cost and increase quality & productivity is to ASK THE TESTERS for their ideas. They are the frontline people who do the testing and can come up with ideas specific to their application/team. While there are some good suggestions about using open source tools and avoiding expensive tools it will work only for small companies and not for MNC's unless there is a strong proponent in the Senior Management to takeup the case.

These are my comments about your suggestions.
1. I agree. I also feel that testers should review requirements with BA's & developers. In my experience I have found gaps between testers and developers in understanding the same written word in english. Having a meeting and clearing their doubts initially will reduce a lot of rework.
2. I think we just need test plan, test cases & a simple report.
3. I do not feel this is correct. Just because the test coverage was good for one particular module during a release(s), we cannot reduce our test effort on these. What happens if throughly tested application bombs because of the newly added code! May be we could focus more on business critical areas and less on the 'not-so-critical' areas. And do a proper impact analysis (not as process, but in its essence).

Thanks for sharing your views!.

Unknown said...

Hi Shrini,
Thats a good write-up I must say...
But I am not quite sure as how can reducing on regression testing can reduce the cost of the software testing. So kindly elaborate on that. And I completely agree with you that, if one is up for providing software testing service then he should first understand that there is nothing called "risk free ways of reducing costs"..

Anonymous said...

You are a dickhead.

Shrini Kulkarni said...

Hi anonymous (a fellow dickhead)-

I know why you posted this anonymously....!!!!

find a better word ...I am very thick skinned ...

Do you have anything ELSE to say about this post?

Shrini

Shrini Kulkarni said...

@Alan,

>>>But I am not quite sure as how can reducing on regression testing can reduce the cost of the software testing. So kindly elaborate on that.

Alan - let me set the context, I am talking about IT services world (not software makers world like MS)... so please read the post as an IT manager.

Traditionally, regression testing (in IT world) is seen as repeating the old (tried and tested) suite of test cases again and again when software under test changes. At times this is viewed as an "over-head" as most of the time the result of such testing is "all tests pass" (unless developers screws up things pretty bad).

The realization in IT world is that *this* testing had to be done whether useful or not!! (you need to prove that software *works* as before or not !!!)

Now, this testing takes the large part of test cycle, hence reduction in this will lead to a significant reduction in testing cost.

That is why people say - automate your regression testing (usually done using COTS tools and applied at GUI of the application) to the max so that you can complete regression testing quickly and be happy to see "all pass" status.

But few of us know - testing is not as rosy as it is depicted here ....

Surprisingly - I have seen very few IT managers "challenging" this notion...

shrini

VIJAY said...

Most of the points you have drafted are more or less based on Agile Testing methodologies and i have some doubts in some of the points you have mentioned.

#2. You have said to eliminate all the documentation, management and metrics stuff from testing activity.
->But if you don't maintain these things, how would you trace your testing process, whether is it going in the right direction within timeline, within budget etc.
Also i feel you cannot answer your clients if something goes wrong in the testing process you follow, unless you have maintianed some amout of documentation about the work done wrt testing. And still in many companies they follow Handshake policies like after completing one phase of testing, acknowledgement are sought from the Clients and we proceed to the next level only after getting an approval.

#5 You have said to reduce "regression testing" on the well tested area.
->By making a Code change even the well tested / stable area are prone to get impacted by it. Hence it is always better to have the regression on the complete end-to-end scenario. And thats the whole point of having regression tests, i believe.

#6 Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
-> Do you mean here to replace Automation Testing with Manual Testing in some pain areas

#8 Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
-> As you said in your previous comments, lunches don't come for free ;). Every tool has its importance. The more costlier the tool, the more powerful it will be. There should be some reason why they have made a particular tool sell costlier and the intelligence in selecting a tool lies in deciding what we need from the tool, i guess.

Except for these points, everything was so good. Very good job indeed.

You are my ex-boss in iGATE(C&IS Automation-your baby). Do you remember me. Just joined the club.

Thanks for your time!

VIJAY said...

Most of the points you have drafted are more or less based on Agile Testing methodologies and i have some doubts in some of the points you have mentioned.

#2. You have said to eliminate all the documentation, management and metrics stuff from testing activity.
->But if you don't maintain these things, how would you trace your testing process, whether is it going in the right direction within timeline, within budget etc.
Also i feel you cannot answer your clients if something goes wrong in the testing process you follow, unless you have maintianed some amout of documentation about the work done wrt testing. And still in many companies they follow Handshake policies like after completing one phase of testing, acknowledgement are sought from the Clients and we proceed to the next level only after getting an approval.

#5 You have said to reduce "regression testing" on the well tested area.
->By making a Code change even the well tested / stable area are prone to get impacted by it. Hence it is always better to have the regression on the complete end-to-end scenario. And thats the whole point of having regression tests, i believe.

#6 Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
-> Do you mean here to replace Automation Testing with Manual Testing in some pain areas

#8 Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
-> As you said in your previous comments, lunches don't come for free ;). Every tool has its importance. The more costlier the tool, the more powerful it will be. There should be some reason why they have made a particular tool sell costlier and the intelligence in selecting a tool lies in deciding what we need from the tool, i guess.

You are my ex-boss in iGATE(C&IS Automation-your baby). Do you remember me. Just joined the club.

Thanks for your time!

Duncan Thomas said...

Shrini,
excellent post. Regression testing really has to be automated for it to be both effective as well as cost efficient.

All the best,

Duncan Thomas,
JadeLiquid Software.

VIJAY said...

Hello Shrini,

A good post. It elucidates most / all of the best practices in Testing Process.

I could see a touch feather relationship between the points and the Agile Testing Methodologies. But again you can only reshape the wheel which is been invented already :) ...

I am more concerned about points # 2, 5, 6 and 8. Can you please comment on my views about them. It would vastly help me to know about the topic better.

#2. You have said to eliminate all the documentation, management and metrics stuff from testing activity.
->But if you don't maintain these things, how would you trace your testing process, whether is it going in the right direction within timeline, within budget etc.
And still in many companies they follow Handshake policies like after completing one phase of testing, acknowledgement are sought from the Clients and then proceed to the next level only after getting an approval.

#5 You have said to reduce "regression testing" on the well tested area.
->By making a Code change even the well tested / stable area are prone to get impacted by it. Hence it is always better to have the regression on the complete end-to-end scenario. And thats the whole point of having regression tests, right ?

#6 Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
-> Do you mean here to replace Automation Testing with Manual Testing in some pain areas

#8 Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
-> As you said in your previous comments, lunches don't come for free ;). Every tool has its importance. The more costlier the tool, the more powerful it will be. There should be some reason why they have made a particular tool sell costlier and the intelligence in selecting a tool lies in deciding what we need from the tool, i guess.


Thanks for your time!

Regards,
VIJAY

Inder P Singh said...

It seems that your draft list of suggestions is broadly based on three premises: 1) Agile Software Development process, 2) Process improvement and 3) Re-use.

One other way to "optimize resources" (I would not say "reduce costs" because I aim to focus at results and not cost) is to look at testing resource deployment. This is important in the case of companies with multiple testers. Typically, testers have times when they wait for the development to complete, bugs to resolve and also the testers wait between testing projects and assignments. If the company is able to productively utilize this idle time (a couple of suggestions follow), more testing can be accomplished and consequently the risks related to the system releases would reduce.

a. Instead of allocating a tester to a single project, allocate him/ her to multiple projects. Of course, this assumes either careful planning by the test manager or that the tester having the maturity to plan their work and commitments responsibly.

b. Train and deploy multiple testers on a single project. During lean times, they can build the required skills and knowledge. During busy times, they can divide the work amongst themselves so that more testing is achieved without stressing out a single person. Further, they may back up other testers on the same project, in case of need, without delaying the project.

c. Quickly re-deploy an available tester on assignments/ tasks/ training. This is easier in case there is visibility of the business objectives and for the work up-ahead required to achieve those business objectives.

Thanks,
Inder P Singh
Blog: http://inderpsingh.blogspot.com/

Shrini Kulkarni said...

@Inder

>>>It seems that your draft list of suggestions is broadly based on three premises: 1) Agile Software Development process,

It seems that you have mis-understood my view point. I have never used the term "agile" or "development" in the post. Even though some of the points that I am proposing (for example: close dev-test interaction)are popular in connection with Agile development - I have nothing do to with that. My views are strictly about doing testing better optimally.

>>> Process improvement

You got me wrong again.. Refer to #2,#7, #9, #10 --- all hint towards "cutting the flab". Show these to anyone who is a software process guy - you will know what I am talking about. Personally I am very allergic to phrase "process improvement" - it sounds like "world peace", "Global Worming" etc.

>>>Re-use.
This is another most abused word in software industry today. It is difficult me to find any item today that is "use and throw". Even toilet tissue paper can be recycled. Through the (repeated) use of the word "reuse" - people are given a false sense of "optimization" and impression that "it is a big deal".

Your other points are well taken and can be implemented after checking their suitability to the context.

Shrini

Anonymous said...

i totally disagree with you i think you are not updated with latest research activities carried out by researchers you are 10 yrs behind mr kulakarni pls try to look out recent research papers even you will also start disagreeing with your suggestions .you sugesstions lead to total failure of the software

Shrini Kulkarni said...

Dear Anonymous --

>>>i totally disagree with you i think you are not updated with latest research activities carried out by researchers

Thanks for disagreeing. What evidence you have to make the statement that I am not updated with research activities. Can you list the research activities that you think I do not (appear to) know?

>> you are 10 yrs behind mr kulakarni

Please substantiate Mr Anonymous. I want to learn from you and researchers that you are referring to?

>>>you sugesstions lead to total failure of the software

That is a bold conjuecture .. how did you come that conclusion.

I encourage you to have a dialog/conversation with me... let me understand your view point...

My email address is on my blog .. drop a mail with details...

Thanks for spending time on my blog and disagreeing with me.

Shrini

Anonymous said...

Hi Shrini,

I totally agree with your solutions ! There's a tool called Kalistick (it's could-based) that is developping these concepts for reducing software testing cost just like you're talking in your article.

Anonymous said...

This same post word for word appears to have been duplicated under a different authors name here: http://www.prowareness.com/blog/?p=367

Shrini Kulkarni said...

@ Anonymous -

If someone duplicates my post and publishes it without giving credit to me - it's OK. People will know the original author.

In fact I feel happy that someone felt that some piece of my writing is good enough to be copied.

that is my take on plagiarism - With so much free stuff over there - real challenge is get eyeballs and ears for your views.

Guys - go and use my work - if it helps you.

Shrini