Tuesday, December 22, 2009

Why GUI Test Automation is Popular and Tempting? Part I - Business context

It is rare to someone on testing in IT world not be aware of "QTP, WinRunner, Rational Robot etc ". For the most in IT world, the word automation is synonymous with GUI automation. Perhaps some even might ask - "Are there any other forms of automation other GUI automation". Lots of how IT world sees automation has to do with how they see testing itself. You might have heard this "celebrated" statement - "Regression testing is more suitable to do automation (read GUI) so that we don't have to have our IT staff do that unnecessary testing manually".

To my surprise, having been associated with IT for many years now, people here still believe in perfect requirement, perfect test case and hence "doing it right first time". In contrast to this, software product world (likes of Microsoft, Google) treats test automation and hence testing in a total different way. These two probably operate from two extreme ends of automation spectrum. While discussion how automation is perceived and realized in IT and software product world can be a separate post in itself, I propose to elaborate here my views on the popularity on GUI Automation in IT.

Testing is necessary evil and should get done quickly

This is the reason #1 that drives GUI automation in IT. Business spends on getting IT application features developed and hence often fails to see a reason why they should spend on "testing" at all. Business feels that IT should get their acts together, get aligned to business (hence IT-business alignment is often a well understood but poorly executed theme) and use the money towards delivering business value. I even have heard some business folks saying "We pay for the application features not for their testing". Hence funding testing becomes IT's problem. That is why they look for solutions to reduce testing (or spend on it). As simple as it sounds, IT pays for testing (not the business)

Testers (read business users or analysts) are hard to find and they hate regression testing

Traditionally, in IT business analysts (aka BA's) are hot commodities. They do most of what we call as "testing". With outsourcing wave sweeping IT, some of BA roles are taken by either BA's from outsourcing vendors or some junior tester working under supervision of BA through highly scripted test procedures. The problem is that these BA resources are few and are in great demand. But as software releases happen, changes come in – testing (regression testing) has to happen and BA's hate regression testing (or all of testing). So, an IT manager needs to find a way out to carry out this regression testing somehow.

How to get testing done fast and cheaper?

If you want to do something fast, give it to machine or a computer program - goes a typical approach of an IT manager. So automation could be the answer. If you position automation as an aid to reduce the cycle time to market chances of you getting funding is very high. Surprisingly this claim is so powerful that when made often goes unchallenged. The magic word "reduction of cycle time" (that too without any qualification) is tempting for many that it will make them to completely overlook bottlenecks in development (programming), bug fixing, investigation and any other non-testing activities. Thus speaking in terms of business language is so important in proliferation and apparent success of GUI automation in IT.

With GUI automation, you can easily make a business case for automation.

To me, the terms like test cases, regression testing, test cycle time etc are the ones that business stakeholders appear to understand very well. As compared to unit tests and other forms of non GUI automation, you can make your case for investment in GUI automation to business stakeholders. Let us say you have 5000 test cases for a regression testing to cover that takes 5 person days to complete, automate 50% of them you can straight way knock off 50% of your testing cycle time reduced. This claim due to its simple math and logic makes a compelling case for GUI automation or simply "automation". What else can be more appealing than investing once in automation (for the sake of making the business case and further forever – down play the maintenance costs) and saving on a percentage proportional amount of automation achieved for 100's of cycles in the future?

Tool vendors help you by talking in business language

Enter tool vendors – with the fancy looking charts, scary quotes like "more than 60% of project cost goes to testing, what are you doing to reduce it" and likes Gartner's and Forrester's to support the claims with numbers, life for an IT manager in charge of testing has never been so easy. Just buy a tool, engage an outsourcing vendor to do GUI automation – all your testing worries are over in one go. Dreaming never ends here. Take a look at any brochures, websites, commercial literatures – they are full of business terms like ROI, Cost of testing, business impact, and Time to market and so on - making a perfect connect with those that matter in making decisions about spending.

Now try doing this (or these) with non GUI tests such as API tests or xUnit framework based unit tests – what will business say? What will IT managers say – "well, developers got to do that testing anyway"? So there is no apparent business case to make so sell automation. Writing unit tests will not speedup your testing cycle, will not reduce time to market, might reduce some portion of that "boring" regression testing – but. So you have no financially oriented incentive to do any automation that is non GUI. Since GUI automation paradigm as so much hardened into dogma – very few challenge it.

If you are the one working in a software product company – you would say "do less of GUI automation, they are very brittle and costly to maintain". You would be surprise to know/hear – the opposite works (or at least appear to work) in IT world.

It is "power of talking in business language, very few challengers for the claims made about automation and excellent support by tool vendors" that makes GUI automation popular and tempting to attempt. Even if you fail – you can blame it on almost anyone or anything around – test cases, tool, automation folks, application changes, changing requirements, test data, operating system patches and so on.

To be continued in Part II


Political Beat said...

Good article. In the past, I felt that problems in GUI automation were caused by poor design of the framework (or no design as is with record/play). As I learned more about that, I recognize that some auto tests are brittle because they rely on uncontrolled data and dependence on uncontrolled background schedulers. Some of those problems are not caused by the GUI.

I feel that you left out testing the business logic, which is of direct interest to the business stakeholders. It's a tougher sell because that assumes there is a non-GUI interface like an API or WebServices. You can find references to that in the Crispin/Gregory Agile Testing book. I'd like to see vendors try to create tools to support that.

Dave McNulla

Anuj Magazine said...

I agree on the part that GUI automation is a brittle and somewhat risk proposition in Product based organizations but in my experience not all product based organizations have the same stand on GUI automation. In my experience, i have seen GUI automation being taken up by Product organizations despite the known Pitfalls. I have seen fewer things that catch the fancy of decision makers other than a well crafted Test automation strategy showing (imaginary) ROI in dollars.
Having said this, i have also witnessed good successes from GUI automation but as you rightly mentioned, it requires a good foresight and knowledge of dependent factors apart from other required skills for Test automation.


Unknown said...

Very good article, spot on. To me, when you compare IT to Product companies, IT sounds more like Consultancies.
Having worked in both I can see exactly where you're coming from.
Needing to create business cases for everything I find very tiring. It's one model to work with which breeds the behaviour your describing. Surely there are other models that rely more on trusting the subject matter experts rather than the people who can speak the business language?

QPT said...


First of all Thanks very much for your useful post.I would like to introduce another good
blog which is having free software testing ebooks,Testing Quiz and Question&Answers, Have a look.

Software Testing Quiz Questions
Free ebooks for Testing and Automation

Unknown said...

Good article Shrini!
I have a feeling that part 2 will be an huge eye-opener, so I cannot wait for part 2.

Kind Regards,

Anonymous said...

I want not concur on it. I regard as precise post. Particularly the title attracted me to read the intact story.

Anonymous said...

Genial post and this mail helped me alot in my college assignement. Thank you as your information.

Anonymous said...

Reading these kind of posts reminds me of just how technology truly is omnipresent in this day and age, and I am fairly confident when I say that we have passed the point of no return in our relationship with technology.

I don't mean this in a bad way, of course! Societal concerns aside... I just hope that as technology further innovates, the possibility of downloading our memories onto a digital medium becomes a true reality. It's a fantasy that I dream about almost every day.

(Posted on Nintendo DS running [url=http://kwstar88.insanejournal.com/397.html]R4i SDHC[/url] DS ComP)

Fake Software Tester said...

It's very true. Sometimes, I feel that we need to pick the right automation tool, not rely on record and playback, but be able to do effective scripting for results. 1 major downside of the GUI testing is that they would check for accuracy of an algorithm, while testing the algorithm itself would be missed.

S Pande said...

Pardon me if I'm naive but I think you've totally missed the point of delivering a good quality product at the end of the produtct life cycle. It's not about replacing one form of automated testing with another. Unit tests are imperative but so are GUI automation tests as well. In the day and age of Agile processes, where companies need to deliver multiple flavours of a product in sprints of about 2 weeks, it's humanly no longer possible to conduct regression testing for a product, let alone multiple versions. I do agree that with Agile, the challenge of more flexible requirements comes in too. However, that's totally dependent on how the tests are designed. Gui automation tests run over night are not just saving tester time but also identify any potential issues the night check-ins are made. GUI automation tools are more popular than ever before and I think there's a good reason for it :)