When automation is required, either by contract or due to technical constraints, ROI computation may not be helpful. Intangible factors may constitute the bulk of the return, and thus arithmetic computations won't indicate the real value of automation. Fortunately in these situations we often aren't faced with questions about the value of automation because it must be employed regardless.
Here goes a conversation with a colleague of mine who wanted me to help him with some ROI calculation for an automation project.
Colleague: Do you have a formula or framework for calculating ROI from automation?
Me: I might … first let me understand what you are looking for.
Colleague: It is simple man … Here is a client who is looking for investing in automation and she is interested in knowing the ROI so that she can take it to her boss with business case.
Me: That is good. What are the elements of ROI you are interested in knowing now?
Colleague: What do you mean?
Me: To me, ROI has three elements – a notion of investment (effort, money, time etc – all these can be interdependent in some way), a notion of "return" (called as benefits – some tangible, meaning quantified in terms of numbers, some intangible – soft benefits – qualitative measures) and finally a timeline usually in terms of direct measures like calendar months, or work/effort months OR indirect measures like number releases, number of test cycles, number of platforms covered etc. Which one is of interest to you…?
Colleague: All three … of course!!!
Me: Then you have some hard work to do gather information, data, expectations, some historical and some current.
Colleague: Well... I thought it is easy to find out ROI … I was told that there are many freely available ROI calculators especially catering to automation … are you aware of them?
Me: Yes, I have seen few of them … not so impressed … One problem that I have with most (or all) of these calculators is that a) they use a highly simplified model of testing that is totally out of context. Meaning you can just apply that to any project, any technology, any tool … you will have some numbers coming out … That is too good to believe 2) They equate automation to human testing literally 1:1 … In my opinion, automation is a different kind of testing – remove the human being (to the extent possible) and introduce the machine (automation script) – then think (dream, pray and wish) that program does EXACTLY like a human.
Colleague: This is too confusing …. Let me try to explain my problem in a different way. Customer is investing x dollars for automation, she wants to know when will she be able to recover the investment and when she will start reaping benefits (possibly without investing anything incrementally). How can we help her?
Me : OK … that is fair … Here we come again to same structure – x dollars (investment), when will she recover the investment (time lines) and when/what benefits she can expect, without possibly not incrementally investing (returns). Let us attack one by one … How your client wants to recover the investment?
Colleague: that is silly question … she wants to save manual testing effort by automating all or whatever is technically feasible. So, cycle time reduction is what she is looking at.
Me: So, the questions are – How much will be the cycle time reduction (assuming that that is possible and worth pursuing), by when, that reduction will be realized? What are the incremental benefits till that point of time? Right? Anything else I am missing?
Colleague: Good … I think now you have understood my problem … what next?
Me: Are all cycles of the same size? What happens to application under test for all these cycles (meaning does it under go change or not?) What is the current test cycle time (manual)? What all happens in a cycle? What are things under tester's control and what are not? When do you repeat cycle (assuming that you repeat cycles)?
Colleague: Oh!! My God … my head is spinning … I will have to get all the information and data... Are you sure these are required for ROI calculation? Anything else?
Me: Yes, at least in my opinion, to give a reasonable picture of ROI where R = cycle time reduction – I would need these. There are some more things that I would require to complete the equation … but let us get started with this ….
BTW, what makes your client believe that machine replicate what humans do? There are things machines are good at and there are things humans are good at. No matter what you do … I think in the context of test automation – machine cannot do what a sapient human tester can do (unless, human by design behave as though they are brain dead and emotionless).
Colleague: No …. No … not again … Do not try to confuse me … I will get you the details that you asked for… then let us fit an ROI formula. Please put you're the tester in you to sleep till then …
Me: (smiled) OK … Please bear in mind that "you cannot compare even one cycle of automated execution to same cycle of manual execution".
While my colleague is out to get the data that I asked for … what do you think? What have been your experiences of calculating ROI with automation … how did you deal with "improbable" yet simplistic model of treating automation execution equivalent to what human tester does – hence talking about cycle time reduction etc? What other returns (benefits) that automation provides, have been successful with your clients? How did you quantify them?
I work in IT services industry. Day in and out, I hear people asking me such things - while I attempt to explain them the hazards of the simplistic model of testing and automation they use in ROI, need for business case to push automation (that requires numbers and quantified measures) makes me to look for innovative ways to articulate what I want to say but in way that "business" people can agree on …
To be continued ….
Here are 3 useful and well written papers on "Automation ROI"
- ROI of Test Automation by Mike Kelly (2004)
- Cost Benefit Analysis by Doug Hoffman (1999)
- Bang for the Buck Test Automation by Elisabeth Hendrickson (2001)
"… every time I hear "let's take a look at the ROI," or "it will increase your ROI," or "all we need to do is use the ROI calculator" some little part of me shrivels up and dies. It drives me insane. I refuse to believe that for products as complex and involved as automation and performance testing services (where you need to understand infrastructure, application architecture, business and use cases, deployment models, culture, risk tolerance, and the other aspects of the design, development, and testing taking place for a project) that you can so easily capture the ROI. If it were that easy you wouldn't be talking to me about it."
- Mike Kelly.