When automation is required, either by contract or due to technical constraints, ROI computation may not be helpful. Intangible factors may constitute the bulk of the return, and thus arithmetic computations won't indicate the real value of automation. Fortunately in these situations we often aren't faced with questions about the value of automation because it must be employed regardless.
-Doug Hoffman
Here goes a conversation with a colleague of mine who wanted me to help him with some ROI calculation for an automation project.
Colleague: Do you have a formula or framework for calculating ROI from automation?
Me: I might … first let me understand what you are looking for.
Colleague: It is simple man … Here is a client who is looking for investing in automation and she is interested in knowing the ROI so that she can take it to her boss with business case.
Me: That is good. What are the elements of ROI you are interested in knowing now?
Colleague: What do you mean?
Me: To me, ROI has three elements – a notion of investment (effort, money, time etc – all these can be interdependent in some way), a notion of "return" (called as benefits – some tangible, meaning quantified in terms of numbers, some intangible – soft benefits – qualitative measures) and finally a timeline usually in terms of direct measures like calendar months, or work/effort months OR indirect measures like number releases, number of test cycles, number of platforms covered etc. Which one is of interest to you…?
Colleague: All three … of course!!!
Me: Then you have some hard work to do gather information, data, expectations, some historical and some current.
Colleague: Well... I thought it is easy to find out ROI … I was told that there are many freely available ROI calculators especially catering to automation … are you aware of them?
Me: Yes, I have seen few of them … not so impressed … One problem that I have with most (or all) of these calculators is that a) they use a highly simplified model of testing that is totally out of context. Meaning you can just apply that to any project, any technology, any tool … you will have some numbers coming out … That is too good to believe 2) They equate automation to human testing literally 1:1 … In my opinion, automation is a different kind of testing – remove the human being (to the extent possible) and introduce the machine (automation script) – then think (dream, pray and wish) that program does EXACTLY like a human.
Colleague: This is too confusing …. Let me try to explain my problem in a different way. Customer is investing x dollars for automation, she wants to know when will she be able to recover the investment and when she will start reaping benefits (possibly without investing anything incrementally). How can we help her?
Me : OK … that is fair … Here we come again to same structure – x dollars (investment), when will she recover the investment (time lines) and when/what benefits she can expect, without possibly not incrementally investing (returns). Let us attack one by one … How your client wants to recover the investment?
Colleague: that is silly question … she wants to save manual testing effort by automating all or whatever is technically feasible. So, cycle time reduction is what she is looking at.
Me: So, the questions are – How much will be the cycle time reduction (assuming that that is possible and worth pursuing), by when, that reduction will be realized? What are the incremental benefits till that point of time? Right? Anything else I am missing?
Colleague: Good … I think now you have understood my problem … what next?
Me: Are all cycles of the same size? What happens to application under test for all these cycles (meaning does it under go change or not?) What is the current test cycle time (manual)? What all happens in a cycle? What are things under tester's control and what are not? When do you repeat cycle (assuming that you repeat cycles)?
Colleague: Oh!! My God … my head is spinning … I will have to get all the information and data... Are you sure these are required for ROI calculation? Anything else?
Me: Yes, at least in my opinion, to give a reasonable picture of ROI where R = cycle time reduction – I would need these. There are some more things that I would require to complete the equation … but let us get started with this ….
BTW, what makes your client believe that machine replicate what humans do? There are things machines are good at and there are things humans are good at. No matter what you do … I think in the context of test automation – machine cannot do what a sapient human tester can do (unless, human by design behave as though they are brain dead and emotionless).
Colleague: No …. No … not again … Do not try to confuse me … I will get you the details that you asked for… then let us fit an ROI formula. Please put you're the tester in you to sleep till then …
Me: (smiled) OK … Please bear in mind that "you cannot compare even one cycle of automated execution to same cycle of manual execution".
While my colleague is out to get the data that I asked for … what do you think? What have been your experiences of calculating ROI with automation … how did you deal with "improbable" yet simplistic model of treating automation execution equivalent to what human tester does – hence talking about cycle time reduction etc? What other returns (benefits) that automation provides, have been successful with your clients? How did you quantify them?
I work in IT services industry. Day in and out, I hear people asking me such things - while I attempt to explain them the hazards of the simplistic model of testing and automation they use in ROI, need for business case to push automation (that requires numbers and quantified measures) makes me to look for innovative ways to articulate what I want to say but in way that "business" people can agree on …
To be continued ….
Shrini
Extraas:
Here are 3 useful and well written papers on "Automation ROI"
- ROI of Test Automation by Mike Kelly (2004)
- Cost Benefit Analysis by Doug Hoffman (1999)
- Bang for the Buck Test Automation by Elisabeth Hendrickson (2001)
Closing thought:
"… every time I hear "let's take a look at the ROI," or "it will increase your ROI," or "all we need to do is use the ROI calculator" some little part of me shrivels up and dies. It drives me insane. I refuse to believe that for products as complex and involved as automation and performance testing services (where you need to understand infrastructure, application architecture, business and use cases, deployment models, culture, risk tolerance, and the other aspects of the design, development, and testing taking place for a project) that you can so easily capture the ROI. If it were that easy you wouldn't be talking to me about it."
- Mike Kelly.
4 comments:
Hi Shrini,
A nice article beautifully written definitely was useful for test managers like me.
I always want to learn and know more on project management and how we need to apply it for testing.
Thanks for giving insights on "ROI". Not many people would be knowing this term (who are aspiring to take sr. levels in testing)
Hope you write more such articles on test management & what test manager’s should know which in turn will help test managers think more and learn more.
Rgds,
Raj
"Are you sure these are required for ROI calculation?"
[Arrrrgh!]
I can almost hear the gears turning in that person's head... Isn't it just a formula like figuring out miles per gallon for operating a car?
Well, no, not at all.
[I once actually had to write a test harness for an ROI estimator in a particular domain. Thank goodness they didn't want me to provide an ROI figure for my effort on the project!]
Shrini, let me just leave a little fieldstone here to say that when someone involved in software doesn't understand that "Garbage In, Garbage Out" applies to all models, everywhere, and that ROI calculations are based on models just as software programs and spreadsheet worksheets are, you get bad thinking. You know that, I know that.
But a lot of people want to point to an oracle. The most sophisticated they want to get is probably thinking/saying "The ROI calculations were thus and such." "We did due diligence; here are the reports on ROI."
But that gets flattened even further into "The ROI will be x in y time units."
Note the sleight-of-mind there? Not "The model predicts that the ROI will be...", but rather "The ROI will be..."
And if they need a scapegoat, someone will be sure to say "So-and-so said [predicted ! promised] the ROI would be..."
I wish I knew how to mitigate it.
Thanks for this post.
An amusing account and constructive for managers perhaps. The nub of the issue you're addressing here is the expectation of a simple ROI business case to automate versus and complex reality.
What you have missed from my perspective is to explore however subtly or not, the question of what is automated, when, how and why. Not only is the issue of simple ROI expectations versus complex realities salient, but so too the spectrum of things it is possible to automate.
I worry sometimes that no-brainer obvious test automation needs are not invested in because of a combination of poor skills and lack of investment (in time mainly).
No-brainer test automation to mind includes performing a manual human based test which leaves behind it the tools needed to repeat this test quickly and efficiently. We need, beyond any doubt powerful regression testing in our business for example. That is we need to reprodcue test regulalry as builds emerge.
The challenge I find in implementing this automation is not convincing management to invest (I am the management that makes that decision) but to inspire the testers to do it, most especially those who are under constant time pressure because of the sheer volume of manual testing that exists before such a culture is in place and/or because they were hired for the wrong reasons to begin with or for reasons of domain expertise, but lack programming skills needed to automate their manual tests efficiently.
The cost is moderate and the gains immediate and evident. It is moderate because the tester doesn't so much play with our software, pressing buttons clicking menus etc, to validate things, but plays with their own software which does these things. It is just an intermediary. This slows down the original test somewhat, considerably even, but subsequently it can be repeated quickly and easily after every build and most saliently with much greater reliability (things aren't missed thanks to boring familiarity with the process).
The testers in turn become more intimately skilled in software development themselves which empowers the conversations they have with software programmers.
Much of which speaks to qualitative ROI elements your refer to. The significant positive cultural gains achieved by instilling a culture or easy reproduction in the understanding that tests are not one off activities but things that must be repeated time and time again due to the changing nature of the software.
All the more true of versioned box software as we develop than it is for one off custom developments, but true for both all the same.
Thanks for an interesting conversation. I hope that the qualitative benefits of generating reproducible tests can be sold on management with less polemic than such an ROI discussion. As an adamant statement of need by the project managers, the cost/benefit of which is a indeed a little too abstract to capture in a single number (or two).
Thanks Bernd ... I appreciate your view points and the time that you have taken to write them here ...
Stay tuned to this blog ...
Hope that you will find other posts intersting
Post a Comment