I received a mail from a friend who asked about estimation approach for test automation. Wow - what a topic to mess up your head on a sunday night. Instead of responding to him in a mail - I thought of writing a post on this so that other can engage some conversation with me on this topic.
Here you do - 10 random thoughts on the topic. Well - I can extend this list to more than 10 items. For now there are 10 items.
6. Test automation is writing code - that involves all that writing code needs. Ask developers what they need. If development world claims that they cracked this problem. Automation folks can simply lift and use that solution.
2. Regardless of what commercial tools claim about "generating code" or similar non-sense around "scriptless" solution - fact remains that any sustainable automation code is similar to software product that automation aims to validate - so do not fall prey to false propaganda around tools that claim easy automation.
4. Ideas/Frameworks like data -driven, keyword, hybrid are simple ideas for automation design. You need to go deeper as ask if I need to write a method/function or a class in automation code - how much time will I need? Do get an answer from say a developer? You might be aware of some crazy metrics around number of lines of code written per day or number of functions/methods/classes written per day. As you can - it only gets murkier if start insisting on measuring productivity of automation guy in terms of these meaningless metrics.
9. Important thing to note is we (folks in software world) are knowledge workers - meaning we do not work in manufacturing assembly line of factory. We deal with abstract things, software that can not be worked in the same way as say "Cars". So how does that change the way we should be viewing estimation of developing automation code - think about it.
5. Depending upon nature of piece of automation work that you would be developing - you would not know in the beginning how much time you spend in thinking and how much you spend putting your thoughts into compilable code. That is biggest challenge developers have. So are we - automation developers.
7. First question you should ask when you are working developing an estimation model for automation - what is my smallest unit of work - that unit which is building block of my whole solution. What answer would you get? Function? class? Next question would be - are all building blocks are similar? How many different types of building blocks do i have?
Compare it say atoms - ask how can I characterize atoms or molecules of automation solution.
2. Once you get clear answer on above question - next you need break down your automation solution into building blocks and size them. Then ask - given a competent developer of automation - how much each unit would take to build - hours, days etc. Then add up time setup, testing, integrating etc. You will get some ball park number of estimate that you need go with as first estimate.
1. What reference should you use for creating your automation solution (or design?) - anything that describes what you want exercise on the application under test. One approach I found useful is create a mind map of application features and attach to each feature what application can do, what data it processes and what checks (note the word check) needs to be done. This is your skeleton reference. Build this first by collating data/information from various references - then make sure all information from each sources is accounted. This is your master reference. Work with multiple sources of reference (Requirements, test cases, use cases or simply walkthrough of application manually to build map of features).
10. Should you use test cases or test scenarios or test steps as basis for automation estimation? As James Bach prefers to call - test cases are like unicorns - how many of them you can fit in a suitcase or fridge? Without knowing what is inside - counting test steps or test cases and using that number as basis for anything (leave alone automation) useful is utterly stupid idea. Never do it - unless you want to mislead someone.
3. Few words about keyword driven framework. Personally I think, there is lots of hype around this simple idea. Keyword could be a verb (also called as action) that describes some feature of application under test. In developers language it is some basic unit of code - typically a method or function. What is a big deal here when you say "let us use keyword driven framework"? It's all hype - no real stuff there. There even more irritating words like "keyword driven or based testing" - so far I have not figured out how to do testing (not automation) using keywords. Same goes for other related buzz words like data driven automation (a marketing term for saying let us use variables instead of hardcoded values) or hybrid framework. Note all these simple ideas had some place 10-20 years back but not anymore. I personally prefer to develop automation pretty much like a developer goes about doing product code - no difference. I hate over simplification by tool vendors and consultants on so called "excel based automation" or script-less automation, automation for your granny - they are simply empty ideas to bully unsuspecting boardroom folks that sign contracts.
How will I summarize - There is no simple solution for estimating automation effort. Keep a watch on how development (programming) community deals with this conundrum and let us use that to build our own model. At present, when developers are working on small units work like use stories and in an iterative model of churning working code (theoretically) in say weekly or fortnightly or monthly basis -
I think the whole problem of "tell me how much time (and resources) you need to develop this solution" - will vanish. You would probably say "let me start with 3 people, I will publish a 1-2 week plan of what I deliver for people to use - let us take it from there".
I think gone are the days where you had 3-6 (or even more) months of lead time for something software to be deployed for use. In the mobile apps world - development times are even shorter. I doubt if anyone would ask you - give me an estimate for automation of this app. It seems that we have solved the problem of estimation by going small and going fast.
I am happy to be corrected on any of views expressed here. Let me not forget to add - when I say automation - my experience has been in IT/IT services world mainly working with commercial off the shelf automation tools. If it were likes of Google or Microsoft - it would be totally a different ball game all together.
Shrini
Here you do - 10 random thoughts on the topic. Well - I can extend this list to more than 10 items. For now there are 10 items.
6. Test automation is writing code - that involves all that writing code needs. Ask developers what they need. If development world claims that they cracked this problem. Automation folks can simply lift and use that solution.
2. Regardless of what commercial tools claim about "generating code" or similar non-sense around "scriptless" solution - fact remains that any sustainable automation code is similar to software product that automation aims to validate - so do not fall prey to false propaganda around tools that claim easy automation.
4. Ideas/Frameworks like data -driven, keyword, hybrid are simple ideas for automation design. You need to go deeper as ask if I need to write a method/function or a class in automation code - how much time will I need? Do get an answer from say a developer? You might be aware of some crazy metrics around number of lines of code written per day or number of functions/methods/classes written per day. As you can - it only gets murkier if start insisting on measuring productivity of automation guy in terms of these meaningless metrics.
9. Important thing to note is we (folks in software world) are knowledge workers - meaning we do not work in manufacturing assembly line of factory. We deal with abstract things, software that can not be worked in the same way as say "Cars". So how does that change the way we should be viewing estimation of developing automation code - think about it.
5. Depending upon nature of piece of automation work that you would be developing - you would not know in the beginning how much time you spend in thinking and how much you spend putting your thoughts into compilable code. That is biggest challenge developers have. So are we - automation developers.
7. First question you should ask when you are working developing an estimation model for automation - what is my smallest unit of work - that unit which is building block of my whole solution. What answer would you get? Function? class? Next question would be - are all building blocks are similar? How many different types of building blocks do i have?
Compare it say atoms - ask how can I characterize atoms or molecules of automation solution.
2. Once you get clear answer on above question - next you need break down your automation solution into building blocks and size them. Then ask - given a competent developer of automation - how much each unit would take to build - hours, days etc. Then add up time setup, testing, integrating etc. You will get some ball park number of estimate that you need go with as first estimate.
1. What reference should you use for creating your automation solution (or design?) - anything that describes what you want exercise on the application under test. One approach I found useful is create a mind map of application features and attach to each feature what application can do, what data it processes and what checks (note the word check) needs to be done. This is your skeleton reference. Build this first by collating data/information from various references - then make sure all information from each sources is accounted. This is your master reference. Work with multiple sources of reference (Requirements, test cases, use cases or simply walkthrough of application manually to build map of features).
10. Should you use test cases or test scenarios or test steps as basis for automation estimation? As James Bach prefers to call - test cases are like unicorns - how many of them you can fit in a suitcase or fridge? Without knowing what is inside - counting test steps or test cases and using that number as basis for anything (leave alone automation) useful is utterly stupid idea. Never do it - unless you want to mislead someone.
3. Few words about keyword driven framework. Personally I think, there is lots of hype around this simple idea. Keyword could be a verb (also called as action) that describes some feature of application under test. In developers language it is some basic unit of code - typically a method or function. What is a big deal here when you say "let us use keyword driven framework"? It's all hype - no real stuff there. There even more irritating words like "keyword driven or based testing" - so far I have not figured out how to do testing (not automation) using keywords. Same goes for other related buzz words like data driven automation (a marketing term for saying let us use variables instead of hardcoded values) or hybrid framework. Note all these simple ideas had some place 10-20 years back but not anymore. I personally prefer to develop automation pretty much like a developer goes about doing product code - no difference. I hate over simplification by tool vendors and consultants on so called "excel based automation" or script-less automation, automation for your granny - they are simply empty ideas to bully unsuspecting boardroom folks that sign contracts.
How will I summarize - There is no simple solution for estimating automation effort. Keep a watch on how development (programming) community deals with this conundrum and let us use that to build our own model. At present, when developers are working on small units work like use stories and in an iterative model of churning working code (theoretically) in say weekly or fortnightly or monthly basis -
I think the whole problem of "tell me how much time (and resources) you need to develop this solution" - will vanish. You would probably say "let me start with 3 people, I will publish a 1-2 week plan of what I deliver for people to use - let us take it from there".
I think gone are the days where you had 3-6 (or even more) months of lead time for something software to be deployed for use. In the mobile apps world - development times are even shorter. I doubt if anyone would ask you - give me an estimate for automation of this app. It seems that we have solved the problem of estimation by going small and going fast.
I am happy to be corrected on any of views expressed here. Let me not forget to add - when I say automation - my experience has been in IT/IT services world mainly working with commercial off the shelf automation tools. If it were likes of Google or Microsoft - it would be totally a different ball game all together.
Shrini
10 comments:
Shrini,
Nice post! Wow, I have to say for once you and I are in total agreement on the subject (98%, close enough for me).
You nailed a lot of the misconceptions & misperceptions of what is involved in this line of work in testing. I know, been dealing with them for over 20 years.
As my favorite saying goes "It's Automation, Not Automagic!" And you definitely show it isn't magic we need to get this one done.
Hi Shrini,
I had my let us take it form there phase and now I am asked to prepare a estimate for the automation of a project. Would you say it's not right if I created a plan based on all available test cases, considering that I have gone through each of them during manual tests and have a good idea about the application.
Hi Shrini,
I had my let us take it form there phase and now I am asked to prepare a estimate for the automation of a project. Would you say it's not right if I created a plan based on all available test cases, considering that I have gone through each of them during manual tests and have a good idea about the application.
Ram,
As I said in my post - test cases are one of many reference points that you can consider to arrive an estimate. I hope you are wise enough not to succumb to count test cases or classify them into some simple medium complex categories and assign an arbitrary factor to estimate. This is common practice to ask how many simple test cases are there, how many complex test cases - the question in itself is not bad. But the answers it leads to are very dangerous. This approach for sampling test cases leads to very strange insight that is utter useless for for automation
When you are using test cases for automation - you would ask "is this test case complex"? Now - think about this. What is complexity for human and what is complexity for machine or a program.
Using test cases as basis for automation - takes you to such error paths that you need to be aware of.
Shrini
Nice list, and I laughed at the randomness...
Another factor that I've used in estimating: the balance between short term and long term. We often have a choice between starting quickly and investing in up-front design.
Starting quickly shows some payoff quickly, but in the long run usually costs more in maintenance and refactoring. Up-front design (framework, library of common functions, perhaps training), could have the benefit of long term productivity (create more tests & spend less time in maintenance).
Depending on the circumstances of the project, and the level of commitment, either the short term or long term approach may be more appropriate.
Totally agree about Keyword Driven, Data driven , Hybrid frameworks, this is just marketing jargon.
Typically automation starts on the regression tests, so manual test team has fair idea about complexity of test cases for manual execution. Automation expert surely need to define the SMC from automation perspective, also create WBS for common functions, libraries, external components like custom reports, suite driver etc. This is what worked for me in Web automation projects using tools like Winrunner, QTP and Selenium.
The mobile app projects, as you rightly said, are short term where you don’t have separate phase for automation. For mobile app test automation project that I worked recently, the challenge was to automate tests parallel to development, so we had additional efforts estimated to mitigate run-time app changes in resource ids etc.
Totally agree about Keyword Driven, Data driven , Hybrid frameworks, this is just marketing jargon.
Typically automation starts on the regression tests, so manual test team has fair idea about complexity of test cases for manual execution. Automation expert surely need to define the SMC from automation perspective, also create WBS for common functions, libraries, external components like custom reports, suite driver etc. This is what worked for me in Web automation projects using tools like Winrunner, QTP and Selenium.
The mobile app projects, as you rightly said, are short term where you don’t have separate phase for automation. For mobile app test automation project that I worked recently, the challenge was to automate tests parallel to development, so we had additional efforts estimated to mitigate run-time app changes in resource ids etc.
Hii Shirin,
Can You Please Give ME Your Email Id.
As I Want to know more About Automation Scripting.
Plese Send Ur Email id To me.
mohit.suri34@yahoo.co.in
Hi Shrini,
Thanks for the post, I really liked the bit where you mentioned that basing your estimate on the number of test cases is a stupid idea. I completely agree with you as I've through similar situation and I found it ridiculous.
thanks,
Thanks for sharing such an informative post on test automation. I also wrote the same lines on my blog - http://bit.ly/1nhPKmc
Post a Comment