Test estimation itself is mystery or magic tool that every test and project manager is trying to master these days with very little success - I should say. Then Test estimation for Automation gets little more complicated as it involves other software - automation tool. It is like a estimating for full-fledged application development and testing project. Treat it like nothing less. Ask your manager or his/her colleague PM how they do estimation for complete project (dev and test) - Take some leaves out of their experience.
Some thoughts to get you started off :
1. Like software development project - automation also has its own similar lifecycle
a. Automation Planning
b. Test cases Analysis (= Requirements phase)
c. Automation Design (= Design Phase)
d. Coding and Unit Testing of Scripts
e. Automation builds
f. Source control and Testing, Defect management
g. Deployment on Test lab - try run - fix and re-run
h. Sign off
So be sure to factor in time for all these - just focusing on test case and their complexity is a surely leads to underestimation
2. Following are common and assumed to be factored by the PM - make sure you check whether your team is up there.
a. Required tools licenses
b. Team having training usage of QTP and some development experience
c. References like coding guidelines, Config Mgmt guidelines and other dos and donts kind of documents
d. Setting up of test environment - this is very imp. Mostly we assume that it is there and when you are starting the project you will spend more than you estimated time in getting Test environment up and running for whole group. Take this point little more serious if you are running an “offshore-onsite” type of automation.
e. Framework - supporting structure other than the tool - decide whether you need it or not
f. Other tools and software requirements like - VSS, Database, shared drive to keep common stuff etc
3. Test cases - you need to look at test case complexity from a different angle when you are automating. It does not help in classifying test cases as simple, complex etc. You should look it from over all development of common code perspective. Take a set of logically related test cases and think how many re-usable functions you will require - each for navigation, data input and verification. More the number of functions that a test case needs – more complex it will be for automate. Hence take more time. While estimating always consider a group of test cases not individuals. In a test case items that need to considered are - number of steps, number of inputs and number and type of verification points.
When you are asked for estimation for automation - ask for time to analyze the target test cases and then make judgment call. If you are asked to estimate in a quick and dirty way - shoot back asking - how much of error in estimation they are willing to take - 20-30%? Tell them you will refine your estimates once you have a complete look at test cases. As it happens in development you will revise your estimate after requirements (if it is allowed) - Do it after test case thorough analysis....
Rest in part 2 ...
Shrini
Wednesday, February 22, 2006
Monday, February 20, 2006
Pairwise Testing ...
This is an interesting topic in testing related to testing of a feature that is influenced bt multiple variables. Here is one blog post from Apoorva Joshi - which is like single reference that points out to several others notable one - each two Famous Michael's in testing community - Michael Hunter of microsoft and Michael Bolton.
http://criticsden.blogspot.com/2005/02/pairwise-testing.html
Right now, I am too anxious to make this post so that I can comeback later with my comments ...
Here are my quick questions about Pairwise testing
1. Why pair is imporatant? What about triplets or 4 variables at a time?
2. How is concept of "Orthogonal" or "Taguchi Arrays" related to Pairwise testing
I will study and come back on this ... Meanwhile enjoy reading above thread ...
Shrini
http://criticsden.blogspot.com/2005/02/pairwise-testing.html
Right now, I am too anxious to make this post so that I can comeback later with my comments ...
Here are my quick questions about Pairwise testing
1. Why pair is imporatant? What about triplets or 4 variables at a time?
2. How is concept of "Orthogonal" or "Taguchi Arrays" related to Pairwise testing
I will study and come back on this ... Meanwhile enjoy reading above thread ...
Shrini
Server Virtualization
In computing, virtualization is the process of presenting a logical grouping or subset of computing resources so that they can be accessed in ways that give benefits over the original configuration. This new virtual view of the resources is not restricted by the implementation, geographic location or the physical configuration of underlying resources. Commonly virtualized resources include computing power and data storage.
A good example of virtualization is modern symmetric multiprocessing computer architectures that contain more than one CPU. Operating systems are usually configured in such a way that the multiple CPUs can be presented as a single processing unit. Thus software applications can be written for a single logical (virtual) processing unit, which is much simpler than having to work with a large number of different processor configurations.
Virtualization is about running an Operating System (the guest OS) on the top of another OS (the host OS). This technique enables running several virtual machines with different OSes at the same time on the same hardware. VMWare, MacOnLinux, and Xen are examples of virtualizer software. Virtualization requires guest OSes to be built for the host machine processor. It should not be confused with emulation, that do not have this requirement: When an OS runs on the top of a virtualizer, its code runs unchanged on the processor, whereas an emulator has to interpret to the guest OS code. MAME or Basilisk are examples of emulators. Binary compatibility is another different feature: it is the ability of an OS to run applications from another OS. For instance BSD systems are able to run Linux binaries.
Avenues for Virtualization software
1. MS Virtual Server 2005 - http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx
2. VMWare – www.vmware.com
3. XenSource - http://www.xensource.com/ (from Xen Open Source community)
References:
http://www.virtualization.info/
A great article with an introduction to Virtualization [kernelthread.com]
Microsoft Virtual server Road Map :
http://www.entmag.com/reports/article.asp?EditorialsID=87
Advantages of Virtualization
1. Increase utilization of existing Server hardware
2. Easy maintenance
3. Help in Business continuity and Disaster Recovery initiatives
Shrini
A good example of virtualization is modern symmetric multiprocessing computer architectures that contain more than one CPU. Operating systems are usually configured in such a way that the multiple CPUs can be presented as a single processing unit. Thus software applications can be written for a single logical (virtual) processing unit, which is much simpler than having to work with a large number of different processor configurations.
Virtualization is about running an Operating System (the guest OS) on the top of another OS (the host OS). This technique enables running several virtual machines with different OSes at the same time on the same hardware. VMWare, MacOnLinux, and Xen are examples of virtualizer software. Virtualization requires guest OSes to be built for the host machine processor. It should not be confused with emulation, that do not have this requirement: When an OS runs on the top of a virtualizer, its code runs unchanged on the processor, whereas an emulator has to interpret to the guest OS code. MAME or Basilisk are examples of emulators. Binary compatibility is another different feature: it is the ability of an OS to run applications from another OS. For instance BSD systems are able to run Linux binaries.
Avenues for Virtualization software
1. MS Virtual Server 2005 - http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx
2. VMWare – www.vmware.com
3. XenSource - http://www.xensource.com/ (from Xen Open Source community)
References:
http://www.virtualization.info/
A great article with an introduction to Virtualization [kernelthread.com]
Microsoft Virtual server Road Map :
http://www.entmag.com/reports/article.asp?EditorialsID=87
Advantages of Virtualization
1. Increase utilization of existing Server hardware
2. Easy maintenance
3. Help in Business continuity and Disaster Recovery initiatives
Shrini
Friday, February 17, 2006
Automation of Setup programs ..
Let me make a blanket statement - My first impression and view is that "Setup programs" in general are not suitable for automation. They are best tested manually - unless you work for company like InstallShield whose products themselves help in creating setup programs. Those seup programs having 5-6 steps and taking just folder name as input - are not going provide return on investment for automation.
You decide to play devil's advocate and say -"I don’t agree with you" and insist on automation - Here is one approach.
1. Identify how big is setup program - how many steps are there? 10? 20? or more than 20? what are the variations possible? 100+? If yes - automation will help you.
2. Now identify the most dense and cluttered screen/step in setup that takes large number of inputs. Automate that screen only and proceed and automate let us say top 5 critical screens with 5 scripts.
3. In my opinion - there is no need to automate the setup flow unless there are more than 20 steps and 100+ variations possible.
Look at the beauty and usefulness of such analysis - you are trying to do automation, in the process, you ask so many questions. Think and create those scenarios which otherwise would never have been explored if you are following a structured and scripted test plan. See the value here. At the end you may or may not automate all those scenarios but while trying to automate and tying to convince that automation is way to go - you have tested and enriched test scenarios.
To quote Michael Bolton (www.developsense.com), a friend and mentor - "Often automation, in itself may not lead to good testing or value directly but during the course of preparation whatever the analysis you do and questions you ask "How can I verify this"?”Why should I automate this?" "What can go wrong here" - are VALUABLE and should be done.
How? By always playing a devils advocate - "why" and "why not" If you stop questioning and just accept what is being told to you - you stop learning and cease to become a Tester...
Shrini
You decide to play devil's advocate and say -"I don’t agree with you" and insist on automation - Here is one approach.
1. Identify how big is setup program - how many steps are there? 10? 20? or more than 20? what are the variations possible? 100+? If yes - automation will help you.
2. Now identify the most dense and cluttered screen/step in setup that takes large number of inputs. Automate that screen only and proceed and automate let us say top 5 critical screens with 5 scripts.
3. In my opinion - there is no need to automate the setup flow unless there are more than 20 steps and 100+ variations possible.
Look at the beauty and usefulness of such analysis - you are trying to do automation, in the process, you ask so many questions. Think and create those scenarios which otherwise would never have been explored if you are following a structured and scripted test plan. See the value here. At the end you may or may not automate all those scenarios but while trying to automate and tying to convince that automation is way to go - you have tested and enriched test scenarios.
To quote Michael Bolton (www.developsense.com), a friend and mentor - "Often automation, in itself may not lead to good testing or value directly but during the course of preparation whatever the analysis you do and questions you ask "How can I verify this"?”Why should I automate this?" "What can go wrong here" - are VALUABLE and should be done.
How? By always playing a devils advocate - "why" and "why not" If you stop questioning and just accept what is being told to you - you stop learning and cease to become a Tester...
Shrini
Sunday, February 12, 2006
Tips for Developer Testing ....
Do stop by at this blog post to get Braidy tester Michael Hunter (Microsoft) tips for Developer Testing ...
http://blogs.msdn.com/micahel/archive/2006/01/25/TestingForDevelopers.aspx
Shrini
http://blogs.msdn.com/micahel/archive/2006/01/25/TestingForDevelopers.aspx
Shrini
In quest of automation tools ...
As automation is catching up like wild fire in software testing field – people are frantically searching for cost efficient ways to do automation. Some cash rich companies are investing in industry standard and proven automation solutions from leading tools vendors like Mercury, Rational, Compuware and Segue – other “not_so_rich” companies are struggling around “open source” free tools. Some of the product companies like Microsoft, CISCO – invest in developing their own in house tools. So, broadly we have three categories of automation tools in testing – Commercial automation tools, Open source tools and in-house tools. The first two are available to testing community at large. The information about commercial tools is rather well known and is available at respective websites
Mercury - http://www.mercury.com
Compuware - http://www.compuware.com
Rational - http://www-306.ibm.com/software/awdtools/tester/functional/index.html
(It is quite surprising to see that the information about "once highly popular automation tool" Rational Robot - has gone so deeper into IBM site - it is hardly visible link on IBM main site)
Segue - http://www.segue.com
Here are some free tools on web (a partial list based on my own searching of such free tools).
1. Watir – Web application testing in Ruby http://wtr.rubyforge.org/
2. Watir Web Recorder - http://www.mjtnet.com/watir_webrecorder.htm
3. Open source testing tools http://opensourcetesting.org/
(Note that these are “TESTING” tools not “AUTOMATION” tools)
4. Selenium- A test tool for web applications- http://www.openqa.org/selenium/
5. TestMaker – Framework for Test automation of web based applications and Web services. http://www.pushtotest.com/Downloads/
This is ever growing list – Good thing is that more and more people are investing in developing Open source tools. This will build up pressure on Commercial automation tool vendor to offer tools that are cheaper price and rich in functionality.
Shrini
Mercury - http://www.mercury.com
Compuware - http://www.compuware.com
Rational - http://www-306.ibm.com/software/awdtools/tester/functional/index.html
(It is quite surprising to see that the information about "once highly popular automation tool" Rational Robot - has gone so deeper into IBM site - it is hardly visible link on IBM main site)
Segue - http://www.segue.com
Here are some free tools on web (a partial list based on my own searching of such free tools).
1. Watir – Web application testing in Ruby http://wtr.rubyforge.org/
2. Watir Web Recorder - http://www.mjtnet.com/watir_webrecorder.htm
3. Open source testing tools http://opensourcetesting.org/
(Note that these are “TESTING” tools not “AUTOMATION” tools)
4. Selenium- A test tool for web applications- http://www.openqa.org/selenium/
5. TestMaker – Framework for Test automation of web based applications and Web services. http://www.pushtotest.com/Downloads/
This is ever growing list – Good thing is that more and more people are investing in developing Open source tools. This will build up pressure on Commercial automation tool vendor to offer tools that are cheaper price and rich in functionality.
Shrini
Further on the road in becoming a finest software tester ...
One of my blog reader asked my views on things that a good tester should invest on and personal traits and qualities of a good tester. I did write about it at my blog in this post. Here is a sequel to it
1. First important thing become a great testers is to question. Question around you anything - be it Door Knob, Watch, your vehicle, your access card, your rice cooker, Gas stove, TV, cell phone. Get curious about everything around you. Find bugs in everything that you see. World is a giant software and is full of bugs. Find bugs there. Then finding them in software will become fun.
2. Powerful observation: can you observe things that others don’t see? Can you notice that little color fading on the billboard hoarding? Can you find error or mistake in Prime ministers reported speech? How about in annual report of Infosys or IBM? Be aware of anything around see deep into it. Smart testers find bugs by observing carefully and are always curious about things - they never stop.
3. Memory and analytical skills. Take tests or training to improve memory. Solve puzzles, play chess, Suduko, jigsaw puzzles. Few of these tricks are mentioned in this blog post.
4. Becoming a good tester should be your goal - "logging tonnes of bugs" or "getting expertise in automation" - will come as side effects - they will be natural to you. Remember - there are no shortcuts - be lifetime student of learning, questioning and observation.
Have you read articles from James Bach and Michael Bolton on Critical thinking? If not, they should be good beginning. Don’t expect immediate results. Depending upon how committed are you and how fast you get into the mode of questioning and observation - it could take about 1-2 years to call yourself a descent tester. That is my rough estimate.
Current industry trends focus on process, factory thinking and want to make testing as routine and predictable - real testing is always interesting, instantaneous and dynamic. You can not do good testing by a following a “pre-laid out list” - You need to think.
Shrini
1. First important thing become a great testers is to question. Question around you anything - be it Door Knob, Watch, your vehicle, your access card, your rice cooker, Gas stove, TV, cell phone. Get curious about everything around you. Find bugs in everything that you see. World is a giant software and is full of bugs. Find bugs there. Then finding them in software will become fun.
2. Powerful observation: can you observe things that others don’t see? Can you notice that little color fading on the billboard hoarding? Can you find error or mistake in Prime ministers reported speech? How about in annual report of Infosys or IBM? Be aware of anything around see deep into it. Smart testers find bugs by observing carefully and are always curious about things - they never stop.
3. Memory and analytical skills. Take tests or training to improve memory. Solve puzzles, play chess, Suduko, jigsaw puzzles. Few of these tricks are mentioned in this blog post.
4. Becoming a good tester should be your goal - "logging tonnes of bugs" or "getting expertise in automation" - will come as side effects - they will be natural to you. Remember - there are no shortcuts - be lifetime student of learning, questioning and observation.
Have you read articles from James Bach and Michael Bolton on Critical thinking? If not, they should be good beginning. Don’t expect immediate results. Depending upon how committed are you and how fast you get into the mode of questioning and observation - it could take about 1-2 years to call yourself a descent tester. That is my rough estimate.
Current industry trends focus on process, factory thinking and want to make testing as routine and predictable - real testing is always interesting, instantaneous and dynamic. You can not do good testing by a following a “pre-laid out list” - You need to think.
Shrini
Subscribe to:
Posts (Atom)