Thursday, May 29, 2008
Steve McConnell opens his famous book "Software Estimation: Demystifying the Black Art " with above statement ... Development community is fortunate to have some one like Steve McConnell to help them with solving this puzzle called "Estimation"
I am working on developing a test estimation model. I think in today’s testing world, this is a biggest and the most complex testing problem to be solved. As I am reading and researching about this topic and formulating my initial thoughts, I am thinking about testing, testing models, questioning, thinking, modeling, bug investigation, test-stop criteria, non-linearity, size of testing work and so on.
Following is a list of challenges/questions that I am searching answers for …
1. Model of Testing: Testing is an evaluation activity as against Development that is a construction activity. How to think about estimating a “evaluation/Analysis” based activity?
2. Sapient Testing model that requires critical thinking, questioning, modeling is essentially a non linear activity where as development can be a relatively bounded activity of starting from a spec to a working code. For example if a particular testing task takes x hours, 10 such units will take 10x hours … we can not make such extrapolation right?
3. Sizing testing activity is still a big problem – what is the unit of testing task? Atom? Or molecule? What is the building block?
4. Tasks like bug finding, investigation, test design are difficult to quantify?
5. When do we stop testing? What are exit criteria for test cycle? This will set upper limit for testing scope?
6. How many test cycles we need? How do we estimate?
7. What are the human factors involved the process of test estimation?
8. How do we address problem of "slipped bugs" in case of plain and straight forward "scripted testing" (write test cases and execute them word by word and log the bugs and end of the cycle). What about bugs or tests that come out of specs, exploratory tests? (Something that we worry heavily in IT services Industry – how much testing to do? There are penalty clauses for missed bugs)
9. What about productivity numbers used. In estimation -> we first talk about sizing the work, then apply productivity figures to arrive at effort then split the effort into schedule. How do you deal with numbers like – 10 test cases executed per person per day? Should you believe them or question what a test case is?
10. Is estimation a guess work?
11. Is estimation similar to the work of a fortune teller, weather forecaster, Election analyst, Stock Market analyst, A punter in horse race, or a gambler? After all in test estimation we tend to predict the future right?
Any views? Are these questions important while arriving at a test estimation model?
Interesting quotes on Prediction:
"Those who have knowledge, don't predict. Those who predict, don't have knowledge.” - Lao Tzu
“Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” - Peter Drucker
Tuesday, May 27, 2008
Continuing on this discussion on test effort and manual testing – there is another popular variation in which Automation tool vendors claim “improved time to market” and/or “reduced cycle time”. In this post, let me dissect these claims and see how true and credible is this claim.
First of all let freeze what one mean by “Time to market” and “Cycle time”. Let me define the terms as bellow in the context of a traditional/Waterfall model type of software development.
Time to market is a time window between the time your start development (requirements and so on) till you ship the product in market for general public consumption. Depending upon whether you are doing a major release or minor release, this window may span from few months to few years (as in the case of Windows VISTA)
Cycle (when used without any qualification indicates a cycle of development and Testing) time is a time window for a software product under development. The cycle time can be divided into Development time and testing time. A development cycle starts with deliberation of requirements, design of features to be implemented. A test cycle starts with a development team “releasing” the code for testing to begin and test cycle ends with test team completing the planned testing for that cycle. During this period development team can fix the bugs reported. Hence for all practical purposes, a cycle time implies a time window between start of design/requirements until test team completing the testing/development team is ready to release next build.
So it is apparent from above definitions that cycle time is a subset of time to market.
Automation can reduce cycle time (less of a problem) only if (check how many items or situations that “automation” can control)
- Automated tests run without reporting ANY bugs on the software (A bug reported by automation means – some investigation and confirmation by manual execution that the bug reported by automation is indeed a bug with the application)
- Automated tests DO NOT report any run time errors (A runtime error by a automated test means some investigation and re-run.)
- Development team INSTANTLY fixes any bugs reported by Automation and these fixes are so well done that there will be no further verification (manual or automated) required
Manual testing (a very small portion indeed) that happens in the cycle does not report any bugs. All the manual tests pass.
- If manual testing reports some bugs those bugs will be fixed INSTANTLY without requiring any further verificationBug reporting time, triage, investigation (if any) is so small that they are negligible.
Automation can reduce/Improve time to Market only if –
- All the items mentioned under “Cycle time” and
- Business stakeholders do not worry about outstanding bugs. They take the decision to ship the product as soon as Automation test cycle is completed (because automation cycle is NOT expected to report any bugs). So the end of the automation test cycle, shipping the product is a logical thing to follow.
If you analyze these situations, you would notice that many of the factors that influence cycle time or time to market are not under the control of “test automation”. These factors are to do with development team, quality of the code, quality of requirements, number and nature of the bugs reported by both manual and automated test execution and above stakeholder’s decision about those bugs reported. One can not claim that there will be cycle time reduction or improved time to marked JUST because x% of test cases are automated. A big and unrealistic generalization - only an automation tool vendor can afford to make.
So next time when someone says automation reduces time (either cycle time or time to market) – do quiz them and say “What do you mean”?
Bonus Question : Can automation accelerate your Testing? If yes, under what circumstances?
Next Post : Can automation address the IT problem of Limited (human) resoures and tight deadlines?
Saturday, May 17, 2008
I wrote about Goal displacement part I here. Surprisingly .. no comments yet ... :(
Here is example # 2
Software Testing Certifications: A conversation between a certification enthusiast (CE) and a critique (CC – that is me)
CE : Do you know about this certification for software testing, this is very popular in Europe and US?
CC: Yes, I do but I am not sure if this really helps in evaluating skills of our testers.
CE: I think it does. I have heard about exam it seems pretty exhaustive and coves all aspects of testing.
CC: Umm … what do you think the certification is testing? What and how does it evaluate testing skill of the candidate?
CE: Certification tests the “knowledge” of the tester, familiarity to terms, definitions and experience of the candidate…
CC : Really? How? How a certification tests knowledge and experience of the tester?
CE : By carefully selected questions and evaluation by testing experts. The certification exam is based on a body of knowledge and questions – mostly objective type. I think one can rely on the exam.
CC : So … you are saying an objective (yes/no and multiple choice) type question paper is used to test the skill and experience of a tester. Does the exam allows theory type of questions? Can the tester debate on an issue? Does the exam involve any kind of putting the tester into real testing situation? Does it observe the tester in action ?
CE : Come’on, how it is possible? Certification bodies have their limitations, they can not set up “practical exams” to watch tester doing testing and then rate. Do you want certification body to set up a audio/video facility to allow for debates, questioning and real time situation simulation?
CC : Don’t you think, that you would right and reasonable way to assess the skill of a tester not using a fixed set of questions that emphasizes on memory recall, reproduction of text of study material?
CE: That is right … Look at feasibility of having such exams … what about cost of administering such tests? What about evaluation? It would be costly. That is why the certification bodies might have created a scheme of tests that are easier to evaluate and conduct on mass scale. This will enable a relative cheaper exam and allow more people to take the exam Right?
CC: Well …good point. But what is the goal of certification exams? What they attempt to achieve?
CE: Evaluate and Assess skills and experience of a software tester.
CC: But your current exam seems to be structured in such way that it is easier to administer and evaluate.
CE : Ummm… that is correct.
CC: See this is what I call as Goal displacement. Certification bodies wanted to design, administer and evaluate a system of exam to evaluate and assess skill of a tester. But they seem to have taken the path of designing an exam system that is easier to administer.
The goal of software testing certifications is to act as a mechanism of evaluating skills in software testing and provide a benchmark for the talent in that space. As the popularity of such certifications grow, so grows the need for system for mass administration and evaluation. This causes the change in the examination pattern and evaluation mechanism to facilitate the mass administration. Any part of the exam that is good from skill evaluation perspective will be dumped if that does not lend itself to easier evaluation of exam results.
Software estimation models (especially test estimation models) often suffer from Goal displacement problems. Other day I told my colleage " The reason why we struggle with test estimates is that we use simplified model of testing while estimating but reality is different hence estimates are typically go wrong". For that he said -- "OK .... we know that actual testing models are complex (non linear and involve critical thinking/Questioning etc). That is why we use simplified models of testing ".... How strange ..!!!!
This is another example of goal displacement right away ... To make estimation possible - change the testing model itself ... use a simple one. So the goal of using a good and resonable testing model that happens to be complex (and does not lend itself that easily to estimation) is replaced by a goal of finding a testing model that is simple and easier to estimate ...
what should be your goal - good testing or estimation?
This post is for all those IT managers who are considering outsourcing testing and listening/talking to various vendor presentations on testing and automation.
I was in a conversation with a client other day. He asked me “Can you reduce the number of resources that you have deployed currently for regression testing of this application to say by half in next six months or so by using “Automation”?
I was not shocked to hear this since for past few years, I have seen many clients in IT application testing space have had similar expectations out of outsourced testing and hence on automation. In an Outsourced IT application testing domain, this is how a service provider positions Automation – A means to cut down the cost of testing (resources).
Client’s expectation of reduction in testing resources (with no reference to scope of testing) using automation in a specified time frame stands on following assumptions or beliefs.
- Regression testing is simply executing a set of test cases and reporting results nothing more than that.
- Regression testing gives the confidence that nothing is broken – all that was working previously is “intact”
- Automated test cycle means zero human involvement hence percentage of automation should directly result in proportionately reduction in human effort of testing. For example if 50% of test cases are automated, effectively manual test cycle time can be reduced to that percentage.
- Automation is turn-key. Once you have an automation solution developed, one can just keep using it unlimited number of times without any extra cost and effort
- Operationally, automated test cycle ALWAYS takes less (fraction of manual test cycle version) time – there would be no additional effort required to investigate results and chase script failures.
Let us look at costs of creating and owning automation (note that majority of these are of recurring types)
- Tool evaluation, proof of concept and other initial investigation costs
- Automation tool cost and recurring licenses/upgrade costs
- Resource Training costs
- Costs associated with automation environment (development and execution) – servers, applications, connectivity etc
- Costs associate with manual test cases cleanup, rework, test data, clarifications and any other effort to facilitate automation
- Automation development/ testing/review and acceptance costs
Costs of setting up and maintenance of automation execution environment
- Costs of investigation of automation results, failures
- Costs of re-runs (testing automation code fixes) and any required additional human testing
- Costs of maintaining automation code – design and structure of automation extent and frequency of application changes, automation tool changes (new versions etc) and change in application platform determine this cost component
How do these cost components compare with costs associated with human testing? Will an automated test cycle always be quicker than human testing cycle? Is speed of excution all that matters to you? Can these two versions of testing cycle be compared? Can one hour of automated testing (test execution) be compared with 1 hour of skilled human testing?
Test automation if not “appropriately” applied can be highly expensive and painful. If you are dependent on automation for achieving project GoLive timelines – be aware you could be risking the release. When someone says they can reduce testing effort by automation – be sure they are selling you something and they do not understand fully what is testing and what automation is.
Test automation is High risk item in business. Before you invest money or even before you make any business decision that based on automation capability – make sure you, as an IT manager is aware of darker side of automation.
If you are getting a sense that I am discouraging Automation and regression testing – Yes, I am. Considering the approach that today’s IT World is taking automation – I am against such approaches to Automation – but NOT to all other forms of automation that can happen.
So next time you sit in a vendor presentation and hear this "Test automation reduces manual test cycle time and effort" - Please ask "HOW" and "UNDER WHAT CONDITIONS". I am sure the presenter will have tough time in answering your question as these are not a common questions to these "automation snake oil" seller. In case if you get an answer, make sure to check relevance of those parameters to YOUR project and application context.