Thursday, May 29, 2008

Mission - Test Estimation model

It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.—Fred Brooks

Steve McConnell opens his famous book "Software Estimation: Demystifying the Black Art " with above statement ... Development community is fortunate to have some one like Steve McConnell to help them with solving this puzzle called "Estimation"

I am working on developing a test estimation model. I think in today’s testing world, this is a biggest and the most complex testing problem to be solved. As I am reading and researching about this topic and formulating my initial thoughts, I am thinking about testing, testing models, questioning, thinking, modeling, bug investigation, test-stop criteria, non-linearity, size of testing work and so on.

Following is a list of challenges/questions that I am searching answers for …

1. Model of Testing: Testing is an evaluation activity as against Development that is a construction activity. How to think about estimating a “evaluation/Analysis” based activity?

2. Sapient Testing model that requires critical thinking, questioning, modeling is essentially a non linear activity where as development can be a relatively bounded activity of starting from a spec to a working code. For example if a particular testing task takes x hours, 10 such units will take 10x hours … we can not make such extrapolation right?

3. Sizing testing activity is still a big problem – what is the unit of testing task? Atom? Or molecule? What is the building block?

4. Tasks like bug finding, investigation, test design are difficult to quantify?

5. When do we stop testing? What are exit criteria for test cycle? This will set upper limit for testing scope?

6. How many test cycles we need? How do we estimate?

7. What are the human factors involved the process of test estimation?

8. How do we address problem of "slipped bugs" in case of plain and straight forward "scripted testing" (write test cases and execute them word by word and log the bugs and end of the cycle). What about bugs or tests that come out of specs, exploratory tests? (Something that we worry heavily in IT services Industry – how much testing to do? There are penalty clauses for missed bugs)

9. What about productivity numbers used. In estimation -> we first talk about sizing the work, then apply productivity figures to arrive at effort then split the effort into schedule. How do you deal with numbers like – 10 test cases executed per person per day? Should you believe them or question what a test case is?

10. Is estimation a guess work?

11. Is estimation similar to the work of a fortune teller, weather forecaster, Election analyst, Stock Market analyst, A punter in horse race, or a gambler? After all in test estimation we tend to predict the future right?

Any views? Are these questions important while arriving at a test estimation model?

Interesting quotes on Prediction:

"Those who have knowledge, don't predict. Those who predict, don't have knowledge.” - Lao Tzu
“Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” - Peter Drucker



Alan said...

Good post, but you may be fighting a losing cause. Test estimations are entirely coupled with the development work they are testing and I don't think they can be done in isolation. I whipped out some thoughts on this and posted them here.

Shamus said...

As Alan mentioned in a previous comment, I believe the test estimations are tightly linked with the kind of development work being done. Think of this analogy: A builder is contracted to build a house (i.e. think developer in this case), and they are given a blueprint of what the house should look like (i.e. which way doors should open, size/structure of walls, specific measurements). Any other specific information usually comes directly from the general contractor and future home owners (i.e. development manager and customer). The builder now sets out to build the house as directed by these guidelines.

Now, the house undergoes inspections at various stages in the building phase (i.e. think development stages like alpha, beta, etc. here) by the "testers", who in this case is the future home owners. Now, while they may have agreed to the blueprints, and written "directions" on what to do, when they do their "exploratory testing" of the house at these various stages, they are bound to come up with errors/changes that were just not foreseen given the earlier materials of blueprints and written instructions, even though they went over these documents time and time again with various stakeholders in the project.

What does this have to do with test estimation? Well, how does one really know when you are done building the house? Who decides how long the development/building, fixing, testing stage continues? There are situations that come into play that were not thought about and planned for in the blueprint and final estimation stage. Granted there are different kinds of house building as there is with developing software -- think custom homes v.s. cookie cutter homes as compared to custom developed software v.s. shrink wrapped software.

So I believe in this case, the software test estimation model might differ slightly as we get more into what kind of project you are working on. Trying to come up with an all encompassing model might help in general terms, but as with other kinds of "building", there are so many dynamics to take into account. As you mentioned yourself, tasks like bug finding, investigation are difficult to quantify.

We as testers have all sorts of tools, models that we can follow to give us general guidelines on what kinds of tests to plan for, and from that we can give crude estimates, but as any testers knows, once you get into it, you can be led down a path that was not thought of at the time for a multitude of reasons (technology changed, wrong stakeholders were involved, etc..)

As James Bach mentioned, trying to answer this question would take many articles, and is not an easy question to answer.