Mission - Test Estimation model

It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.—Fred Brooks

Steve McConnell opens his famous book "Software Estimation: Demystifying the Black Art " with above statement ... Development community is fortunate to have some one like Steve McConnell to help them with solving this puzzle called "Estimation"

I am working on developing a test estimation model. I think in today’s testing world, this is a biggest and the most complex testing problem to be solved. As I am reading and researching about this topic and formulating my initial thoughts, I am thinking about testing, testing models, questioning, thinking, modeling, bug investigation, test-stop criteria, non-linearity, size of testing work and so on.

Following is a list of challenges/questions that I am searching answers for …

1. Model of Testing: Testing is an evaluation activity as against Development that is a construction activity. How to think about estimating a “evaluation/Analysis” based activity?

2. Sapient Testing model that requires critical thinking, questioning, modeling is essentially a non linear activity where as development can be a relatively bounded activity of starting from a spec to a working code. For example if a particular testing task takes x hours, 10 such units will take 10x hours … we can not make such extrapolation right?

3. Sizing testing activity is still a big problem – what is the unit of testing task? Atom? Or molecule? What is the building block?

4. Tasks like bug finding, investigation, test design are difficult to quantify?

5. When do we stop testing? What are exit criteria for test cycle? This will set upper limit for testing scope?

6. How many test cycles we need? How do we estimate?

7. What are the human factors involved the process of test estimation?

8. How do we address problem of "slipped bugs" in case of plain and straight forward "scripted testing" (write test cases and execute them word by word and log the bugs and end of the cycle). What about bugs or tests that come out of specs, exploratory tests? (Something that we worry heavily in IT services Industry – how much testing to do? There are penalty clauses for missed bugs)

9. What about productivity numbers used. In estimation -> we first talk about sizing the work, then apply productivity figures to arrive at effort then split the effort into schedule. How do you deal with numbers like – 10 test cases executed per person per day? Should you believe them or question what a test case is?

10. Is estimation a guess work?

11. Is estimation similar to the work of a fortune teller, weather forecaster, Election analyst, Stock Market analyst, A punter in horse race, or a gambler? After all in test estimation we tend to predict the future right?

Any views? Are these questions important while arriving at a test estimation model?

Interesting quotes on Prediction:

"Those who have knowledge, don't predict. Those who predict, don't have knowledge.” - Lao Tzu
“Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” - Peter Drucker

Shrini