Wednesday, May 17, 2006

A note on Test estimation ...

I was discussion with one of my colleague about test case estimation where a question was raised about “how to handle creep in terms of number of test cases? Let us say initially if we estimate x number of test cases and this number at later point of time becomes 2x”

My response was -

Estimation is always an iterative process. You typically make estimation in terms of test case early in the test cycle - that is in planning phase. Make all the stake holders clear that "Estimates are based on current understanding of the application and test requirements and are likely to change". Have this as main "Disclaimer" in the test plan or estimation document.

This will give flexibility later in the cycle to ask for more resources and time. If you make your PM and other stakeholders crib or complain - Tell them as test team progresses, test team will get more understanding and initial estimates are likely (mostly will) change. Just say "I told you so" and show them the line in the test plan.

This is diplomatic way of handling future uncertainties in test estimates

Other side of this situation is that - test cases number increasing from 2000 to 4000 will happen in test design phase. So you can write all those 4000 test cases if time permits and execute only important ones during execution phase.

So far, as I have seen in this industry - Test effort estimation happens by experience, manipulation and intelligent planning. It is more of negotiation and communication skills than any science or proven method.

When development - in spite having about half a dozen estimation techniques, international bodies or knowledge, certifications like PMP - fail more often in estimating development effort, we, in test while estimating, can start with some good value and keep it open for future updates.

Please spare testing community from getting subjected to scrutiny for estimation... We are learning

Shrini

4 comments:

Raj_Narayanan said...

'Requirement Creep' and hence 'Testable Requirement Creep' are the Project Liabilities, requiring change in the entire Project Plan and hence the Test Plan. Additional requirements are overheads to the project, which any Project Plan must include in words before sign-off by the customer. Any requirement that creeps in after the sign-off deserves the revision of the Project Plan. Horse-trading has to be done with the client concerned and the plan ought to be put forth to them.

The whole issue boils down to the crux of the problem - Effort Estimation, which depends on various factors;
* Experience of the test-team
* Complexity of the application
* Budget (Time/Cost)
* Management will et al

Estimation is something related to speculation (based on proven scientifc methods/heuristics). Speculation is a pure probability which is again a gamble.

As the saying goes - "Fail to plan; plan to fail", planning is required but nothing in this world is perfect. Certain amount of deviation goes in anything - by nature.

Realise this we are mere mortals. Athiests please excuse me :-)

Raj_Narayanan said...

'Requirement Creep' and hence 'Testable Requirement Creep' are the Project Liabilities, requiring change in the entire Project Plan and hence the Test Plan. Additional requirements are overheads to the project, which any Project Plan must include in words before sign-off by the customer. Any requirement that creeps in after the sign-off deserves the revision of the Project Plan. Horse-trading has to be done with the client concerned and the plan ought to be put forth to them.

The whole issue boils down to the crux of the problem - Effort Estimation, which depends on various factors;
* Experience of the test-team
* Complexity of the application
* Budget (Time/Cost)
* Management will et al

Estimation is something related to speculation (based on proven scientifc methods/heuristics). Speculation is a pure probability which is again a gamble.

As the saying goes - "Fail to plan; plan to fail", planning is required but nothing in this world is perfect. Certain amount of deviation goes in anything - by nature.

Realise this we are mere mortals. Athiests please excuse me :-)

Anonymous said...

"Estimate means to form an approximate judgment or opinion regarding the worth, amount, size, weight, etc., of calculate approximately."
The word itself is used to mean a rough figure, so I don't see any need to defend what has been put forth. Also it is well known fact that effort estimation is approximation, no matter what you adopt as method for achieveing it.The only way to get nearer to actual figure is to plough back what one learns from first estimate to next time one does the estimate for the same project (within the project lifecycle) ie you need to repeat the excercise a number time (at appropriate phase of the project).One will have better estimate of effort if he/she adopts the corrected estimate next time in a similar project (but then there are those who think no two project is similar!)
-shrinath

S Basu said...

In order to prevent the Creep in the Test Effort Estimation the best way is to get the Test Plan reviewed by the Key Reviewers and stake holders. The initial estimation though should have the disclaimer as pointed out by the author.

Also keeping the disclaimer in the estimation sign off document truely provides the test manager to burgain for more resources or time if needed based on the Budget and Management will of the project.

Even if someone uses proven models like TPA for Test Estimation, someone needs to be judicious in terms of selecting the number of TCs and arrive at the effort estimation by keeping an eye on the cost/time factor.