Jamie Dobson writes this piece of "reality" with respect to statistics/metrics and numbers.
“... that human beings will always work toward their defined success criteria."
True and very revealing for all metric enthusiasts. Just let people know what they will be measured on - they will modify their work pattern and output to suite positively on the measurement criteria. For example, a test team is measured on number of bugs the team logs, you will see more and more bugs and if a test team is measured on number of test cases they execute, you will see testers executing increasing number of test cases.
One thing that happens is what I refer as "goal displacement" - Goal of doing “required” work getting displaced by doing *that* work in *that* way as described/interpreted by measurement criteria. Can you see a problem here?
When working with social/cultural setup involving human beings, introduction of "monitoring/measuring", typically causes a "shift" in a overall behavior of the group towards "what is being measured" instead of "what is required". We tend to believe that people behave the same with or without a measurement system in place – we are wrong
This is the side effect that I am referring when a metrics program is introduced in a software project setup....
Are you aware of this? What steps can be taken to address the side effects?
Shrini
8 comments:
You can look up Hawthorne Effect in wikipedia to read more about this.
The two most commonly used methods for combatting this are metric testing (aka common sense) and creating normalized metrics.
The most common test is an adverse behavior test where you just say - what could happen if we measure this. That may lead you to not use a metric, or to balance it against an opposing metric.
For example, if a dev team is measured by how many lines of code they write, they are going to write lots of crappy code. A better metric may be (lines of code / defects found by test). In fact, just about any time you only use ONE metric you will likely run into problems.
There's also a concept of metric privacy - i.e. you don't have to expose everything you measure to the entire team.
Some other questions to ask yourself when creating metrics include:
Can you describe a good result?
If you analyzed this metric in the future, would it be useful?
Does the measurement mean the same thing to everyone? Does it accurately answer a question?
Is it credible? Is it clear?
There's a start.
Srini,
I totally agree that there would be a shift in the behavior of the group towards "what is being measured" instead of "what is required" once metrics are brought into software project setup if the group does not understand the value and importance of metrics in their project.
If the group is educated on:
> how the metrics/statistics would help their project
> how these statistical measures would help the individual increase their 'efficency, performance' by concentrating on the weak points in the project
then the group would readily embrace the implementation of metrics/statistics as they understand what is required now.
So, once the group understands that the metrics/statistics is to help them and not to judge them, they would definitely concentrate on "what is required" and not on "what is measured".
Any Comments?
-Ajay
Hi Shrini,
We have two goals here.
1. Minimizing Goal Displacement
2. Implementing Metrics
One way which i see is to categorize the Bugs Logged by the Team into the category as to what impact the bug they logged would have in later stages. A tester might log a very simple bug, others might log a scenario which is not common and would have led to major rework at later point of time.
What i feel is that more important will not be the number of bugs they are logging but the quality and severity of the bugs. Hence the Team if told these guidelines along with metrics then definitely we will not see an increase in the number of bugs being logged, however a possibility of good bugs being logged might increase which is benificial.
Same way it also applies to the development team, if we check lines of code / defects found by test then they also will not tend to write crappy code which otherwise would have been written in case the metrics explained to them was just that they would be measured by Number of Bugs or lines of code.
Hence The Goal displacement can be minimized to a certain extent.
Your views?
Madhukar.
@Madhukar -
Key goals (questions) I am trying to answer/solicit answers are -
1. How measument system impacts that way people work and their output?
2. At one side we have people saying "We got to measure otherwise how will know where we stand" and other side are the people like me "what other (bad) things will happen if you measure".
How to balance these two sides?
3. Since the statement "human beings will always work toward their defined success criteria." is true - how to deal with challenge of making people work towards intended output (as desired by stakeholders)rather than towards work based on "arbitrary" difined success oriented criteria.
So at this point of time I am not thinking about "implimenting" a successful metrics program - that is too far.
As you have mentioned, my problem is about "how to reduce goal displacement"
Shrini
@Ajay
>>>then the group would readily embrace the implementation of metrics/statistics as they understand what is required now.
I don't think that is the case, I do not think we can address the problem of goal displacement by proper education and communication.
These are explicit measures but the cause or problem is implicit.
No one will accept explicitly, "I will work in a way to produce best results as per established metrics"
It happens by inner human programming. This inner programming of an individual gets amplified and grows in leaps and bounds in a group/social setup. Thus the issue gets complicated.
Stakeholder's assurance that "metrics are not for judgement" might help in some situation but might prove to be counter productive as people will really start thinking about "why metrics" and take their mind away from work (a displacement of different kind)
I think the problem being an implicit, solution and communication of solution should also be implicit.
One way to reduce the ill effects of goal displacement is the reduce or eliminite the gaps between "what is being measured" and "what is required".
A real tough problem in software !!!!
Shrini
I completely agree on "goal displacement".Its a very common scenario.
What can be done is making team aware of what metrics is meant for and how beneficial it is?Usually this information does not flow down to the complete team with only the leads and managers being aware of this.
Alan is right, the Hawthorne effect is interesting.
That's for creating a discussion, but, why doesn't anyone comment on my site! You should have linked back to me!
Regards, Jamie.
http://jamiedobson.co.uk/?q=node/56
Post a Comment