Perils of Quantification – what harm metrics can do for you?

This is a hurriedly written post (just to make sure that I do not lose the thought – "fieldstone" in Jerry Weinberg's terminology) – I plan to use this as a place holder for expanding ideas on this topic … Please bear with me for a while with this "being cooked" idea.

I stumbled on something that Michael Bolton said about metrics – in response a Google group discussion thread. Michael mentions "What you want to beware of, in particular, is turning rich information (stories about bugs, problems, risks, value) into impoverished data (numbers).

I think that is a great way (rather an interesting way) to think about "software metrics". To me, software metrics are great way to "squeeze", heavenly simplify" and "horribly trivialize" rich information about bugs, test ideas, problems, risks, value about software. While they provide a simplistic view of rich and often qualitative/subjective data/information – there is huge danger of "oversimplifying and information loss".

Many people argue with me saying "quantification" – associating something that we try to understand in term of numbers – is essential for science and engineering. Some even quote "you cannot improve anything that you cannot measure". I feel that the "urge" for measuring, notion of be quantitative /objective is simply "over emphasized". Let us consider the perils (ill effects) of quantification. Some entities lend themselves for quantification – say counting. Counting people, counting vehicles on road, counting fruits on a tree, marks a student scores in an exam. Many entities that are related to humans especially are difficult to quantify – tend lose lots of information when quantification is attempted. This is very true with software.

Consider following quantified information – what do you think? What do you lose when you quantify …

1. One tsunami
   
2. 1 billion Indians
   
3. 1.3 billion people in the world below the poverty line of 1$ /day
   
4. 8 million people affected with AIDS disease
   
5. Software Quality of Six sigma
   
6. In 2003 there were 6328000 car accidents in the US.
   

   Finally
   

   6300 bugs in Windows 2000 ..
   

Notice that each of these numbers have rich information about loss of life, health of people, quality of life and so on. By squeezing rich information into a number, we lose the information. Numbers can be manipulated, argued in any way you want, they hide information, you can be cheated by numbers. Numbers are single dimensional where as information they tend to represent are often multidimensional.

"As proven by modern accounting scandals, you can make the numbers say whatever you want" – Mike Kelly

To be continued …

Shrini