Wednesday, November 19, 2008

Perils of Quantification – what harm metrics can do for you?

This is a hurriedly written post (just to make sure that I do not lose the thought – "fieldstone" in Jerry Weinberg's terminology) – I plan to use this as a place holder for expanding ideas on this topic … Please bear with me for a while with this "being cooked" idea.

I stumbled on something that Michael Bolton said about metrics – in response a Google group discussion thread. Michael mentions "What you want to beware of, in particular, is turning rich information (stories about bugs, problems, risks, value) into impoverished data (numbers).

I think that is a great way (rather an interesting way) to think about "software metrics". To me, software metrics are great way to "squeeze", heavenly simplify" and "horribly trivialize" rich information about bugs, test ideas, problems, risks, value about software. While they provide a simplistic view of rich and often qualitative/subjective data/information – there is huge danger of "oversimplifying and information loss".

Many people argue with me saying "quantification" – associating something that we try to understand in term of numbers – is essential for science and engineering. Some even quote "you cannot improve anything that you cannot measure". I feel that the "urge" for measuring, notion of be quantitative /objective is simply "over emphasized". Let us consider the perils (ill effects) of quantification. Some entities lend themselves for quantification – say counting. Counting people, counting vehicles on road, counting fruits on a tree, marks a student scores in an exam. Many entities that are related to humans especially are difficult to quantify – tend lose lots of information when quantification is attempted. This is very true with software.

Consider following quantified information – what do you think? What do you lose when you quantify …

  1. One tsunami
  2. 1 billion Indians
  3. 1.3 billion people in the world below the poverty line of 1$ /day
  4. 8 million people affected with AIDS disease
  5. Software Quality of Six sigma
  6. In 2003 there were 6328000 car accidents in the US.


    6300 bugs in Windows 2000 ..

Notice that each of these numbers have rich information about loss of life, health of people, quality of life and so on. By squeezing rich information into a number, we lose the information. Numbers can be manipulated, argued in any way you want, they hide information, you can be cheated by numbers. Numbers are single dimensional where as information they tend to represent are often multidimensional.

"As proven by modern accounting scandals, you can make the numbers say whatever you want" – Mike Kelly

To be continued …



Rajkumar Pandian said...

Very nice thoughts...I had always had this problem, converting all the work that I've done to numbers.When I see my tests,bug verification work reduced to numbers(metrics),I was never satisfied and had a feeling that the numbers didn't speak much about the work I've done.

If I'm to report my work, it will be voluminous.So how do I make others (stake holders) understand the work I've done?

Shrini Kulkarni said...

>>> If I'm to report my work, it will be voluminous.So how do I make others (stake holders) understand the work I've done?

Tell them a story ... may be more than one. If you were to report an event lets say a sport event - can mere number justify? can simply score or win/loss result will help?

I know it is a challenge when most people around would be looking for numbers -- "just give me the status (the number) and get away" .... Give it a try... start with a small story...


Anonymous said...

It is always possible to quantify your work into numbers and being a tester, metrics says the whole lot of things and has miraculous effect on heads sitting in the conference.