When Testing Fails ...

Carl Segan once famously said “Science is a self correcting process – an aperture to view what is right”. Carl Zimmer in an article on Indian express says “… science fixes its mistakes more slowly, more fitfully and with more difficulty than Sagan’s words would suggest. Science runs forward better than it does backward.” According to Zimmer, checking the data, context and results of a published scientific work is not of much interest to journals that take pride in “first” publishing ground breaking new research. For scientists scrambling for grants and tenures – checking published work is not attractive and often an exercise not worth its effort. As a result of this, Zimmer says, original work/papers often stand with little or no investigation or scrutiny by peers. This surely, is bad for science. Zimmer, suggests the community to have focus on “replication” and setting aside time, money and journal space to save the image of science as self correcting pursuit of knowledge in Carl Segan’s words.

Well … does this have anything to do with software testing? I recall James Bach once saying “it is easy for bad testing to go unnoticed”. Like science and scientific community’s social fabric – software testing world (software world in general) has built-up layers of wrapping and packaging. It is easy to find some reasons in one or more of these layers in the ensemble of social systems leading to few missed bugs that cost few days of outage of say- a popular e-commerce website. Any query or investigation on the effect of outage or episode of missed bugs would set-up a nice blame game looping all the way to testers, developers, business analysts, test data, environment, lack of technical/business knowledge and host of other things. Like it happens in published science works – hunting down the culprit or group of culprits would be time consuming job. In any case it is regressive job and takes valuable resources from many “progressive” tasks. Right? In Zimmer’s terms – spending time on production bugs is similar to running backwards. Does testing work well when running backwards? Do stakeholders like it when testing is running backwards? Not sure if published research wrong or publishing a new perspective on the basis of an existing work can be as productive as hunting down a missed bug and trying locate the reasons for its birth in the first place.

While process enthusiasts, Quality assurance people, Sick-Sigma (oops… six sigma) people might protest and insist on full blown root cause analysis of all bugs reported in production. SEI/CMM folks might make it mandatory to document all lessons learnt from missed bugs and refuse to sign off the project closure unless this is done. In spite of this – in a fast paced life of software where cycles between conception and real-working-software, are shrinking, ever increasing number of platforms (don’t forget mobile platform) – who has time to look at root cause analysis reports and all those missed bugs?

I remember a business sponsor once saying – “I can’t wait for testing to complete, once the development and put the stuff into production. If something breaks, we have enough budget provisioned to take care of any production failures.” Here, failure to put (buggy) software in production is more disastrous than waiting for “perfect” software to develop. Back to layers of social system in software testing – it appears to easy to hide bad testing – unless you screw-up very badly, the chances are that your stakeholders will never notice what good testing can get them.

I often wondered looking at some testing that happens in some organizations – how they are managing to stay afloat with such bad testing? The reason is probably – when testing fails, it is difficult to attribute it to testing and call it as such. It requires courage. How many testing leaders are willing to admit their testing sucks without hiding behind fancy looking metrics, root cause analyze reports and charts? That is the effect of “software testing” as “social process”