2 Notorious “E”’s of Testing

Efficiency is doing things right; effectiveness is doing the right things – Peter Drucker

Let me make a dig on these two notorious and the most abused terms - testing effectiveness and efficiency. Raj Kamal has a post that discusses this aspect here.

To me, effectiveness has a notion of "degree of serving the purpose". For example, we can say "this measure" taken to curb the inflation has been effective (means it appears to have served its purpose). This medicine is effective in slowing down the disease. So when talking about effectiveness with respect to testing - we should map the results to the mission of testing and say the techniques, approaches that you have deployed served their purpose or not. Remember, as testers we serve our stakeholders. Different stakeholders have different expectations from testing. Testers form their mission to suit those expectations.

So, in order to be effective in testing, we need to understand the possible stakeholders, their expectations and which one to focus on. That would lead to testing mission. Any testing that happens, should serve the mission. Along the way, testers employ different approaches, techniques, tools and methods. Few of these would be "effective" in serving the mission and hence serve the stakeholder the information that they are interested in knowing, few many not. Therefore, if you are thinking about articulating about effectiveness in testing, think about stakeholders first, then their expectations, then the testing missions, then approaches, tools and techniques, finally link all of them to the results that you produce. I am not sure if a simplistic metric of an equation that counts "reified" entities like bugs and doing some math (like taking cube root of sum of all bugs and so on). Bugs are not real things but they are the emotions and opinions of frustrated "someone" who matter in your project context. Can you quantify frustration?

Also remember, since there could be multiple stakeholders (hence multiple testing missions), your testing (approach, tools and techniques) cannot be effective for all missions. Accept this fact and you don't have to be guilty about it. This becomes very visible when there are contradicting expectations and hence contradicting missions. Learn to negotiate with the stakeholders, try to iron out conflicts and state in your test strategy which missions you are focusing on and why.

Now, let me come to the term "efficiency". You might have heard people saying – "this vehicle is fuel efficient", "this equipment is energy efficient", "this worker is efficient". To me, the term efficiency is related to the notion of degree of conversion of deployed input (human and machine capital) to desired outcomes. Let us take the example of an internal combustion engine and put the definition of efficiency into perspective - The ratio of useful work to energy expended. Like effectiveness, identifying the best way to convert useful energy for testing to useful results that our stakeholders value – is never a simple task. There is no one right way to things also. To serve multiple stakeholders and testing missions, we as testers need to employ a diverse set of techniques, tools and methods. Hence there can be multiple ways to define "efficiency" with respect to software testing.

While Peter Drucker provides a simple framework for thinking about these terms, I would say it is too simplistic and rudimentary model to apply it to software testing. We neither have one "right" way of things nor a specific set of "right things to do". There are many right ways to do things and there are many right things to do. Who defines the notion of "right"? Our stakeholders. Therefore, it is very important to align our work as testers, to what stakeholder expect. First step towards this is to identify our stakeholders. Have you done that for your project?

I would like to highlight another thing here. Since the notions of efficiency and effectiveness as applied to software testing, are multi-dimensional and cannot be reduced simple set of numbers. Avoid temptation to simplify these parameters into simple metrics defined in terms of entities like bug counts and test cases counts etc. Think broadly and deeply, consider multiple stakeholders and testing missions.

In short, effectiveness deals with "fitness of approach/tool/techniques to serving mission" and efficiency deals with "conversation rate of deployed capital (humans and machines) to intended output". In other words, effectiveness is about "how powerful is your way of doing things" and efficiency is about "how well you do things". Both of these parameters are important indicators of testing work and are multidimensional in nature.

Shrini