To tweet or to blog ...?

Due to me crazy traveling and work, I have not been able to write as frequently as I would have liked, for the blog. At least half a dozen potential blog posts are waiting to see the light of the day from my side. It is just not working out.

I have been on twitter (for starters - you can take this as a quick blogging or microblogging) quite active in recent days. Happy to see many following me now. It is suiting me for now as I need not feel guilty of not being able to discharge my duties as a blogzen (no!!! this word has not yet been used by someone previously)of software testing blogosphere.

So till, I get to full time blogging - please follow my thoughts on twitter. I have added twitter feed on this blog to facilitate for my readers to catch up with what I am working on ....

Thanks for being my blog reader ....

Shrini

10 ways to make automation difficult or ineffective

Here goes another 10 items list for automation. If you are in IT or IT services space and manage/deliver automation solutions -- make sure you stay away from these as these items have high likely hood of making automation inefficient/ineffective and difficult.

This list is an extension of a topic and this list (of 10 items again) for test automation outsourcing

10. Wild Desire to automate 100%

9.Attempting to automate existing test cases without scrutinizing them for “suitability” to automate

8. Mapping test case to script 1:1 linear model – falling prey to deceptive traceability and gold plated reporting.

7.Not building automation solution bottom-up , unidentifiable building block of the solution.

6. Trying only one type of automation or attacking only one layer of the application – Farther you go from code, messier it gets.

5. Focusing only test execution related tasks

4. Treating automation as scripting – ignoring “generally accepted good software development practices for hygiene.

3. Failure to involve developers from the beginning – Not attempting to testability or automatability of the application.

2. Jumping to automation to speed up testing or save cost before fixing testing problems – inadequate, inefficient and broken.

1. Failure to arrive (formulate) at the right mix of human testing and automated test execution.

0. Using Automation as solution to testing problems.

I reiterate that these are applicable mostly to COTS driven, GUI functional Testing automation that is typical in IT/IT services environments. WI might have to rewrite some of these for xUnit type formalized unit testing (that is also automation and some call it even as "testing").

Shrini

Is this a bug?

I was flying from Cape Town to Bangalore through emirates flight. It is convenient to online check-in. I do it as I can choose an aisle seat. But for the second time, I got into problem while doing online checking for emirates. Probably the Internet connection was slow – in both occasions, emirates online application did not respond and I had to close the browser after 5-10 minutes of frustrating wait – staring at screen.

Here is the bug that frustrated me…

· I try to do an online check in and would like to change the seats.
· Application hangs when trying to save the changes.
· Close the browser.
· Try again to do check-in
· Get a message that the passengers have been checked in.

Fine – how will I know what are my seat numbers? How do I view my check-in details.

Apparently there are no easily reachable ways to gather information. Probably there is none. How do I search where is “view check-in details” or “view eBoarding Pass” or something similar? I tried site map, tried “Help” and tried “search”… could not figure out the link for viewing check-in details.


Is this a bug? If you are a tester will you catch this bug? If you are a developer will you accept that this is a bug? I am sure most people will say “if this is an intended functionality (I think, it is), then it should be documented requirement specifications. Once it is there, tester can write the test case and developer will make sure that the functionality is coded and tested”. Some testers might say this is “nice to have feature” …

What might have happened here? Requirements problem? Development problem? Or a Testing problem?

10 ways to reduce cost of software testing

In current economic situations, IT folks worry about one thing – “reduce cost”. I have been frequently asked “how to reduce testing cost”. A no brainer answer would “do not do testing … at all”. How many buy this idea … can current breed of IT applications sustain with less or no testing at all? When I use the term testing –I am referring to “non-programmer” testing.

Here is my draft list of suggestions ...

1. Closely work with developers, do some parallel testing with them as the product/feature is getting developed
2. Identify and eliminate non-testing activities that occur in the name of process, documentation, management, metrics etc.
3. Analyze and profile every application under the portfolio to determine “stable” and “well tested” areas of the application. These areas should receive the least or no testing effort.
4. Analyze the test scripts suite and remove redundant, worn out ones. Aim to reduce scripted test repository as small as you can.
5. Review and reduce “regression testing” on the basis of “well tested/stable areas” of the application
6. Switch from resource intensive and highly scripted testing approach to highly improvisational exploratory /rapid testing approaches
7. Plan testing in small but frequent cycles (Session based exploratory testing approach) – reduce planning and management overheads
8. Analyze and reduce the usage of costly tool licenses - especially those do not help in testing directly (test management tools)
9. Cut down on lengthy test plans, testing reports, dashboards – switch to simple but frequent test reporting.
10. Simplify defect management process – reduce defect life cycle – resort to informal/quick defect communication.


Some this advice might look like a simple common sense (eliminate waste, focus on tasks that impact end result DIRECTLY). With so much selling happening about “testing tools”, “factory models”, “cheap and best testing services” – any common sense is difficult to come by.

How would IT community react to these suggestions – most likely response would “This would not work, how can we reduce testing, those test cases, processes, metrics, management practice?”. These suggestions would be most likely to be rejected on the grounds that testing cost needs to be reduced without “compromising quality”. Many IT folks think that quality comes from test scripts, processes, metrics, testing tools, automation etc. I am afraid quality is not such a simple thing.
Again, there are no free lunches here … if you are thinking about reducing cost of testing, there are always risks of impacting quality (roughly goodness or confidence in the product) in one or other way. If you approach the problem (cost vs quality) from a quality side (improve testing - test better, deeper and wider), then the chances of achieving good quality and also “some” cost benefits are more likely. However, if you approach it from “cost” side of the equation – you might do achieve that albeit some impact on overall goodness/quality of delivered product.

Note that some suggestions mentioned in this list call for some smart testers who can think on their feet, work with least supervision, least (optimum) documentation and processes and so on. I think, the focus should shift from process, tools, management, documentation to Skill. There can be problems in getting such resources in IT scenario (especially in outsourced/offshored world)

You have choice … which side you would like to approach the problem ..?

[update 20/Apr/2009] A colleague of mine reacted to this list saying "These are too risky suggestions and he would not recommend any of these. Business prudence is totally missing".

I think he was expecting to see some "low risk and high return" type of suggestions - like those "cheap" and "best" items. I still do not understand - there can be no risk free ways of reducing (testing) cost -unless you are totally spending like crazy without any thinking. We do not seem to have such risk free - free lunches - why fool ourselves and the client in believing such "non existent" things?

Another suggestion that came up was "Let us use standardized processes". How standardization can reduce cost? what is the cost bringing in standardization itself?
May be the expectation is that standardization will make each tester behave identical to another like robots. Are robots cheaper? may be? may be not .... They at least do not whine about working on week ends :)

Shrini

When "Process" stops working for you ....

Other day I overheard a Test manager speaking to his team “As a CMMi Level 5 company, I don’t think we are following processes. We often talk English rather than showing supporting metrics that are back bones of any CMMi Level 5 organization. If you don’t measure then how can you improve? Considering the economic slowdown, it is high time that we should start showcasing our continuous productivity improvements or else we will lose the client"

What is happening here?

Most managers somehow (more so in current economic situations) confuse skill, human ingenuity and expertise to metrics/measurement. When customer cribs about “value” and quality of work delivered – she really is cribbing about people and their skill (not about metrics and measurements). When people hear about customer cribs … managers suddenly jump and say “let us collate some metrics and show client that we have delivered the value (which they will dump eventually)” and push the core issues about skill below carpet. This “hide and seek” game goes on until we lose the client. This pattern has to break and unfortunately I have no simple solution for that (probably no one has). Few of us appear to know the root of the problem now.

If following processes would ensure quality and being very serious about metrics is HOLY – then our problems would have been solved long ago …. Why people do not follow process? Is it because they are so tough and stress full to follow? Is it because they are difficult to understand? Probably people follow process and we have stopped being critical of whether process is doing any useful thing are not …. That is the start of the problem. Glorifying process beyond its own utility (ask process – it would probably say … beyond this, I cannot add any value). I understand process (whatever is the definition) provides some common framework within which people with diverse educational/technical/social background work to produce consistent output so that whole thing can be managed easily. Beyond certain point (this limit might vary from context), process cannot help any further. It calls for people's skill to deliver - process then becomes an enabler or mere Hygiene factor. Just walking or eating alone can not keep you healthy all the time. Do you know where is the limit beyond which "following process" can no longer help?

There is a big fuss about “using English than numbers” … Why there is so much faith on numbers? Why qualitative subjective wordings are such a waste? Why not we express everything in numbers all the time – our hunger, happiness, intelligence (yes there is IQ test), pain, sorrow, emotion (yes there is emotional quotient), commitment, enthusiasm, creativity and what not all human attributes are so rich and multi dimensional that poor numbers can express a minute part of them. And we refuse to use qualitative measures saying that “objective is better than subjective”. Many would like humans to behave as if they are machines so that they can be objectively measured. A sad reality…. Perils of advanced economic world. Hunger for objective interpretation of human attributes is probably has reached its crescendo. I am waiting for the downfall of that raise. Will it come?

There is a big deal about “improving productivity in testing .. We must meet SLA’s and show continuous improvement in productivity”. I am STRONG opponent of usage of the word “Productivity” in testing in general terms. When people say productivity, they typically refer to speed – number of units produced per unit of time. Much like in a shop floor assembly line. There might some portions of testing that one does that are “speed sensitive” but by and large skilled testing is not about “speed” more than it is about “coverage”, “identifying tough to find problems”, “asking right questions”, “seeking information”, “building on available information”, “investigation” and many more. Probably not more than 5% of good testing is speed sensitive… most of it is not … then what is the meaning of “productivity” when it is applied as “serious generalization” to all testing. I PROTEST ….

Finally, come’ on, let us accept there many ways we can improve (many) things without measuring them (at all) at least in poor numbers. We all do it in our day today interactions with our near and dear ones in family and those out side in society. So there are clear exceptions to the statement “you can not improve if you can not measure”. I strongly oppose the statement. Too poor generalization that suites machines and mechanical constructs well, than human beings in a social structure.

C-DLICE’ing in software Testing

Let me take credit for making this pneumonic up: C-DLICE. I was listening to Michael Bolton's video interview on youtube. He said testing is more than verification, validation and confirmation. It is about Challenging claims, Discovery, Investigation, Learning and Exploring. Any skilled tester would do one or more of these activities as part of testing. By explicitly chaining them in a pneumonic, a tester can focus on a specific aspect of the interaction with the test subject.

Let me expand the pneumonic –

C – Confirmation. Other than traditional words like Verification and Validation (whatever may be the meanings of these terms) most people on this planet think that sole aim of doing testing is "confirmation". It is seen confirmation of claims made about the product, conformation of what developers "felt" that they created in response to requirement specifications that received and interpreted to their best of abilities. The confirmation about some specific user expectations (assumed to be routed through specifications into the software product). In basic form confirmation is somewhat like "Click this button, such and such thing should happen – Does it happen?" While confirmation is important aspect of testing, any testing that focuses on confirmation will become boring, brain dead and poor way to think about testing. Notions like anyone can do testing, process plays important role in testing, testing without test cases and requirements is not possible – are creation of confirmation oriented testing. I will not dwell upon challenges in confirmation oriented testing. And there is a big deal called "reference" against which you confirm – specifications. If your reference is wrong, ambiguous and incomplete – so will be your confirmation. That is the weakness of confirmatory testing.

Though my pneumonic is more about DLICE, I will still keep this "C" in there to remind us that confirmation may be as important as other letters in the pneumonic.

D- Discovery. While we test tester, we discover information about software, certain behaviors. It is like discovering an unknown island. As product grows bigger in size (in terms of codebase), discovery becomes important. No user would use the software as per the user manual. Discovering way in which software could be used and misused is important aspect in testing. Discovering is about finding information about unknown areas. For a growing software application, every time there is more to test than before – more to cover than before. Under such circumstances, you constantly discover the application, its variations, behaviors and so on.

L- Learning- This is a freaky one. A significant part of testing is implicitly spent on learning about everything that software under test. Be it business domain, technology domain, community of users using the software, cultural and social set up in the organization producing the software, we learn all time. Learn about how the software is constructed, deployed, distributed and so on. Often, I have seen people downplaying "learning" aspect of testing as they would like to position themselves as "experts".

I – Investigation As testers we investigate claims about the product. How people perceive the product? Investigate inconsistencies. Investigate bugs, Investigate impact of a new technology, software change on the over image of the software. Investigation is about focused information gathering, analysis on certain events. Examining the evidence etc. Investigation starts off as open ended.

C- Challenge (used as verb) – As testers we need to constantly challenge the assumptions, beliefs around how people think about software. What each stakeholder thinks about the capabilities of the software? Challenge the premises and so on. Challenging would require the design of tests, experiments etc to expose the weakness of an aspect of software.

E – Explore – Somewhat similar to Discovery, Exploration enables any information gathering exercise. Explore market conditions. Exploring is about taking a tour. Exploration helps in modeling the problem space. Exploration is more open ended than investigation.

Notice that each letter is has some overlap with others. You can learn while discovering or challenging a claim or exploring a feature. You can investigate something by exploring it or discovering it. You can challenge something by investigating it or learning about it and so on. One way to think about DLICE is – Discover like Magellan, Columbus, Learn like a learning a new language, Investigate like Sherlock Homes, Challenge like Lawyers, Explore like exploring moon's surface or deep African jungles.

Few practical themes to apply dlice'ing

1. When there is a new thing most people around you know little about - something that you do not understand well , then – Discover, Explore and Learn
   
2. When there is something that several others know but you do not – Learn through exploration, discovering
   
3. When there is"suspense" or "mystery" about a thing – Investigate – a defect, a strange behavior etc.
   
4. When there is some that is "well known" to you (you are pretty sure) about some claim – Challenge it and prove your point ( backed up by prior discovery, exploration, investigation and learning)
   

So, next time you feel bored doing testing, try switching your focus … try doing some investigation, discover new ways of using software or explore an area of software and so on. You would find that testing is always interesting but you were told about only one dimension of it (confirm, find bugs, check it passes tests) so you felt low or bored about doing testing that way ….

HAPPY C-DLICE'ing

Shrini

Context Driven Testing gets a boost – to grow stronger…

It is a fantastic "new year" gift (and also week end feast to many like me who is spending better part of Christmas/new year in front of laptop) to all testers… Dr. Cem Kaner (along with James Bach) has posted about context driven testing with some new definitions and articulation. I belong to context driven testing community with many others. Context driven testing community has proclaimed its philosophy, guiding principles since its inception. As years went by, various groups of people (quite a few "unidentified" ones) started pushing false propaganda about our community. This I believe might have driven Cem and James to "rework" the original principles and overall articulation about context driven community.

I have personally been part of many discussions, where I heard people just attacking the theme of context driven testing by saying "Everything in this software world happens with the context. Only a fool would work without context … so talking big and calling oneself as context driven tester is no big deal. Context is forever one … every one applies the context to best of their ability and knowledge". The new articulation of "Context aware" - attempts to describe such people. If you think about a practice first, then tailor/modify that practice to the context – you would be a "context aware".

Another interesting point this post makes is that some people in agile software development community have found so much common with context driven testing that they were claiming agile and context driven testing as one and the same. With so much focus on people and their interactions – agile and context driven testing can be said to have common roots – "people focus" (as against process/standards focus). I would say Cem's articulation could have been much stronger when it comes to the differences between agile and context driven testing. Insistence of 100% automated unit tests, compulsory standup meetings, TDD is a must and all sorts of "standard" stuff is clearly deviation from agile manifesto (choice of people over Processes) as articulated by James Bach in his STAR West 2008. I see context driven testing community distancing from agile community on these areas …

Also look the new, cracked definition of context driven testing – "Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing." I am sure most of my friends who are not in the CDT (context driven community) will say … oh yes … that is a common sense. We work for stakeholders, customize our testing approaches to context … what is big deal? Why are you making common sense a big thing?" My response to such people would first "so called common sense in this case is not so common … so there is a big deal here. Secondly, thinking about practices first then context is "context aware" not "context driven".

Critical Focus on "best practice" has been rather mild I would say (appears in only 4 times in the post). Cem, apparently left the proponents of best practices, lightly by saying "Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context…. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven."

Context imperial testing is somewhat similar to what I normally refer as "goal displacement" (a term I learnt from James Bach). Instead of designing and adapting the practices to project /organization/group context, Context imperialists would RATHER change the project/group/organization itself to suit the "best practices that they are aware of. A recent example … someone said, we cannot afford to do testing this way as it will not allow us to collect the metrics that we need, so let us change the process so that we can collect and use the metrics". It is sad state of affairs that, people will get away with such "context imperialism".

Context driven testing community's view point on "detailed specifications, detailed test script documentation etc" has always been perceived as "means to promote exploratory testing". Many people, whom interacted, would immediately say "Oh!!! You mean to say do exploratory/adhoc testing then" in response to my statement that "In context driven community, we do not insist on detailed specification … In fact some of us think that is a crime to refuse to test without specifications". In reality, the crux of the matter is, in CDT, testers need to cope up with whatever information that they get and start from there to gather information that they need. In those situations where time, information, people with knowledge - are rare commodity, context driven testing is clear winner. It prepares its tester for such eventual realities.

Finally the assertion that "There are no context-driven techniques" – should put all those statements/viewpoints to rest such as "Exploratory testing is a context driven testing technique". Neither ET, nor context driven testing is a testing technique in itself…

Overall, this post is a great milestone in the history of Context driven testing … should be mentioned in context driven website.

Shrini