What's big deal with testers? Hit that punching bag !!!!

 Once in every 1 or 2 years - testing team or function comes under "question". I have seen this consistently for last 20 years. For business leaders, Heads of Business verticals - testing has been a punching bag. Whenever leadership changes or sales go low or profits fall - suddenly discussion on "efficiency" and throughput per person becomes an important metric. Why do we need such a big testing team? why cannot developers do all the testing required? Why not end users do the testing required? why cannot be automate this stuff and get testing head count out of equation? Why cannot we do crowd sourcing? - are the questions that business leaders ask. Testing leads and managers - start running from pillar to post to justify why they exist -- come up with all sorts weird metrics and process maps to show why testing required. But business leaders are hell bent on cutting that "extra" fat in the system. There go "testing team " out of the company.

What is big deal about testing or testers ? They ask. First of all let us ask what is/ required to do testing? Knowledge of the system or business domain or technology that used to build it?  Let us explore...

Business analysts know the product inside out as they have "defined" it (by writing down BRD - business requirements document). They should be the ones best suited to do all or required testing. No?

Developers too claim that they know system inside out  - as they have created it. They know nuts and bolts of the system where each one is. Should they be the best ones to test the system?

End users are best to test the system as they are the ones that eventually use the system they know what should be there in the application. Are the the best party to test?

Before we respond to above possibilities - let us also look at how industries shaped the views about testing. In software world (or software enabled businesses), broadly there are 3 types of industries - Software product companies, IT services companies and IT organizations or captive units of other (non software) businesses - likes of banks or pharma or Automobiles etc. Each of these have a very specific and unique "culture" about software and hence about testing.

Software product companies call software development as "Engineering" and all activities about creating a software is put under one umbrella "software engineering". Testing is part of "engineering". Pre-agile Era had testing as part of "development" life cycle and developers did all testing that they could. Agile and Post Agile (now DevOps era) killed "testing" function or skill or requirement by few beliefs or practices. Agilists extend logic of all testing should be done by developers and said "quality" is every ones' responsibility. Thru practices like TDD and formalized unit testing - testing as seen as excercising or executing every line of code (they wrote). Testing would mean writing code - hence programming and testing at some point merged into one skill. If you are in an engineering team you did not do coding means - you are either a business analyst or project manager (or scrum master - not ring master !!! :) 

IT organizations  or captive units are next big communities - looked at testing as either some necessary evil or something that programmers or technologists should do. Business leaders of these groups saw entire software development as an alien thing and kept a safe distance from it. For these software or technology was a support function and thought any investment in activities (like testing) related to software development as "distraction" from their main business function. Technology teams in such organizations too had a strange problem or approach. They thought all required knowledge of business domain or application resides with business so technology job is sincerely translating whatever business gave as requirements. No questions asked. They also propagated a view that technology testing is a shallow happy path and basic validation. They believed its not their cup of tea to learn business domain.  So where did  bulk of testing go? To business and End users in the form of "User acceptance Testing".

IT services companies did not have any of such challenges about thinking about domain, business, technology and testing. They simply supplied what their clients wanted. Hence IT services companies had mix of both IT org and product company cultures. IT services companies did not have to have their testing culture. In the beginning (pre-2000) - by riding outsourcing wave - went and told their outsourcing bosses - "testing is low risk work - we can take it away and do it for very low cost". I have heard pricing models of testing services on unit basis - xx $$ per test case. How crazy was that? When they ran out of business supplying "low skilled" (almost brain dead) "executors" of test cases (called manual testers) - they started selling them as business analysts. In that parlance - business analysis did some testing but knew business domain and could "write" BRD's. Even today many testers think that "business analysts" are one notch above testers in terms of salary and org hierarchy.  That option too soon ran out. These companies then started selling failed or low skilled developers as "automation engineers". This going even today..... skilled testing has been dying... not dead yet.

I ask -- what is big deal about testing - what makes testing  "tick" in todays situation? Share your views.

 

Sure Ways to Reduce Test Cycle Time through Automation

World is simple, we complicate the world for the sake of it – says a friend. There is a simple concept called "automation" and another one "testing cycle time". Why these terms can not be simply understood without much fuss and inquiry? he often argued with me. This friend is a manager and works for IT services company. A constant pouncing on me by this topic of test-cycle-time-reduction-through-automation and related hype, frustration and feeling of achievement, made me to think deep about basic laws or golden principles that govern this phenomenon of cycle-time-reduction.For the benefit of my blog readers, here, I make them public. Read them, understand them, implement them and be blessed. A caution here any criticism and cynicism about these laws will have harmful consequences to the beholder. These are golden principles and axiomatic about test automation!!!!

First principle - “About Testing”: Strongly believe that software testing is a deterministic, highly repeatable and structured process – somewhat akin to a step-by-step procedure to produce say a burger or car. You have to believe that given a fixed scope of testing, it always takes a finite and fixed time (effort) to complete testing. Needless to say, you have to have absolute faith in testing processes and standards. Your faith in the power of processes and standards making testing predictable and repeatable is an important success factor. It is also necessary to abandon any misconceptions you might have about relationship (or dependency) between testing and automation. You should treat testing truly as a mechanical, step-by-step and repeatable process to ensure Quality. Positive ideas about metrics to improve testing and consistency are - a sure bonus. You should resist and fiercely oppose any attempts to link automation and testing, claiming that automation can and should work independently.

Second principle – “Definition and Meaning of testing cycle”: Never challenge or probe definitions and meanings of word “Testing cycle”. Keep it (testing cycle time) loosely defined so that you can flip in any direction when confronted by a skeptic challenging your claim of cycle time reduction. The more the vagueness of the term “testing cycle and cycle time”, higher are the chances of meeting goal of reduction of cycle time through automation. Any variables and factors that make “testing cycle” somewhat unclear – should be ignored and not discussed. It is important to have faith in the fact that “testing cycle time” is universally known term and does not need to be redefined in any context. You would be looked upon as genius if you simply talk about “cycle time reduction” and omit the unnecessary qualifier “testing” or “test” (as in “test cycle time”).

Third principle - “Playing to the gallery”: Use words like “business needs”, “business outcome”, “business processes”, “success through business alignment”, “Time to market” etc., liberally during all communications related automation and testing cycles. Confront any opposition by skeptics about automation and its connection to cycle time by “calling authority” – say “Business/Market needs it”. Strongly believe in statements like “Automation will help releasing products and services faster – and hence will improve customer satisfaction and company bottom-line”. Your success in achieving cycle time reduction depends upon how often and how strongly you make reference to “business” and “business needs”. These powerful words that do all the magic required. You need right rhetoric to spread the message. Another important keyword here is “market”. Make sure you thoroughly mix up and use the terms “Testing cycle time” and “Time to market” interchangeably. The more you talk about “Time to market” and how automation can directly help it – more you look authentic and convincing.

Forth principle - “Motivations and Incentives”: Believe economists when they say “incentives” drive change and motivate people. How can this not work here? The last but not least of the measure that one needs to be aware of is to provide incentives for people to reduce cycle time through automation. Define performance goals of individuals involved in the automation in terms of cycle time they reduce through automation. Penalize those that question the idea and fail to meet the performance goals. The automation initiatives tend to be successful in their stated goals of cycle time reduction when they are integrated with performance goals of individuals involved in the game – especially automation team. Also make sure (when automation is done by a decentralized team) to define and impose the performance goals ONLY on automation team and suppress any attempts to include manual testing team. After all automation and testing are not related in anyway and it is the responsibility of automation team to bring about cycle time reduction. Right?

If you find these principles and ideas rather strange and are intrigued – one of the several possibilities is that you might be working in a software product organization as opposed to IT and IT services organization. The folks in software product organizations that build software – unfortunately approach automation some what “differently” and cycle time might not be a very familiar term to you.

If you are not from software Product Company and still confused about the ideas in post and you think they are inconsistent or incoherent with each other – do comment. That is a good sign that you are thinking about the topic.

Let me know if there are other ideas and principles that you might have successfully used to reap benefits of cycle time reduction through (and only through) automation – I would be more than happy to incorporate them with due credit.

Shrini

Can Automation reduce cycle time or improve time to market?

Continuing on this discussion on test effort and manual testing – there is another popular variation in which Automation tool vendors claim “improved time to market” and/or “reduced cycle time”. In this post, let me dissect these claims and see how true and credible is this claim.

First of all let freeze what one mean by “Time to market” and “Cycle time”. Let me define the terms as bellow in the context of a traditional/Waterfall model type of software development.

Time to market is a time window between the time your start development (requirements and so on) till you ship the product in market for general public consumption. Depending upon whether you are doing a major release or minor release, this window may span from few months to few years (as in the case of Windows VISTA)

Cycle (when used without any qualification indicates a cycle of development and Testing) time is a time window for a software product under development. The cycle time can be divided into Development time and testing time. A development cycle starts with deliberation of requirements, design of features to be implemented. A test cycle starts with a development team “releasing” the code for testing to begin and test cycle ends with test team completing the planned testing for that cycle. During this period development team can fix the bugs reported. Hence for all practical purposes, a cycle time implies a time window between start of design/requirements until test team completing the testing/development team is ready to release next build.

So it is apparent from above definitions that cycle time is a subset of time to market.


Automation can reduce cycle time (less of a problem) only if (check how many items or situations that “automation” can control)

• Automated tests run without reporting ANY bugs on the software (A bug reported by automation means – some investigation and confirmation by manual execution that the bug reported by automation is indeed a bug with the application)
• Automated tests DO NOT report any run time errors (A runtime error by a automated test means some investigation and re-run.)
• Development team INSTANTLY fixes any bugs reported by Automation and these fixes are so well done that there will be no further verification (manual or automated) required
  Manual testing (a very small portion indeed) that happens in the cycle does not report any bugs. All the manual tests pass.
• If manual testing reports some bugs those bugs will be fixed INSTANTLY without requiring any further verificationBug reporting time, triage, investigation (if any) is so small that they are negligible.
  

Automation can reduce/Improve time to Market only if –

• All the items mentioned under “Cycle time” and
• Business stakeholders do not worry about outstanding bugs. They take the decision to ship the product as soon as Automation test cycle is completed (because automation cycle is NOT expected to report any bugs). So the end of the automation test cycle, shipping the product is a logical thing to follow.

If you analyze these situations, you would notice that many of the factors that influence cycle time or time to market are not under the control of “test automation”. These factors are to do with development team, quality of the code, quality of requirements, number and nature of the bugs reported by both manual and automated test execution and above stakeholder’s decision about those bugs reported. One can not claim that there will be cycle time reduction or improved time to marked JUST because x% of test cases are automated. A big and unrealistic generalization - only an automation tool vendor can afford to make.

So next time when someone says automation reduces time (either cycle time or time to market) – do quiz them and say “What do you mean”?

Bonus Question : Can automation accelerate your Testing? If yes, under what circumstances?

Next Post : Can automation address the IT problem of Limited (human) resoures and tight deadlines?

Shrini