This is cricketing time (World Cup at West Indies starting from March 11). Being one the avid cricket watcher and enthusiast – I will be eagerly hooked on to idiot Box for all matches ….
When we are talking about cricket, boundaries, sixes and all excitement of cricket – How can I forget – testing? Thanks to Pradeep, a fellow Rapid/Context Driven Tester (oops I have started calling myself a Rapid Tester ….) I can find a blog post here talking about hitting boundaries and sixes in Testing.
You will find some interesting debates and discussions about Pradeep’s post at BJ Rollison’s Blog here, here and here
This is an interesting discussion about linking boundaries in testing to that in cricket. Most of us while learning the ropes of test design – have dealt with “Boundary Testing” or “Boundary Value Analysis”. Learning about boundaries for a new tester is like learning ABC.
Let me dig around and present some interesting stuff on this topic.
Introduction and the context
BVA or BVT is considered as Testing/Test design technique in black box testing methodology. Black box testing takes an external perspective of the test object to derive test cases.
What is a boundary?
- A separation, natural or artificial, which marks the division of two contiguous properties.
- The line or plane indicating the limit or extent of something
When applied to Test design – I would say a boundary for a variable is a periphery or a temporary limit at which some noticeable change in the application behavior happens. Some of these boundaries are created are due to application intended behavior or documented specifications – business rules (say “for getting a driver license, the applicant must be at least 18 years old”). Some are “implicitly” created by program/code implementation – A value greater than 4294967295 can not be stored in an unsigned integer variable. At each of these boundaries – some change in the behavior happens - that is what we would like observe. Boundaries are special things – strange things happen near and beyond them.
Additionally note that –
- It helps (by the way of simplification) to apply the term “boundary” to parameters, variables, attributes, properties, fields of a given test problem. Using the term “boundary” any bigger context as in “boundaries for the program or application or software” – can complicate the task of the thinking and modeling the test space. I would like to call anything that can be manipulated during the usage of a software application as a variable.
- During boundary testing – each variable is taken one at a time – meaning effects of more than one variable’s boundaries are not considered. Combination testing (pairwise, orthogonal arrays etc) can help in considering multiple variables with 2 or more considered at a time.
- There can be multiple boundaries for a given parameter – few imposed by application functionality or business rules and few others by programmatic implementation via temporary and permanent storage and processing of the information/data.
- A given parameter being analyzed for boundaries can be further modeled to be consisting of several subsets each having their own boundaries or edges. Identifying these subsets for a parameter can help greatly in identifying multiple boundaries. By the way these subsets are referred as “Equivalence classes”. The technique is also referred as “partitioning”
- If we can identify “n” equivalence classes (each having at least 2 boundaries) then we have identified “2n” boundaries for a parameter.
Why boundary?
It is believed that “bugs lurk around boundaries” – developers tend to be “lazy” or “careless” around boundaries. Note that this is belief and at times can be wrong. Hence, testing or designing tests “focusing on boundaries” is a “low hanging fruit”. How many times you have seen a tester jumping to enter a “huge – Stttttttttttttttttttttttttttttting” in an edit box that is designed to receive a string input. It is really tempting to check “stuff” at boundaries.
Given the scale of emphasis on boundaries, sometimes I get a feeling that “developers” have become intelligent and tend to “take care” of these.
What is a boundary Value Testing and Analysis?
Wikipedia: Boundary value analysis is software testing related technique to determine test cases covering known areas of frequent problems at the boundaries of software component input ranges.
FOLDOC (Free online dictionary of computing, UK): A test data selection technique in which values are chosen to lie along data extremes. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a systems works correctly for these special values then it will work correctly for all values in between.
Examples of boundary value tests:
- Maximum negative, maximum positive, and 0 inputs or outputs;
- Input or output strings at size limits, 1 character beyond size limits, the empty string, and strings of 1 character;
- Input or output numeric values at size limits and 1 beyond size limits;
- Empty input files and files with one character in them;
- For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1);
- If input conditions specify a number of values n, test with (n-1), n and (n+1) input values;
Apply the above to output conditions (e.g., generate table of minimum and maximum size); - If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Where applicable?
A test object or a problem that can be modeled in terms of variables and domains. An approach to Domain Testing.
How boundary testing is done?
Here is how most of the boundary testing I have seen, happens –
- Identify the fields (variables – mostly input type) for the feature to be tested
- Refer to an available specification (raise alarm or shout if you don’t have one) or ask developer and get the boundaries for the fields identified above in step. Optionally raise an issue or bug on specification if it does not “clearly” states the boundaries.
- Derive a matrix containing the fields, corresponding boundaries and a set of test inputs at each of these boundaries.
- Declare that boundary testing has been done.
What is the problem here?
Let us analyze above process of performing BVA and BVT (boundary value testing). Above process is built upon several (reckless) assumptions that are often not “thought thorough” and “incorrect”. If your assumptions are false – your BVA/BVT looses its value and credibility.
- BVA/BVT is a universal Technique. If you ask any “upcoming” tester about “test design techniques” – invariably he will start and mostly end with “BVA”. Not knowing when BVA can not be applied can be “dangerous and costly”.
- Specification is a must for doing boundary analysis.
- Developer can/will tell you what is the boundary – Just go by what he/she says.
- All the variables can be modeled to have a linear scale with an upper bound and lower bound – there are 2 fixed boundaries per variable.
Boundary value Exploration -
Traditional approach to boundary testing – the whole process appears to be very simple – a notion of few fixed/well known and documented boundaries and testing with values around these.
Instead of BVA – I propose to coin this term BVE – Boundary Value Exploration (I just checked Google to make sure that someone has not already used this term).
Let me give you few thoughts around this new term as I am thinking about it. In next few posts I will cover the details with examples.
Instead of looking at test variables as numbers on a linear scale, in BVE, we would be exploring the variables for interesting behaviors – our objective would be discover as many as boundaries as possible and then study them. These boundaries would then be subjected to all sorts of “tortures” so that they can reveal some “interesting information”
While modeling and exploring boundaries for a parameter of a testing problem or a feature, I suggest to approach it from two angles. Please note that all these parameters are explicit (meaning those can be traced to specifications or product code). I would cover “implicit” parameters later (next few posts)
BVE – Outsider’s View:
Anything that is visible to an end user – typically that is/can be manipulated during the program usage is covered while viewing like an outsider. This is a typical black box approach. I use the term “outsider” to indicate that details like actual programming implementations, programming language, data types etc are either not available or test designer takes a deliberate view to temporarily ignore them.
While performing BVE as an outsider, you would consider things like business rules, constraints imposed by the application behavior (black box), specifications, error messages, application logs etc.
BVE – Insider’s View:
This is all about “code and related information related to implementation”. When performing BVE from insiders view – you would be looking into various details like programming languages, data structures, data types, use of relational operators, loops etc. You will use the information gathered during “BVE –Outsiders view” and link them to various data storages used for data processing. In process you might discover new boundaries, corresponding values.
Let me conclude this part of BVE – by saying
“There are boundaries to be explored (followed by analysis) with respect boundary testing – some of them appear to be fixed (until proved otherwise), some are not clearly understood, some are not known but can exist."
After all this, one thing is *nearly* sure “There are no fixed boundaries” and there is a great deal of suspense, and mystery around these boundaries, waiting to be solved.
Shrini
Updates :
There are two other interesting references to boundary testing
1. Boundary Testing by Mike D Kelly and James Bach
2. Testing with numbers by Karen N Johnson (thanks BJ for pointing out this one)
