Friday, March 09, 2007

Boundary Value Exploration - BVE

Happy New Year and here is my first blog post in this New Year. I can not believe that I did not blog for nearly 3 months. I am thankful to the readers of my blog for posting comments on my previous posts and really keeping me on my toes. I am privileged to have likes of Cem Kaner to comment on few blog posts.

This is cricketing time (World Cup at West Indies starting from March 11). Being one the avid cricket watcher and enthusiast – I will be eagerly hooked on to idiot Box for all matches ….

When we are talking about cricket, boundaries, sixes and all excitement of cricket – How can I forget – testing? Thanks to Pradeep, a fellow Rapid/Context Driven Tester (oops I have started calling myself a Rapid Tester ….) I can find a blog post here talking about hitting boundaries and sixes in Testing.

You will find some interesting debates and discussions about Pradeep’s post at BJ Rollison’s Blog here, here and here

This is an interesting discussion about linking boundaries in testing to that in cricket. Most of us while learning the ropes of test design – have dealt with “Boundary Testing” or “Boundary Value Analysis”. Learning about boundaries for a new tester is like learning ABC.

Let me dig around and present some interesting stuff on this topic.

Introduction and the context

BVA or BVT is considered as Testing/Test design technique in black box testing methodology. Black box testing takes an external perspective of the test object to derive test cases.

What is a boundary?
  • A separation, natural or artificial, which marks the division of two contiguous properties.
  • The line or plane indicating the limit or extent of something


When applied to Test design – I would say a boundary for a variable is a periphery or a temporary limit at which some noticeable change in the application behavior happens. Some of these boundaries are created are due to application intended behavior or documented specifications – business rules (say “for getting a driver license, the applicant must be at least 18 years old”). Some are “implicitly” created by program/code implementation – A value greater than 4294967295 can not be stored in an unsigned integer variable. At each of these boundaries – some change in the behavior happens - that is what we would like observe. Boundaries are special things – strange things happen near and beyond them.

Additionally note that –

  • It helps (by the way of simplification) to apply the term “boundary” to parameters, variables, attributes, properties, fields of a given test problem. Using the term “boundary” any bigger context as in “boundaries for the program or application or software” – can complicate the task of the thinking and modeling the test space. I would like to call anything that can be manipulated during the usage of a software application as a variable.

  • During boundary testing – each variable is taken one at a time – meaning effects of more than one variable’s boundaries are not considered. Combination testing (pairwise, orthogonal arrays etc) can help in considering multiple variables with 2 or more considered at a time.

  • There can be multiple boundaries for a given parameter – few imposed by application functionality or business rules and few others by programmatic implementation via temporary and permanent storage and processing of the information/data.

  • A given parameter being analyzed for boundaries can be further modeled to be consisting of several subsets each having their own boundaries or edges. Identifying these subsets for a parameter can help greatly in identifying multiple boundaries. By the way these subsets are referred as “Equivalence classes”. The technique is also referred as “partitioning”

  • If we can identify “n” equivalence classes (each having at least 2 boundaries) then we have identified “2n” boundaries for a parameter.

Why boundary?

It is believed that “bugs lurk around boundaries” – developers tend to be “lazy” or “careless” around boundaries. Note that this is belief and at times can be wrong. Hence, testing or designing tests “focusing on boundaries” is a “low hanging fruit”. How many times you have seen a tester jumping to enter a “huge – Stttttttttttttttttttttttttttttting” in an edit box that is designed to receive a string input. It is really tempting to check “stuff” at boundaries.
Given the scale of emphasis on boundaries, sometimes I get a feeling that “developers” have become intelligent and tend to “take care” of these.


What is a boundary Value Testing and Analysis?

Wikipedia: Boundary value analysis is software testing related technique to determine test cases covering known areas of frequent problems at the boundaries of software component input ranges.

FOLDOC (Free online dictionary of computing, UK): A test data selection technique in which values are chosen to lie along data extremes. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a systems works correctly for these special values then it will work correctly for all values in between.

Examples of boundary value tests:

  • Maximum negative, maximum positive, and 0 inputs or outputs;
  • Input or output strings at size limits, 1 character beyond size limits, the empty string, and strings of 1 character;
  • Input or output numeric values at size limits and 1 beyond size limits;
  • Empty input files and files with one character in them;
  • For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1);
  • If input conditions specify a number of values n, test with (n-1), n and (n+1) input values;
    Apply the above to output conditions (e.g., generate table of minimum and maximum size);
  • If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.

Where applicable?
A test object or a problem that can be modeled in terms of variables and domains. An approach to Domain Testing.


How boundary testing is done?

Here is how most of the boundary testing I have seen, happens –

  • Identify the fields (variables – mostly input type) for the feature to be tested
  • Refer to an available specification (raise alarm or shout if you don’t have one) or ask developer and get the boundaries for the fields identified above in step. Optionally raise an issue or bug on specification if it does not “clearly” states the boundaries.
  • Derive a matrix containing the fields, corresponding boundaries and a set of test inputs at each of these boundaries.
  • Declare that boundary testing has been done.


What is the problem here?

Let us analyze above process of performing BVA and BVT (boundary value testing). Above process is built upon several (reckless) assumptions that are often not “thought thorough” and “incorrect”. If your assumptions are false – your BVA/BVT looses its value and credibility.

  • BVA/BVT is a universal Technique. If you ask any “upcoming” tester about “test design techniques” – invariably he will start and mostly end with “BVA”. Not knowing when BVA can not be applied can be “dangerous and costly”.
  • Specification is a must for doing boundary analysis.
  • Developer can/will tell you what is the boundary – Just go by what he/she says.
  • All the variables can be modeled to have a linear scale with an upper bound and lower bound – there are 2 fixed boundaries per variable.

Boundary value Exploration -

Traditional approach to boundary testing – the whole process appears to be very simple – a notion of few fixed/well known and documented boundaries and testing with values around these.

Instead of BVA – I propose to coin this term BVE – Boundary Value Exploration (I just checked Google to make sure that someone has not already used this term).
Let me give you few thoughts around this new term as I am thinking about it. In next few posts I will cover the details with examples.

Instead of looking at test variables as numbers on a linear scale, in BVE, we would be exploring the variables for interesting behaviors – our objective would be discover as many as boundaries as possible and then study them. These boundaries would then be subjected to all sorts of “tortures” so that they can reveal some “interesting information”

While modeling and exploring boundaries for a parameter of a testing problem or a feature, I suggest to approach it from two angles. Please note that all these parameters are explicit (meaning those can be traced to specifications or product code). I would cover “implicit” parameters later (next few posts)


BVE – Outsider’s View:

Anything that is visible to an end user – typically that is/can be manipulated during the program usage is covered while viewing like an outsider. This is a typical black box approach. I use the term “outsider” to indicate that details like actual programming implementations, programming language, data types etc are either not available or test designer takes a deliberate view to temporarily ignore them.

While performing BVE as an outsider, you would consider things like business rules, constraints imposed by the application behavior (black box), specifications, error messages, application logs etc.


BVE – Insider’s View:
This is all about “code and related information related to implementation”. When performing BVE from insiders view – you would be looking into various details like programming languages, data structures, data types, use of relational operators, loops etc. You will use the information gathered during “BVE –Outsiders view” and link them to various data storages used for data processing. In process you might discover new boundaries, corresponding values.

Let me conclude this part of BVE – by saying

“There are boundaries to be explored (followed by analysis) with respect boundary testing – some of them appear to be fixed (until proved otherwise), some are not clearly understood, some are not known but can exist."

After all this, one thing is *nearly* sure “There are no fixed boundaries” and there is a great deal of suspense, and mystery around these boundaries, waiting to be solved.

Shrini

Updates :

There are two other interesting references to boundary testing

1. Boundary Testing by Mike D Kelly and James Bach

2. Testing with numbers by Karen N Johnson (thanks BJ for pointing out this one)

7 comments:

Bj Rollison said...

Hi Shrini,

This is an excellent summation. I especially like the section you labeled 'What is a boundary' and specifically the additional notes.

Boundary testing (as well as equivalence class partitioning, combinatorics, state transition testing, etc) are 'functional" testing techniques. I don't refer to them as "black box" test techniques because that is somewhat misleading.

In fact, boundary testing (as well as EQ partitioning, etc) can also be applied while designing tests from the white box test approach (using the traditional connotation of white box testing (e.g. implicit knowledge of the code (program language, intermediate, or assembly).

(I know there is a huge debate regarding black box vs. white box. IMHO, the debate is mostly philosophical and lacks academic or practical merit at this point for practioners of software testing.)

Anyway, this is one particular class of defects that can be prevented or detected earlier at the unit testing level in 99% of the cases. Personally, I think finding boundary class defects at the integration and system level of testing is an indication to me that developers lack the basic skills to design adequate unit tests. And, in an effort to push quality upstream and detect defects earlier I would probably spend some time with the developers and review their unit tests. But, that is a tangent.

Personally, I am not a big fan of giving things new names. The jargon we use in our profession is convoluted enough. I am not sure we need a new acronym to describe this technique. The ability of a person to "analyze' something is based on their skill, knowledge, and experience which is similar to exploratory testing.

However, that being said, there are many ways to approach the topic to help people better understand and apply the technique, and that is the ultimate goal you and I share.

- Bj -

James Bach said...

Hi Shrini,

I think it's important to create new terminology to reflect the emergence of new awareness of what we're doing.

I already have a way of describing what you call BVE. I call it exploratory boundary testing.

Exploration is quite important in boundary testing, because programmers have no solid idea what their boundaries actually are. No one can tell you that. They are just guessing.

Boundary testing depends on your model of the product, and that is not easy to get right. The reason testers exist is because programmers, in fact, often don't get it right.

Boundary testing is an example of the embarrassing lack of content in the traditional ways of discussing test techniques. As it is described by most authors, it strikes me as intellectually empty; a content-free idea. Another such content-free idea is equivalence class partitioning. So, I appreciated you trying to supply a little more substance to it.

-- James

Rajesh said...

Hi Shrini,
This is a good summarization of boundary value analysis. Boundaries can also be looked in a different way; Boundaries with proximity to the inputs and boundaries with proximity to the outputs. Consider a simple to that will help explain the concept. An application that takes two dates as input and outputs the difference between the two dates in number of days. The input boundaries are the obvious; dates with start of the month, end of month, start of year, end of year, months with 30 days, months with 28 days, months with 31 days, year 2000, today's date etc. The output boundaries are the again the values 30, 31, 28, 365.
From my experience, most of the testers focus on the input boundaries and not enough importance given to output boundaries. The output boundaries are unfortunately not defined clearly in most cases from the requirements or from the design documents. In order to find the output boundaries the best way will be to speak with your developers.
Regards
Rajesh

Shrini Kulkarni said...

Hi Rajesh,

Thanks for your points and emphasis on "output value boundaries". I somehow not inclined to "ask" the developer for the boundaries.

I would like to explore them - while keeping myself open on any information that will help me in my exploration. It is like finding your way in a jungle using whatever means that is available. I am sure tools like maps will become less useful ....

Mallikarjun Reddy said...

Hi Shrini,
My experience is that most of the times the bugs found are not simple boundary cases but many times its the bugs in combination testing. If developer is using integer variables like a and b, testers most of time will do the boundary analysis for this variables, but somewhere in the code developer may be using stuff like c=a+b; where c is another integer variable, but here there is case of integer overflow which can be missed out. But this also blows out the number of test cases with increasing variables. So Should'nt boundary value analysis be a job of code based tester ?

Shrini Kulkarni said...

Malik,

Traditional BVA deals with one variable at time and is extende other variables in problem domain. Key here is boundary investigation of one variable in isolation.

If you want to consider the effects of multiple variables and their boundaries - Combination testing, pair combinations tests are to be used. Tools/techniques likes "allpairs", "orthogonal arrays" are to be used.

Here are few links that could be of help

http://www.pairwise.org/
http://blogs.msdn.com/micahel/archive/2004/04/28/122702.aspx

http://www.developsense.com/testing/PairwiseTesting.html

http://www.burtleburtle.net/bob/math/jenny.html

http://en.wikipedia.org/wiki/All-pairs_testing

As far as using code based boundary testing --- I am of the opinion that a tester can use all available sources to to BVE, an active collaboration with developer can add great value to the process of BVE.

Modelling test space as input and output variables and using EQ classes can be of great help in BVE

I would not be comfortable in only developer running BVE with the knowledge of Code.

Shrini

NidHi said...

Hi,

Can anyone explain me that how boundary testing is different from equivalence partitioning?

Nidhi Verma