All Pairs Testing TechniqueBy

Usha Sunder30-Mar-2009

Introduction "Mater artium necessitas" means "Necessity is the mother of invention" in

latin. In software industry, everyone is trying hard to reduce the burden of work and created many application & automat ion tools. The created application were tested by a team of testers. Those were released after gone under process of quality test.

Everyone is trying to create and test the error free application because of the fallibility of its human designers and its own abstract, complex nature, software development must be accompanied by quality assurance activities. It is not unusual for developers to spend 40% of the total project time on testing. For life -critical software (e .g. flight control, reactor monitoring), testing can cost 3 to 5 t imes as much as all other activities combined. The destructive nature of testing requires that the developer discard preconceived notions of the correctness of his/her developed.

Software Testing techniques (Manual Testing) The importance of software testing and its impact on software cannot be

underestimated.

The greater visibility of software systems and the cost associated with software failure are motivating factors for planning, through testing. It is not uncommon for a software organization to spent 40% of its effort on testing.

Software testing methods are majorly div ided into 5 types.

Black box testing

White box testing

Grey box testing

Non functional testing

Regression testing

Cont,… Exhaustive Testing:

Software testing techniques applied in Black box testing are

Equivalent partit ioning techniques

Boundary value analysis

Cause Effect Graphing techniques

Comparison Testing

Software Test Engineers applying these techniques and trying hard to reduce the no. of test cases for their application. Though the technique reduce the test cases considerable, we are not sure about the coverage of the application.

Testers Critical Situation If you are a tester , you would have found yourself in one of the following

situation

1. After a week release of the software, customer is filing a bug and it seems to be a serious bug. Though you tested the application thoroughly, you don’t know where you gone wrong?

2. If you are in a client meeting and the customer is asking you how to test a product with in a short duration of time, ?????

3. If your customer doesn’t know about the testing and asking the team members to do complete testing (ie. 100% coverage) what would be your reaction?????

Common Manual Testing Issues 1. Time Consuming & Cost driven

2. More no. of Resources are required for testing

3. Multiple times of testing the same set or subset of test cases in a different phase.

4. Tester does not come to a conclusion that they cover the application completely or not. (ie) Not sure about the coverage of the application.

Measuring the Test Plan Coverage - An Approach

Sources of defects are usually due to unexpected interactions between parameters.

Testing of all possible combinations is infeasible due to budget and timeConstraints

One approach is to test all two-way interactions between parameters.

If a test plan covers all the two way interactions between parameters, then we arebalancing of completing the testing within reasonable budget vs. risk of defects

due to interactions among three or more parameters

So a good way of measuring a test plan coverage is to find how much percentage of pair wise combinations are covered in the test plan

“Test Everything without really Testing” There are few proven technique available in the market, they are referred

as "Pairwise testing", "combinatorial Method" and "Orthogonal Arrays"(Taguchi Method)(actually, each of these is similar but different). Generally we call these technique as "ALL-PAIRS“.

Let’s start with an example and analyze the “All Pairs” advantages.

Example 1:

Assume that we have a simple software application called Product-A. A one-screen GUI application with a listbox, checkbox and OK button.

Product A A list box contains the 4 values and checkbox is either check or uncheck.

Exhaustive Combination –Product A If we are performing combination for this testcases, we will end up in

4x2=8 (Cartesian product) combination of testcases. i.e

Testing the Product-A doesn’t require much time for our tester. Let’s add little complexity to the Product-A application and name it as Product -B .

Example 2: Product -B Our Product-B application is having

1. Listbox (9 values)

2. Checkbox

3. Rad io button

4. Text box.

Constraint

Text box: +ve values between 1-100

Values present in the objects Listboxcontains 0,1,2,3….. 9

Checkbox is either check or uncheck

Radio button is either on or off

Textbox constraint is, it should take no’s ranges between 1-100.

Exhaustive Combination for Product B Exhaustive combination of the Product-B is,

Listbox =10

checkbox=2

Radio=2

Textbox=100(not taking negative conditions)

10x2x2x100=4000 (Cartesian product)

No. of testcases =4000 (not considering negative condition)

Exhaustive Combination Including negative > 4000 testcases

Applying STT for Product B

list box is having 0-9 has the value. Zero is a unique value (neither +venor –ve) and rest of the values we treated as Others.

Zero and Others (1-9) are the reduced values of the listbox We cannot reduce the values of Checkbox and also radio button so we

treat the values asit is in our column. Textbox -The constraint of the textbox is , it can take the no’s

ranges between 1-100. So we will reduce the values asValid Integer , Invalid Integer , Alpha-Special Char

*** STT-Software testing technique

Can we reduce? Exhaustive Combination Including negative > 4000 testcases

Software Technique Product-B = 2x2x2x3=24 testcases(includes negative conditions)

Can we reduce the combination fu rther down?

Yes we can reduce it. Let us use “All-Pairs” techniques ("Orthogonal Arrays"(Taguchi

Method),”Pairwise testing" and "combinatorial Method") and will see the result

All Pairs Technique to the Product -B

Order your variables so that the one with the most number of values is the first and the least is as last..

Step 1:

Step 2 : Now let's start filling in the table. Each row of the table will represent a

unique testcase/scenario. We will fill the table column by column. Look at how many values there are in column 2. Here, we see that the Listboxcolumn has 2 values. That’s how many times you will need to insert the values of the first column, Textbox.

Step 3: Column three is checkbox. The user either check or uncheck the

checkbox. Now let's start pairing the column-3(checkbox) with the column-1(textbox). You attained the following combination.

Step 4: Let’s make sure we have all our pairs between column-3(checkbox) and column-2(listbox). We have a 0 and Check, but wait – there’s no 0 and

Uncheck. We have an Other and Uncheck, but there’s no Other and Check. Let’s swap around the values in the second set in the third column. Now the combination will be like the following table:

Step 5: Now check whether every value is interacted with every other value in the

combination.Now we will follow the same procedure and start doing the combination for the fourth column (radio).

Step 6: Let's cross verify whether the all the values are paired with eachother.

When we verified with the column-1(textbox) and column-4(radio) , the combination is proper but when we compared with the column-2(listbox) and column-4(rad io) we again started getting the issue which we faced for column2(listbox) and column-3(checkbox).0 and on is there but no O and off similarly Others and Off is present but Others and On is missing.

Now we will do the similar kind of swapping for the 3rd set of 4th column.

Analyzing Report

Exhaustive Combination (including negative condition)>4000

Applying Software Technique , the no of testcases has been reduced to 24

All pairs technique reduced the testcases to 6 testcases. We fit every pair of values into six cases

Example 3: Product -CLet’s say Version 3.0 of our multiplier

adds two more checkboxes. checkbox2

will give the factorial value of the output,

and checkbox3 will convert the output into Hexadecimal notation. So we have to

add two more columns to our table andenter our values.

Follow the same procedure for

1-4 co lumn in Product-B. The next step is start doing pairing

with the checkbox2(5th column).

Step 8:

Let’s make sure each column has at least one pair with our newly add ed fifth column: Column 2 is OK: (0/Yes 0/No, Other/Yes, Other/No),Column 3 is OK: (On/Yes, On/No, Off/Yes, Off/No),and Column 4 is OK: (Checked/Yes, Checked/No, Unchecked/Yes, Unchecked/No). Notice t hat the on off sequence in the last set in the third column is no lo nger arbitrary as it was when we hadonly three columns filled in. We need it in that order now for our new value pairs.

Step 9:

Let’s again make sure each column has at least one pair with our newly added column: Column 2 is OK. We have a 0/Dec 0/Hex Other/Dec Other/Hex. However, column 3 is problemat ic: We do have an On/Dec and Off/Hex, but we’re missing On/Hex and Off/Dec pairs. Column 4 is OK: Checked/Dec, Checked/Hex, Unchecked/Dec, Unchecked/Hex. Column 5 is OK: We have a Yes/Dec, Yes/Hex, No/Dec, and No/Hex. This time, we can’t fit in our missing pairs (On/Hex and Off/Dec) by simply swapping around values. If we did, then other pairs would get out of whack. Instead, we simply add two more testcases that contain these pairs. Hence, the blank rows.

Pairing Explanation Let’s again make sure each column has at least one pair with our newly

added column: Column 2 is OK. We have a 0/Dec 0/Hex Other/Dec Other/Hex. However, column 3 is problemat ic: We dohave an On/Dec and Off/Hex, but we’re missing On/Hex and Off/Dec pairs. Co lumn 4 is OK: Checked/Dec, Checked/Hex, Unchecked/Dec, Unchecked/Hex. Column 5 is OK: We have a

Yes/Dec, Yes/Hex, No/Dec, and No/Hex.

Cont… This time, we can’t fit in our missing pairs (On/Hex and Off/Dec) by

simply swapping around values. If we did, then other pairs would get out of whack. Instead, we simply add two more testcases that contain these pairs. Hence, the blank rows.

Cont… The other variable values are purely arbitrary. They need to be filled in

with some value, but we don’t care which value, because we already have all our pairs. Go ahead and fill in the empty cells as you desire, and there you have it – all combinatorial pairs in eight cases instead of all

combinations in 96!

Additional Considerations: Despite its ability to dramatically reduce the number of test cases needed to

achieve what may be an appropriate level o f test coverage, combinatorial testing will not magically fix all your testing problems. You still need to think about how and when using a combinatorial test technique isappropriate. it may not be an appropriate method for all situations (mission critical and safety critical applications just to name a few).

Not a silver bulletin, more work still to do. Apply Input-Output (IO) Analysis to maintain fault-detection

Don’t be tempted to find all-pairs of all variab les, just because you can. You won’t find all your bugs by using this technique exclusively.

Don’t limit use of combinatorial test design to input variables

1. Not a silver bulletinThe combinatorial design technique provides a systematic

approach to running a subset of combinations, since you can’t run all combinations. Many times even this approach yields too many tests to execute in the allotted time. Testers may try to identify “important”combinations to test based on requirements and design specifications, historical fault data, risk analysis (e.g., generic risks lists, etc), and engineering judgment. These techniques could further reduce the number of tests; however, it remains very difficult to determine the impact on the fault-detection capability of the test set using these techniques and it is just as likely that these techniques actually increase the number of tests.

Typically the combinatorial method requires you to build a “test data model.” If the reduced test data model matches the software implementation, the fault-detection capability of the test set will likely be very good. If it doesn’t match you may have problems. In a nutshell first apply the combinatorial test design technique described in this article (select test data values for each input variable then consider how to test combinations of these program inputs).

Apply Input-Output (IO) Analysis to maintain fault-detection

The combinatorial technique does not consider program output variables. For a class of programs, many times not all program inputs affect all program outputs. You might want to consider using Input-Output (IO) Analysis to determine which program inputs affect program outputs. This information can then be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set.

Under this approach, we generate combinatorial tests from the perspective of the program outputs. For each output, we generate tests using only the inputs which influence it. For input which does not affect the output, we can use any value from their set of test data values. There is much more to it, but this is a topic for a different day.

Don’t be tempted to find all-pairs of all variables, just because you can.

Somet imes, a particu lar variable will o r will not exist, depending on the value of another variable (i.e . mutually exclusive conditions). For example, let’s say the same system that processes mortgages will also handle two other home finance products: home equity loans, and home equity lines of credit. There might be different business rules for processing line-of-cred it applications. Maybe the different products are not offered in the same group of Regions, or perhaps there are different categories for LTV among the different home finance products.

It wouldn’t make sense to add a “product” variable with values “Standard Mortgage”, “Home Equity Loan” and “Line of Cred it” into our combinations matrix because some of the resulting pairs (and therefore, test cases) would be undefined.

Cont….

If you did, you might generate a test case that pairs a line-of-credit application with values that are not available for that product. This is different than negative testing, where you would be trying an option that the system doesn’t expect. Here, you couldn’t try that option at all because it doesn’t exist. You would be creating buggy test cases.

One solution is to create separate matrices for the individual products. In this example, you can create individual matrices because the business rules are so different for each Home Equity product.

You won’t find all your bugs by using this technique exclusively.

Testing combinatorial pairs will only test to see whether any two variables conflict, not if three or more conflict. Also, if you forget to include a variab le, or you decide to exclude one from the matrix, its values won’t be meshed with the rest of the variables. Lastly, reducing

the number of values per variab le (as we did with the Textbox in the first example) could cause an important pair to be missed.

Don’t limit use of combinatorial test design to input variables

Throughout this presentation, we talked about variables as input to a system. Remember, variab les can also mean test environments, paths through a system, and outputs. A common usage of the combinatorial technique with non-input variables is in setting up test environments for Internet applications. Often, web apps need to be tested on a host of OS/Browser combinations.

ConclusionIn Mathematical theory, the combinatorial test design technique is a

thoughtful method when test planning. There are tools available in the market which will generate the combination fo r the user. One of the great tool is “All Pairs”. Download the all-pairs calculator and try it out. Using it, you can quickly generate test cases that have a good chance of finding bugs, instead of mindlessly copying and pasting text into test case templates. Also remember there is still more work to do to reduce your

test set down to its optimal number of test cases while still ma intaining its fault-detection capability.