Test Design Workshop Module 4 - British - BCS · Test Design Workshop Module 4 Pairwise Testing Stephen K. Allott. ... Test No Phone Feature Location 1 Nokia Voice UK 2 Ericsson Text

www.electromind.com Copyright 2007 ElectroMind Ltd1

Test Design WorkshopModule 4 Pairwise Testing

Stephen K. Allott


Assumptions

You understand the reasons why we test softwareYou appreciate why it is impossible to test everythingYou’re familiar with a risk-based approach to testingYou are able to design logical & physical test casesYou know some black box test design techniquesYou’ve heard that using techniques is effective

You are a Professional Tester!


Potential test case explosion!

Money market application

2 types of deposit

5 terms

6 currencies

120 combinations

Web application

8 browsers

3 plug-ins

6 client OS

3 servers

3 server OS

1296 combinations

Online travel site

3 types of carrier

3 classes of travel

3 kinds of fare

3 types of ticket

81 combinations


Options

Give up, don’t testTest all combinations and delay the projectSelect a few tests and hopeRe-run tests that have been run beforeSelect tests that are easy to create & runList all the combinations and run the first fewList all the combinations and select a random subsetBy magic, choose a special, small, subset that finds more defects than you might expect

Adapted from “A practitioner’s guide to software testing, Lee Copeland”


Pairwise Testing Definition

Pairwise testing techniques reduce the number of test cases by testing all pairs of variables rather than attempting to test all of the combinations for all the values for all the variables


Pairwise Testing Theory

Many faults are eitherSingle modeDouble mode

Some metrics to support the theory existAnecdotal evidenceCorporate programmes

There are two recognised methods:Orthogonal arraysAllpairs algorithm


Simple orthogonal array L4(23)

A B C

Row 1 1 1 1

Row 2 1 2 2

Row 3 2 1 2

Row 4 2 2 1

Given the numbers 1 and 2 how many pairs exist?

1,1

1,2

2,1

2,2

This orthogonal array has 4 rows, 3 columns and two values per column


Simple orthogonal array L4(23)

A B C

Row 1 1 1 1

Row 2 1 2 2

Row 3 2 1 2

Row 4 2 2 1

Given the numbers 1 and 2 how many pairs exist?

1,1

1,2

2,1

2,2

This orthogonal array has 4 rows, 3 columns and two values per column


Orthogonal Array L9 (34)

A B C DRow 1 1 1 1 1Row 2 1 2 2 2Row 3 1 3 3 3Row 4 2 1 2 3Row 5 2 2 3 1Row 6 2 3 1 2Row 7 3 1 3 2Row 8 3 2 1 3Row 9 3 3 2 1

This orthogonal array has 9 rows, 4 columns and three values per column










On-line travel site

Airline typeScheduled, Low-cost, Charter

Cabin First Class, Business, Economy

Fare typeOne-way, Return, Stopover

Fare conditionsFlexible, Restricted, No changes allowed

3 X 3 X 3 X 3 = 81 combinations



Airline Cabin Fare CondRow 1 1 1 1 1Row 2 1 2 2 2Row 3 1 3 3 3Row 4 2 1 2 3Row 5 2 2 3 1Row 6 2 3 1 2Row 7 3 1 3 2Row 8 3 2 1 3Row 9 3 3 2 1Each variable becomes a column in the orthogonal array



Airline Cabin Fare CondRow 1 Scheduled 1 1 1Row 2 Scheduled 2 2 2Row 3 Scheduled 3 3 3Row 4 Low-cost 1 2 3Row 5 Low-cost 2 3 1Row 6 Low-cost 3 1 2Row 7 Charter 1 3 2Row 8 Charter 2 1 3Row 9 Charter 3 2 1

For each column we fill out the possible choices



Airline Cabin Fare CondRow 1 Scheduled First 1 1Row 2 Scheduled Business 2 2Row 3 Scheduled Economy 3 3Row 4 Low-cost First 2 3Row 5 Low-cost Business 3 1Row 6 Low-cost Economy 1 2Row 7 Charter First 3 2Row 8 Charter Business 1 3Row 9 Charter Economy 2 1

Now repeat for the second column



Airline Cabin Fare CondRow 1 Scheduled First One-way 1Row 2 Scheduled Business Return 2Row 3 Scheduled Economy Stopover 3Row 4 Low-cost First Return 3Row 5 Low-cost Business Stopover 1Row 6 Low-cost Economy One-way 2Row 7 Charter First Stopover 2Row 8 Charter Business One-way 3Row 9 Charter Economy Return 1

Now fill out the third column



Airline Cabin Fare CondRow 1 Scheduled First One-way FlexibleRow 2 Scheduled Business Return RestrictedRow 3 Scheduled Economy Stopover No changesRow 4 Low-cost First Return No changesRow 5 Low-cost Business Stopover FlexibleRow 6 Low-cost Economy One-way RestrictedRow 7 Charter First Stopover RestrictedRow 8 Charter Business One-way No changesRow 9 Charter Economy Return Flexible

Now the array is complete



Airline Cabin Fare CondTest 1 Scheduled First One-way FlexibleTest 2 Scheduled Business Return RestrictedTest 3 Scheduled Economy Stopover No changesTest 4 Low-cost First Return No changesTest 5 Low-cost Business Stopover FlexibleTest 6 Low-cost Economy One-way RestrictedTest 7 Charter First Stopover RestrictedTest 8 Charter Business One-way No changesTest 9 Charter Economy Return Flexible

Each row becomes a test case – you can add more if you wish


Summary of Orthogonal Arrays

Identify the variables (columns)Determine the number of choices for each columnLocate an orthogonal array of the right sizeMap the testing problem onto the arrayFill in any blank valuesConstruct the test casesAdd more tests if you wish to cover specifically known risky business scenarios


What size array do we need?

Variable Choices

Browsers IE5, IE6, Firefox, Opera, Mac, Safari, A,B

Plug-ins None, Real Player, Media Player

Operating systems W95, W98, W2000, NT, XP, XP-PRO

Server IIS, Apache, WebLogic

Server O/S NT, XP, Linux

How many possible combinations are there?

What do you think is the likely reduction using Pairwise?


Using Orthogonal Arrays

We require an array with 5 columnsColumn 1 needs 8 levels (browsers)Column 2 would need 3 levels (plug ins)Column 3 would need 6 levels (O/S)Columns 4 & 5 would need 3 levels (Server, Server O/S)

Ideally we need (81 61 33) but it doesn’t exist!The next largest is L64 (82 43)

2 columns with 8 choices 3 columns with 4 choices

So rather than 1,296 tests we need only 64 tests!


Allpairs Algorithm

Generates fewer tests In our example, it would generate just 48 tests compared to 64 from using orthogonal arrays

To see how to construct the All Pairs table consider three variables that take 3,2 and 2 values respectively

For example, three mobile phones that have two features (voice & text) to be tested both in the UK and overseas, 12 combinations in total . . .


Test No Phone Feature Location1 Nokia Voice UK2 Ericsson Text UK3 Motorola Voice UK4 Nokia Text UK5 Ericsson Voice UK6 Motorola Text UK7 Nokia Voice Overseas8 Ericsson Text Overseas9 Motorola Voice Overseas10 Nokia Text Overseas11 Ericsson Voice Overseas12 Motorola Text Overseas

All combinations


Step 1

Phone Feature Location

Nokia

Nokia

Ericsson

Ericsson

Motorola

Motorola

Variables as columns

Descending order

Leave blank rows


Step 2


Nokia Voice

Nokia Text

Ericsson Voice

Ericsson Text

Motorola Voice

Motorola Text

Variables as columns

Descending order

Leave blank rows

Fill out 2nd column


Step 3


Nokia Voice UK

Nokia Text Overseas

Ericsson Voice Overseas

Ericsson Text UK

Motorola Voice UK

Motorola Text Overseas

Add third variable

Pair with 2nd column

Section 3 is arbitrary

6 tests instead of 12!


Now add a 4th variable

Phone Feature Location Tarrif

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK





Nokia Voice UK 1

2

2

1

2

1

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


1. Pair columns 4 & 2




Nokia Voice UK 1

2

2

1

2

1

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


1. Pair columns 4 & 2

2. Next pair with 3

3. Check 1 & 4 – OK



With a 5th variable it gets harder

Phone Feature Location Tarrif Time

1

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK





1 AM

PM

PM

AM

PM

AM

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


It works for columns 1,2,3




1 AM

PM

PM

AM

PM

AM

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


It works for columns 1,2,3 but not for 4 !




1 AM

PM

AM

PM

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


Swap middle section for column 5

Erase the rest

Think again!


Solution for 5 variables


1 AM

PM

AM

PM

PM

AM

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK


Final section turns out to be the same



A 6th variable will not fit!

Phone Feature Location Tarrif Time Variable 6

AM

PM

AM

PM

PM

AM

1

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK





AM I

J

J

I

J

I

PM

AM

PM

PM

AM

1

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK





AM I

J

J

I

J

I

PM

AM

PM

PM

AM

1

2

2

1

2

1

Nokia Voice UK

Nokia Text Overseas


Ericsson Text UK

Motorola Voice UK



Needs 8 tests rather than 96!


AM I

J

J

I

J

I

J

I

PM

AM

AM

PM

PM

PM

AM

1

2

2

2

1

1

2

1

Nokia Voice UK

Nokia Text Overseas

Nokia Text Overseas


Ericsson Text UK

Ericsson Text UK

Motorola Voice UK



Using Allpairs

Download allpairs from www.satisfice.comCreate a tab separated input fileFirst line contains the variable namesAt command line enter:

Allpairs input.txt > output.txt


PICT

Implements allpairs algorithmAllows you to specify constraintsDownload from www.pairwise.orgCreate TXT file, one line per variableAt command line enter:

PICT input.txt > output.xls


Pairwise Testing – Summary

Tests all pairs rather than all combinationsFor example with 4 variables and 3 choices for each you would require 81 test cases whereas you can cover all Pairwise input combinations with only 9 tests!

Pairwise guides us to help findSingle mode defectsDouble mode defects

However, If there is a risk that three values (triples) in combination might cause a failure, include that combination in your tests


Unbelieveable!For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all Pairwise input combinations with only 15 testsFor 10 variables having 26 values each, all combinations leads to 141,167,095,653,376 test cases. This can be reduced to a more manageable 1094 using Pairwise.Consider the case of 10 parameters with 10 values each. All pairs will do it in 177 cases. The smallest number of test casespossible is somewhere between 100 and 177, perhaps 130. But compared to the alternative of 10 billion test cases to achieve all permutations, 177 is not too bad.Consider 100 variables, 2 choices

Using orthogonal arrays you need 101 testsUsing All pairs you need only 10 tests


Other Useful References

Allpairs algorithm, James Bach www.satisfice.comCohen et al 1997, The AETG system, an approach to testing based on combinatorial design www.argreenhouse.com/papers/gcp/AETGieee97.shtmlN J A Slone maintains a library of over 200 Orthogonal Arrays see www.research.att.com/~njas/oadir/index.htmlTelcordia Technologies AETG web solution for test case generation from requirements www.argreenhouse.comCaseMaker is a tool for test case design and test data generation using several techniques including Boundary values, Equivalence Classes and All pairs www.casemaker.de


Acknowledgements

With thanks and grateful acknowledgement to Lee Copeland and James Bach whose inspiring books have helped many testers gain a deeper and better understanding of software testing

A Practitioner’s Guide to Software Test Design, Lee Copeland, Artech House, 2004, ISBN 1-58053-791-XLessons Learned in Software Testing, Kaner, Bach, Pettichord, Wiley, 2002, ISBN 0-471-08112-4


Test Design WorkshopStephen K. Allott

[email protected]

Documents

Test Design Workshop Module 4 - British - BCS · Test Design Workshop Module 4 Pairwise Testing Stephen K. Allott. ... Test No Phone Feature Location 1 Nokia Voice UK 2 Ericsson Text