Practical Pairwise Testing with PICT · Specify important combinations ‘Happy path’ BC tests Seeded input ensures important combinations are tested Tahoma seed not used because

T19 Concurrent Session Thursday 05/08/2008 3:00 PM

Practical Pairwise Testing with PICT

Presented by:

BJ Rollison Microsoft

Presented at: STAREAST Software Testing Analysis & Review

May 5-9, 2008; Orlando, FL, USA

330 Corporate Way, Suite 300, Orange Park, FL 32043 888-268-8770 904-278-0524 [email protected] www.sqe.com

BJ Rollison Bj Rollison is a Test Architect with Microsoft’s Engineering Excellence group where he designs and develops technical training curriculum, and teaches various testing techniques and approaches to testers and developers at Microsoft. Bj started his professional career in computers building custom solutions for small and medium sized businesses for an OEM company in Japan in 1991. In 1994, he joined Microsoft and worked on the Windows 95 team and on the Internet Explorer team. He became the Director of Testing in 1999 responsible for planning and organizing training for more than 6000 Microsoft testers. In 2003 Microsoft began to refocus its software processes on engineering practices and Bj decided to move to a Test Architect role and share his passion for teaching and mentoring. Bj also teaches software testing courses at the University of Washington, and sits on software testing advisory boards at the University of Washington, the University of California Extension Santa Cruz, and Lake Washington Technical College. He is also a frequent speaker at international software testing conferences.

Bj Rollison Test Architect

Microsoft

http://www.TestingMentor.comhttp://blogs.msdn.com/imtesty

Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers

Unleash Their Potential!

“There are more tests in a program with 70 branches then there are teaspoons of water in the Pacific Ocean”

– William Hetzel



Design an extremely small subset of tests from all possibilitiesQualify the important attributes and capabilities of the application under testIdentify defects and other potential risksProvide accurate information to the decision makers for informed risk-assessmentWithin a limited amount of timeThat provides a high degree of confidence for the entire team



Functional testing technique in which the tester:methodically analyzes parameter interaction in a complex feature setin order to systematically select a finite subset of tests from all possible combinations

Uses a mathematical approach based on historical failure indicators and empirical dataHelps reduce overall risk by

High defect detection effectivenessHigh levels of structural coverage



Combinatorial testing is useful when testing a feature in which:

Parameters are directly interdependent

Parameters are semi-coupled

Parameter input is unordered



Independent parameters

No direct orindirect interaction

Mathematical formulasVarParam1 + VarParam2 = VarParamOut

Sequential operationsOrdered input



Random evaluation approachesBest guess

Intuition and luckRandom selection

Randomly selected from all combinationsSystematic evaluation approaches

Each choice (EC)Base choice (BC)Orthogonal arrays (OA)Combinatorial analysis (t=n)Exhaustive (AC)



Finding OA is non-trivialUnrealistic expectationsequal number of variables per parameterRedundant tuples in output

Test B I U S Col.5 Col.6 Col.7

1 unck unck unck unck 1 1 1

2 unck unck unck chk 2 2 2

3 unck chk chk unck 1 2 2

4 unck chk chk chk 2 1 1

5 chk unck chk unck 2 1 2

6 chk unck chk chk 1 2 1

7 chk chk unck unck 2 2 1

8 chk chk unck chk 1 1 2

L8 Orthogonal Array

Taguchi Array Selectorhttp://www.freequality.org



Identify one base test (usu. ‘happy path’)Iterate through the variable values for one parameter at a time, while keeping the other parameters constant with base testSatisfies 1-wise coverageSingle error coverage

Test 1 2 3 4 5

Bold No Yes No No No

Italic No No Yes No No

Underline No No No Yes No

Strikethrough No No No No Yes



Basic algorithmList all possible pairsUse greedy search heuristicGenerate one test at a timeInsert as many uncovered pair combinations as possible ineach new test until all t=n covered

Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Bold yes yes yes yes no no no yes no no yes yes no yes no no

Italic no yes yes yes yes yes yes yes no no no no no no yes no

Underline no no yes yes no yes yes no yes yes yes yes no no no no

Strikethrough no no no yes no no yes yes no yes no yes yes yes yes no



Analysis & featuredecomposition

Model parametervariables

PICTTool

ValidateOutput

Modify model

Test

Re-validateOutput

REMEMBER: The output of the tool isbased on the tester’s input...it’s only a tool!

Base ChoiceMatrix

up to t=6

Seeded Input



Interdependent parametersFontsBoldItalic

Semi-coupled parametersColor (Black, white, red, green, blue, yellow)

Size (1 – 1638, including half sizes)

StrikethroughUnderline

Total number of tests = Cartesian product of variables = 1,920 tests



File format (for PICT)param: var1, var2, var3

Identify mutually exclusive variable combinations

Analyze output for “happy path”

Analyze outputfor failure indicators

Copyright © 2008 Bj Rollison. All rights reserved.


Conditional relationships

Exclude mutually exclusive variable states;

often increases total number of tests


Aliasing or Variable equivalence

Equivalence variables which are similar;

may sometimes reduce total number of tests


Weighting variables,Variable preference Give increased weighted

value to important variables; MAY increase

probability of occurrence


Seeded outputSpecify importantcombinations‘Happy path’BC tests

Seeded input ensures important combinations

are tested

Tahoma seed not used because Tahoma is equivalent to Arial


Randomize output to increase breadth of

testing and minimize probability of error due to

missed combinations

PICT/r:nswitch



Invariant conditionsParameters must be different

Example: [Bold] <> [Italic] OR [Underline] <> [Strikethrough]Example: [Strike] <> [DblStrike] AND [SuperScript] <> [SubScript]

Parameter types2 types: Numbers and Strings

Example: Size: 1, 2, 3, 4, 5 and Value: a, b, c, d, eIF [Size] >= 3 THEN Value > “b”;

N-wise output (t = 6 @ effective as exhaustive)Negative testingSub-models



Greatest number (> 50%) of combinatorial defects are simple pair interactions

Randomizing pair-wise combinations with PICT may improve t=2 detection effectiveness*

Subtle defects still being discovered at t=6Effects are cumulative

Number oftests increaseexponentially, but don’t stop atpair-wise (t=2) 0%

20%40%60%80%

100%

t=2 t=3 t=4 t=5 t=6



Pair-wise found 98 percent of all detectable faults seeded in five applications

All faults in four applications

Evaluation of Combination Strategies for Test Case Selection (Grindal, et. al.)

Pairwise



“In this study we found no significant difference in the FDE of n-way and random combinatorial test suites. …the result is not unexpected.”Tests selectedrandomly fromall possiblecombinations# of tests equal to n-way # of tests

(Patrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopu)



" … a set of 29 pairwise tests gave 90% block coverage for the UNIX sort command. We also compared pairwise testing with random input testing and found that pairwise testing gave better coverage.”

The AETG System: An Approach to Testing Based on Combinatorial Design(Cohen, et. al.)



Attrib.exe – path + 6 optional argv.

Findstr.exe – str + 19 optional argv. (421,200 EC tests)

Total # of blocks = 483 “Default” test suite Exhaustive coverage Pair-wise coverage

Number of test cases 9 972 13

Blocks covered 358 370 370

Code coverage 74% 77% 77%

Functions not covered 15 15 15

Total # of blocks = 1317 Hand-crafted tests N=2 coverage N=3 coverage N=4 coverage

Number of test cases 236 136 800 3533

Blocks covered 960 979 994 1006

Code coverage 73% 74% 75% 76%

Functions not covered 11 11 10 10



Regression test suite time reduced(104 man-days to 32 hours)Test design and implementation reduced (44 man-months to 4 man-months)Reduced OS configuration testing 50% with zero bug bounce (ZBB)Increase in automated test development adopting data-driven output of PICT (@ 30 tests/week to 235 tests/week)3 ‘show-stopper’ defects detected prior to releasesaved @ $750,000 (each QFE @ $125K)

















Misuse of technique & toolTraining, knowledge, no silver bullets!

Right values not testedIn-depth system knowledge, stochastic data

Highly probable values get too little attentionWeighted values, include seeded input

Complex interactions are missedPair-wise is just the beginning, up to t=6

May produce a defect and not recognize itCommon to all testing, need better oracles

May be less effective if # of parameters is small*



Best Practice compared to other approaches for testing complex parameter interactions

Applied correctly (BC & t=2 t=6, analysis)Provides critical information earlier

High probability of early defect detectionHigh structural coverage; t=6 ≈ exhaustive

Easy to integrate into automation (data-driven)But,

Requires in-depth technical knowledgeMust verify output – it’s only a tool!Don’t forget Beizer’s Pesticide Paradox!


Helping Testers Unleash Their Potential!TM

http://[email protected]



http://www.pairwise.orgPairwise Testing in the Real World: Practical Extensions to Test-Case ScenariosJacek Czerwonka(http://msdn2.microsoft.com/en-us/testing/cc150619.aspx?topic=306121)

Comparing the Fault Detection Effectiveness of N-way and Random Test SuitesPatrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopuhttp://portal.acm.org/citation.cfm?doid=1159733.1159742

Software and Hardware Testing Using Combinatorial Covering SuitesAlan Hartman, IBM Haifa Research Laboratoryhttp://www.haifa.il.ibm.com/projects/verification/mdt/papers/CombinatorialProblemsinSWTesting-finalDraft180703.pdf



Pseudo-Exhaustive Testing for SoftwareD. Richard Kuhn and Vadim Okun(http://csrc.nist.gov/acts/PID258305.pdf)

Failure Modes in Medical Device SoftwareDolores R. Wallace and D. Richard Kuhn (http://csrc.nist.gov/staff/Kuhn/final-rqse.pdf)

The Combinatorial Design Approach to Automatic Test GenerationD. M. Cohen, S. R. Dalal, J. Parelius, G. C. Patton(http://www.argreenhouse.com/papers/gcp/AETGissre96.shtml)

An Evaluation of Combination Strategies for Test Case Selection M. Grindal, B. Lindström, J. Offutt, and S.F. Andler(http://www.his.se/upload/19352/EvalCombStratTechRepFinal.pdf)

Documents

Practical Pairwise Testing with PICT · Specify important combinations ‘Happy path’ BC tests Seeded input ensures important combinations are tested Tahoma seed not used because