View
217
Download
0
Category
Preview:
Citation preview
T19 Concurrent Session Thursday 05/08/2008 3:00 PM
Practical Pairwise Testing with PICT
Presented by:
BJ Rollison Microsoft
Presented at: STAREAST Software Testing Analysis & Review
May 5-9, 2008; Orlando, FL, USA
330 Corporate Way, Suite 300, Orange Park, FL 32043 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com
BJ Rollison Bj Rollison is a Test Architect with Microsoft’s Engineering Excellence group where he designs and develops technical training curriculum, and teaches various testing techniques and approaches to testers and developers at Microsoft. Bj started his professional career in computers building custom solutions for small and medium sized businesses for an OEM company in Japan in 1991. In 1994, he joined Microsoft and worked on the Windows 95 team and on the Internet Explorer team. He became the Director of Testing in 1999 responsible for planning and organizing training for more than 6000 Microsoft testers. In 2003 Microsoft began to refocus its software processes on engineering practices and Bj decided to move to a Test Architect role and share his passion for teaching and mentoring. Bj also teaches software testing courses at the University of Washington, and sits on software testing advisory boards at the University of Washington, the University of California Extension Santa Cruz, and Lake Washington Technical College. He is also a frequent speaker at international software testing conferences.
Bj Rollison Test Architect
Microsoft
http://www.TestingMentor.comhttp://blogs.msdn.com/imtesty
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
“There are more tests in a program with 70 branches then there are teaspoons of water in the Pacific Ocean”
– William Hetzel
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Design an extremely small subset of tests from all possibilitiesQualify the important attributes and capabilities of the application under testIdentify defects and other potential risksProvide accurate information to the decision makers for informed risk-assessmentWithin a limited amount of timeThat provides a high degree of confidence for the entire team
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Functional testing technique in which the tester:methodically analyzes parameter interaction in a complex feature setin order to systematically select a finite subset of tests from all possible combinations
Uses a mathematical approach based on historical failure indicators and empirical dataHelps reduce overall risk by
High defect detection effectivenessHigh levels of structural coverage
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Combinatorial testing is useful when testing a feature in which:
Parameters are directly interdependent
Parameters are semi-coupled
Parameter input is unordered
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Independent parameters
No direct orindirect interaction
Mathematical formulasVarParam1 + VarParam2 = VarParamOut
Sequential operationsOrdered input
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Random evaluation approachesBest guess
Intuition and luckRandom selection
Randomly selected from all combinationsSystematic evaluation approaches
Each choice (EC)Base choice (BC)Orthogonal arrays (OA)Combinatorial analysis (t=n)Exhaustive (AC)
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Finding OA is non-trivialUnrealistic expectationsequal number of variables per parameterRedundant tuples in output
Test B I U S Col.5 Col.6 Col.7
1 unck unck unck unck 1 1 1
2 unck unck unck chk 2 2 2
3 unck chk chk unck 1 2 2
4 unck chk chk chk 2 1 1
5 chk unck chk unck 2 1 2
6 chk unck chk chk 1 2 1
7 chk chk unck unck 2 2 1
8 chk chk unck chk 1 1 2
L8 Orthogonal Array
Taguchi Array Selectorhttp://www.freequality.org
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Identify one base test (usu. ‘happy path’)Iterate through the variable values for one parameter at a time, while keeping the other parameters constant with base testSatisfies 1-wise coverageSingle error coverage
Test 1 2 3 4 5
Bold No Yes No No No
Italic No No Yes No No
Underline No No No Yes No
Strikethrough No No No No Yes
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Basic algorithmList all possible pairsUse greedy search heuristicGenerate one test at a timeInsert as many uncovered pair combinations as possible ineach new test until all t=n covered
Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Bold yes yes yes yes no no no yes no no yes yes no yes no no
Italic no yes yes yes yes yes yes yes no no no no no no yes no
Underline no no yes yes no yes yes no yes yes yes yes no no no no
Strikethrough no no no yes no no yes yes no yes no yes yes yes yes no
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Analysis & featuredecomposition
Model parametervariables
PICTTool
ValidateOutput
Modify model
Test
Re-validateOutput
REMEMBER: The output of the tool isbased on the tester’s input...it’s only a tool!
Base ChoiceMatrix
up to t=6
Seeded Input
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Interdependent parametersFontsBoldItalic
Semi-coupled parametersColor (Black, white, red, green, blue, yellow)
Size (1 – 1638, including half sizes)
StrikethroughUnderline
Total number of tests = Cartesian product of variables = 1,920 tests
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
File format (for PICT)param: var1, var2, var3
Identify mutually exclusive variable combinations
Analyze output for “happy path”
Analyze outputfor failure indicators
Copyright © 2008 Bj Rollison. All rights reserved.
Copyright © 2008 Bj Rollison. All rights reserved.
Conditional relationships
Exclude mutually exclusive variable states;
often increases total number of tests
Copyright © 2008 Bj Rollison. All rights reserved.
Aliasing or Variable equivalence
Equivalence variables which are similar;
may sometimes reduce total number of tests
Copyright © 2008 Bj Rollison. All rights reserved.
Weighting variables,Variable preference Give increased weighted
value to important variables; MAY increase
probability of occurrence
Copyright © 2008 Bj Rollison. All rights reserved.
Seeded outputSpecify importantcombinations‘Happy path’BC tests
Seeded input ensures important combinations
are tested
Tahoma seed not used because Tahoma is equivalent to Arial
Copyright © 2008 Bj Rollison. All rights reserved.
Randomize output to increase breadth of
testing and minimize probability of error due to
missed combinations
PICT/r:nswitch
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Invariant conditionsParameters must be different
Example: [Bold] <> [Italic] OR [Underline] <> [Strikethrough]Example: [Strike] <> [DblStrike] AND [SuperScript] <> [SubScript]
Parameter types2 types: Numbers and Strings
Example: Size: 1, 2, 3, 4, 5 and Value: a, b, c, d, eIF [Size] >= 3 THEN Value > “b”;
N-wise output (t = 6 @ effective as exhaustive)Negative testingSub-models
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Greatest number (> 50%) of combinatorial defects are simple pair interactions
Randomizing pair-wise combinations with PICT may improve t=2 detection effectiveness*
Subtle defects still being discovered at t=6Effects are cumulative
Number oftests increaseexponentially, but don’t stop atpair-wise (t=2) 0%
20%40%60%80%
100%
t=2 t=3 t=4 t=5 t=6
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Pair-wise found 98 percent of all detectable faults seeded in five applications
All faults in four applications
Evaluation of Combination Strategies for Test Case Selection (Grindal, et. al.)
Pairwise
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
“In this study we found no significant difference in the FDE of n-way and random combinatorial test suites. …the result is not unexpected.”Tests selectedrandomly fromall possiblecombinations# of tests equal to n-way # of tests
(Patrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopu)
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
" … a set of 29 pairwise tests gave 90% block coverage for the UNIX sort command. We also compared pairwise testing with random input testing and found that pairwise testing gave better coverage.”
The AETG System: An Approach to Testing Based on Combinatorial Design(Cohen, et. al.)
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Attrib.exe – path + 6 optional argv.
Findstr.exe – str + 19 optional argv. (421,200 EC tests)
Total # of blocks = 483 “Default” test suite Exhaustive coverage Pair-wise coverage
Number of test cases 9 972 13
Blocks covered 358 370 370
Code coverage 74% 77% 77%
Functions not covered 15 15 15
Total # of blocks = 1317 Hand-crafted tests N=2 coverage N=3 coverage N=4 coverage
Number of test cases 236 136 800 3533
Blocks covered 960 979 994 1006
Code coverage 73% 74% 75% 76%
Functions not covered 11 11 10 10
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Regression test suite time reduced(104 man-days to 32 hours)Test design and implementation reduced (44 man-months to 4 man-months)Reduced OS configuration testing 50% with zero bug bounce (ZBB)Increase in automated test development adopting data-driven output of PICT (@ 30 tests/week to 235 tests/week)3 ‘show-stopper’ defects detected prior to releasesaved @ $750,000 (each QFE @ $125K)
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Misuse of technique & toolTraining, knowledge, no silver bullets!
Right values not testedIn-depth system knowledge, stochastic data
Highly probable values get too little attentionWeighted values, include seeded input
Complex interactions are missedPair-wise is just the beginning, up to t=6
May produce a defect and not recognize itCommon to all testing, need better oracles
May be less effective if # of parameters is small*
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Best Practice compared to other approaches for testing complex parameter interactions
Applied correctly (BC & t=2 t=6, analysis)Provides critical information earlier
High probability of early defect detectionHigh structural coverage; t=6 ≈ exhaustive
Easy to integrate into automation (data-driven)But,
Requires in-depth technical knowledgeMust verify output – it’s only a tool!Don’t forget Beizer’s Pesticide Paradox!
Copyright © 2008 Bj Rollison. All rights reserved.
Helping Testers Unleash Their Potential!TM
http://www.TestingMentor.comBj.Rollison@TestingMentor.com
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
http://www.pairwise.orgPairwise Testing in the Real World: Practical Extensions to Test-Case ScenariosJacek Czerwonka(http://msdn2.microsoft.com/en-us/testing/cc150619.aspx?topic=306121)
Comparing the Fault Detection Effectiveness of N-way and Random Test SuitesPatrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopuhttp://portal.acm.org/citation.cfm?doid=1159733.1159742
Software and Hardware Testing Using Combinatorial Covering SuitesAlan Hartman, IBM Haifa Research Laboratoryhttp://www.haifa.il.ibm.com/projects/verification/mdt/papers/CombinatorialProblemsinSWTesting-finalDraft180703.pdf
Copyright © 2008 Bj Rollison. All rights reserved.Helping Testers
Unleash Their Potential!
Pseudo-Exhaustive Testing for SoftwareD. Richard Kuhn and Vadim Okun(http://csrc.nist.gov/acts/PID258305.pdf)
Failure Modes in Medical Device SoftwareDolores R. Wallace and D. Richard Kuhn (http://csrc.nist.gov/staff/Kuhn/final-rqse.pdf)
The Combinatorial Design Approach to Automatic Test GenerationD. M. Cohen, S. R. Dalal, J. Parelius, G. C. Patton(http://www.argreenhouse.com/papers/gcp/AETGissre96.shtml)
An Evaluation of Combination Strategies for Test Case Selection M. Grindal, B. Lindström, J. Offutt, and S.F. Andler(http://www.his.se/upload/19352/EvalCombStratTechRepFinal.pdf)
Recommended