Andrea Calvagna – University of Catania - IT Angelo Gargantini – University of Bergamo – IT TAP 2009 – Zürich – 2nd July 2009 C OMBINING S ATISFIABILITY

Andrea Calvagna – University of Catania - ITAngelo Gargantini – University of Bergamo – IT

TAP 2009 – Zürich – 2nd July 2009

COMBINING SATISFIABILITY SOLVING AND HEURISTICS TOCONSTRAINED COMBINATORIAL INTERACTION TESTING

CALVAGNA & GARGANTINI - COMBINING SATISFIABILITY SOLVING AND HEURISTICS TO CONSTRAINED COMBINATORIAL INTERACTION TESTING

OUTLINE

Background on Combinatorial Interaction Testing

constraints

logic approach to CIT [TAP 08]

Test generation by satisfiability solving in the presence of constraints – by using Yices

Heuristics - for ordering test predicates

Experimental assessment


INTERACTION TESTING

IT : interaction testing

identify independent features and representative values (the inputs)

test the interaction between them

highly configurable systems Software for families of products (like cell phones) Sw like gcc compiler > 1400 optional features …


HOW MUCH TO TEST?

34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests?

Every possible combination of inputs? Which interactions cause faults?


COMBINATORIAL APPROACH

Pairwise combination instead of exhaustive

Generate combinations that efficiently cover all pairs of values

Extended by t-wise: test all the combinations of t values

Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs (triples,…) reduces the number of test cases, but reveals most faults


EXAMPLE

3 variables with 3 values each: 33 = 27 possible combinations

Combinatorial testing with much fewer tests

Display Mode Color Screen size

full-graphics Monochrome Hand-held

Low resolution 16-bit Laptop

text-only True-color Full-size


TEST SUITE

Test Color Display Mode

Screen Size

1 Monochrome Full-graphics

Hand-held

2 16-bit Text-only Laptop

3 True-color Full-graphics

Hand-held

4 … … …

One test covers many combinations:

e.g. Test 1 covers 3 pairs:(Monochrome, Full-graphics)(Monochrome, Hand-held)(Full-graphics, Hand-held)

pairwise testing can be achieved by only 9 tests

2100 combinations with 10 tests; 1020 200 tests; …


CIT EFFECTIVENESS

Experiments show that CIT is

effective

finds faults that traditional testing may be not able to find

efficient

A low degree of interaction between inputs can already discover most faults Pairwise is the most used

Never with interaction > 6


METHODS FOR BUILDING CIT TEST SUITES

Methods and tools

IPO, AETG, PICT, TestCover, …

www.pairwise.org

Main goal: producing small test suites

Because the problem of generating a minimum test suit for combinatorial testing is NP-complete, most methods and tools use a greedy approach


ADDING CONSTRAINTS

Constraints express relations among inputs: a valid test must not violate the constraints

E.g. if the display mode is in text-only, the color is monochrome

Constrained CIT

Only few test generation methods support constraints, all of them only as “forbidden tuples”

Goal: supporting expressive constraints

Easier to obtain from the requirements


LOGIC APPROACH TO CCIT [TAP08]

formalize pairwise testing: express each pair as a corresponding logical expression, a test predicate

p1 = v1 /\ p2 = v2 p1 and p2 are inputs, v1 and v2 their values

Similarly, the t-wise coverage can be modeled by a set of test predicates, each of the type

p1 = v1 /\ p2 = v2 /\ … /\ pt = vt


TEST = LOGICAL MODEL

test: an assignment to all the inputs of the system

a test ts covers a test predicate tp if and only if it is a model of tp:

ts |= tp

Finding a test suite finding a model for each test predicate


ADDING CONSTRAINTS

With the constraints c1,…, cn, a test is a model of tp and the constraints c_i

ts |= tp /\ c1 /\ … /\ cn

the constraints become first class citizens and they can be represented by logical expressions too

Finding tests becomes a problem of finding a model of a complex expression

many techniques like, constraint solvers, model checkers [AFM08], SAT solver, SMT solvers …


YICES [CSL.SRI.COM]

Efficient SMT solver that decides the satisfiability of arbitrary formulas Expressive language for constraints and input models

Powerful algorithms to find models

YICES

Test predicate

constraints

Model = test


TEST SUITE GENERATION A SIMPLIFIED VERSION

Test predicates

Input model

Test suite builder

Test predicate Model=

test

constraints YICES


MONITORING AND REDUCTION

Monitoring: A test covers may cover several test predicates which can be removed from the pool

Reduction: at the end, a minimal set of tests that cover all the test predicates can be found by a greedy algorithm t1

t2


COLLECTING

Test suite builder

Model = test that covers all the test predicates collected

YICESCollected test

predicatestp1 /\ tp2 /\ tp3

Instead of one test for every tp, collect the tps to build a conjoint


ORDERING

In which order test predicates can be collected?

First simple two policies:

1. the order in which tp are generated

2. random order


RANDOM ORDERING

non deterministic ordering

Like most CIT approaches: run the method 50 times and then take the best result

GOAL: finding a deterministic ordering that performs as better as the random ordering

Best result = min test suite

Worst result = max test suite


DETERMINISTIC ORDERING POLICIES

Several criteria for ordering the test predicates can be defined

Looking at the tests already generated and at the test predicates still to cover


DETERMINISTIC ORDERING POLICIES

touch Counting explored assignments

count the number of assignment var = val contained in the tps which are already contained in a test of the test suite and take the tp which has fewer assignments already touched

min Least used assignments

Consider only the least used assignment in a tp

max Most used assignment

Consider only the most used assignment in the tp

dev Even usage of assignments

Consider the standard deviation in the usage count an try to keep the usage even

/c variants Consider not only the tests in the test suite (test already generated) but also the other test predicates in the collected test predicate


CASE STUDIES

13 specifications with constraints taken from the literature

From 3 ^3 to 8.3 x 10^6 combinations

From 6 to 50 constraints (forbidden tuples)


EXPERIMENTAL RESULTS

Among policies: One policy (touch/c)

performed always better except in two cases in which touch performed better

As generated: bad policy


COMPARISON WITH RANDOM

randommax

min

avg

Bad policies: worse than average of random ordering

Good policies: better than average of random ordering


COMPARISON WITH RANDOM

Best heuristics touch/c meaning

better than min never Random is still a viable option if one can run the algorithms several (at least 50) times

like min 3/13better than avg 9/13 Touch/c gives good resultsbetter than max 12/13

Like max 1/13 If one can run the algorithms only once, a deterministic policy can guarantee better or equal results

Worst than max Never No risk in using touch/c


CONCLUSIONS

Logic approach can be applied to constrained combinatorial testing It allows a more natural (and expressive) formalization of constraints

Good heuristics are defined to order the test predicates and obtain small test suites Random ordering still useful in case of unlimited resources

Prototype tool available at http://cs.unibg.it/gargantini/software/atgt/

http://cs.unibg.it/gargantini/software/atgt/

Documents

Andrea Calvagna – University of Catania - IT Angelo Gargantini – University of Bergamo – IT TAP 2009 – Zürich – 2nd July 2009 C OMBINING S ATISFIABILITY