Establishing Theoretical Minimal Sets of Mutants

ICST 2014

Paul Ammann

Joint work with

Marcio Eduardo Delamaro Jeff Offutt

April 1, 2014

2

Outline

• The situation– Researchers use mutation analysis to evaluate test selection strategies

• The problem– What do mutation scores mean?

• The model– Motivating idea: Minimal mutant sets don’t have redundant mutants

• Need to define notion of redundancy– Main result: Dynamic subsumption = Minimal mutant sets

• Reduced mutation: Is it close to minimal?– Apply model to Siemens suite

• Result: Huge gap– Good news: That’s an opportunity!

Researchers Use Mutation Analysis to Evaluate Test Selection Strategies

Test SelectionStrategy CTest Selection

Strategy BTest SelectionStrategy A

Carefully ChosenArtifacts

“Good”Tests

Deep Analysis

Test Set CSelect Test Sets withTest Selection Strategies Test Set B

Test Set A

20 25 30 35 40 45 50 550.820.840.860.880.900.92

Test Set Size

Mut

ation

Sco

re

Measure

Exactly What Does A Score of 91% Mean?3

The Problem With Mutation Scores

m1 m2 m3 m4

t1 t2 t3 t4 t5

Evaluate 3 Test Sets with 4 Mutants:A: {t1, t2}B : {t2, t5}C: {t3}

A B C0

102030405060708090

100

Mutation Scores for 3 Test Sets

B scores 75%Is that good?

4

Let’s Add Some More Mutants

m1 m2 m3 m4 m5 m6 m7 m8 m9 m10

t1 t2 t3 t4 t5

Evaluate 3 Test Sets with 10 Mutants:A: {t1, t2}B: {t2, t5}C : {t3}

A B C0

102030405060708090

100

Mutation Scores for 3 Test Sets

Now B scores 90%!Did B just get better?

Every test kills m8 What’s the point? The same tests kill m3 and m6. We say that T does not distinguish m3 from m6

Ditto for m5 and m9

5

Let’s Throw Away Some Mutants

m1 m2

t1 t2 t3 t4 t5

Evaluate 3 Test Sets with 2 Mutants:A: {t1, t2}B : {t2, t5}C : {t3}

A B C0

102030405060708090

100

Mutation Scores for 3 Test Sets

Now B scores 100%Did B get even better?

6

All Together Now

m1 m2 m3 m4 m5 m6 m7 m8 m9 m10

t1 t2 t3 t4 t5

Evaluate 3 Test Sets with Various Mutants:A: {t1, t2}B : {t2, t5}C: {t3}

Cumulative Scores

Is B lousy or good?

A B C0

10

20

30

40

50

60

70

80

90

1002 Mutants 4 Mutants

10 Mutants

What about C?7

What Makes a Mutant Redundant?m1 m2 m3 m4

t1 t2 t3 t4 t5

Choose M = {m1, m2, m3, m4}Choose T = {t1, t2, t3, t4, t5}Minimal test sets wrt M and T: {t1, t2}, {t1, t3}, {t4}

Try removing m1: M1 = M - {m1}Minimal test sets wrt M1 and T: {t1, t2}, {t1, t3}, {t4}No change, so m1 is redundant

Basic Idea: Throwing away a redundant mutant has

no effect on the minimal test sets.

Try removing m3: M3 = M - {m3}Minimal test sets wrt M3 and T: {t1}, {t4}A change, so m3 is not redundant

Try removing m4: M4 = M - {m4}Minimal test sets wrt M4 and T: {t1, t2}, {t1, t3}, {t2, t5}, {t3, t5}, {t4}A change, so m4 is not redundant

Ditto for M2 = M - {m2}

8

9

Minimal Sets of Mutants• Definition– M is minimal if it does not contain redundant mutants

• Minimal mutant sets from the definition– Requires computing all minimal test sets, which is NP

complete • We need an efficient algorithm for finding minimal

mutant sets– Turn to dynamic subsumption• Subsumption with respect to a test set

10

Dynamic Subsumption

Test set T

Tests that kill mj

Tests that kill mi

Tests that kill mk

mi → mj mi → mk✔ ✖? ?

11

Efficiently Computing Minimal Sets of Mutants

• Formally: mx dynamically subsumes my wrt T iff– Some test in T kills mx

– Every test in T that kills mx also kills my

• Main result: Mutant set M minimal wrt T = no dynamic subsumption in M

• Properties– Only need to consider mutants in pairs

• Groups of mutants do not make another mutant redundant– Fast – Every minimal mutant set has the same cardinality

• Contrast with minimal test sets

12

What Does This Mean in Practice?• Apply the definitions to the Siemens test bed– See what happens!

• 7 programs– print_tokens– print_tokens2– replace– schedule– schedule2– tcas– totinfo

• Extensive hand-crafted test set

13

Test Characteristics

• Notes:– 512 is an artifact of the Proteum tool

• Our approach applies with any test set– Most tests used were also distinguished– Minimal test set size modest compared to number of tests used

Program Tests Available

Tests Used

Distinguished Tests

Minimal Tests

print_tokens 4073 512 499 12.4print_tokens2 4058 512 479 12.1

replace 5542 512 510 44.4schedule 2650 512 482 14.5

schedule2 1052 512 479 17.1tcas 1608 512 428 41.4

totinfo 4073 512 452 13.3

14

Mutant Characteristics

• “Killed Mutants” means those killed by the test set of size 512• Vast majority of remainder are equivalent

Program Total Mutants

Killed Mutants

Distinguished Mutants

Minimal Mutants

print_tokens 4336 3711 437 28print_tokens2 4746 4042 439 30

replace 11101 8783 2309 58schedule 2109 1838 520 42

schedule2 2627 2132 461 46

tcas 2384 1957 596 61totinfo 6698 5821 835 19

• Most mutants are redundant!– Tiny fraction of mutants are actually minimal wrt 512 tests!– print_tokens: Killing the right 28 mutants guarantees killing all 3711

15

How Good Are Reduced Mutation Strategies?

• We considered five approaches to reduced mutation– STMT: Statement deletion (Proteum SSDL)– ROR: Relation operators (Proteum ORRN)– CON: Replace scalars with constants (Proteum CCSR) – 5RND: 5% Random selection of all mutants– SELECT: “Selective” mutation (Proteum OOAN, OLLN, ORRN, OLNG)

• Method:– Choose test sets adequate for each reduced mutation approach

• wrt test sets analyzed earlier– Compute mutation score

• Against all mutants• Against minimal mutant set• Equivalent mutants hand-identified and removed

16

Reduced Mutation Scores: Raw vs. Minimal

• Notes:– Table entries: Raw Mutation Score: Minimal Mutation Score– Raw Reduced mutation scores make test strategies look good– Minimal Reduced mutation scores do not

Program STMT ROR CON 5RND SELECT

print_tokens 99 : 78 98 : 77 99 : 78 99 : 82 99 : 81print_tokens2 99 : 47 99 : 56 99 : 49 99 : 48 99 : 57

replace 97 : 31 97 : 38 99 : 57 99 : 56 98 : 48schedule 97 : 68 94 : 53 98 : 65 98 : 67 97 : 65

schedule2 97 : 72 92 : 56 98 : 77 98 : 72 97 : 72tcas 88 : 27 90 : 38 88 : 33 94 : 45 93 : 43

totinfo 97 : 38 99 : 59 99 : 39 99 : 54 99 : 60

17

Closer Look: Raw vs. Minimal for STMT

• Raw mutation scores show little variation• Minimal mutation scores show a lot

print_toke

ns

print_toke

ns2

replace

schedule

schedule2

tcas

totinfo0

20

40

60

80

100Mutation Score Minimal Mutation Score

18

Reduced Mutation: Mutants vs. Tests

• Notes:– Table entries: Number of Mutants : Size of Minimal Test Set– Reduced approaches

• Generate many more mutants than minimal• But not nearly enough tests

Program STMT ROR CON 5RND SELECT Minimal

print_tokens 196 : 11 98 : 9 358 : 10 190 : 11 138 : 10 28 : 12.4print_tokens2 203 : 5 192 : 8 445 : 8 198 : 9 244 : 9 30 : 12.1

replace 219 : 23 264 : 27 1053 : 44 443 : 39 499 : 35 58 : 44.4schedule 127 : 10 49 : 7 78 : 13 95 : 12 84 : 10 42 : 14.5

schedule2 117 : 9 75 : 6 119 : 13 110 : 13 121 : 12 46 : 17.1tcas 42 : 12 45 : 14 66 : 14 99 : 24 113 : 18 61 : 41.4

totinfo 110 : 6 167 : 13 469 : 12 294 : 15 332 : 15 19 : 13.3

19

Closer Look: Mutants and Tests for STMT

• STMT usually generates too many mutants• Unfortunately, they aren’t the right ones– Hence, not nearly enough tests

print_toke

ns

print_toke

ns2

replace

schedule

schedule2

tcas

totinfo0

50100150200250

Number Of STMT MutantsNumber of Minimal Mutants

print_toke

ns

print_toke

ns2

replace

schedule

schedule2

tcas

totinfo0

1020304050

Number of STMT TestsNumber of Minimal Tests

20

Discussion

• Huge gap: Reduced mutation vs. minimal mutant sets– Research opportunity!

• The problem with reduced mutation– Reduced approaches don’t consider specific program under test– Maybe it’s time to change that

• Can we analyze specific mutants in a specific program?

• Problem with minimal mutant sets for practical testing– Need mutation adequate tests to compute minimal mutant sets!

• Aren’t we done at that point?

• There is a lot we don’t know about minimal mutant sets– Let’s look at an example from yesterday’s Mutation workshop

Subsumption Graph Example: cal()• 31 nodes of

indistinguished mutants

• 7 nodes of minimal mutants– muJava generated 145

non-equivalent mutants– we only need 7 for

given test set• Static analysis can

refine this graph

21

22

Questions?

• Contact: – {pammann, offutt}@gmu.edu– delamaro@icmc.usp.br