Faults and Regression Testing - Fault interaction and its repercussions

Fault Interaction and its Repercussions

Nicholas DiGiuseppe James A. Jones

UC Irvine Spider Lab

Fault Interaction Defined

Multi-Type Synergy Obfuscation Pass/Fail Independence Pass/Fail Independence

4 Types of Interaction

Fault Interaction Defined

4 Types of Interaction

Multi-Type Synergy Obfuscation Pass/Fail Independence Obfuscation

Fault Interaction Defined

4 Types of Interaction

Multi-Type Synergy Obfuscation Pass/Fail Independence

Synergy

Fault Interaction Defined

4 Types of Interaction

Multi-Type Synergy Obfuscation Pass/Fail Independence

Multi-Type

We Need More Fault Understanding

Conferences: ASE, FSE, ICSE, ICSM, ICST, ISSTA

Years: 2008–2009

27 Fault Localization = 2.25 per conference per year 4 Fault Behavior = 0.333 per conference per year

Motivating Example

Expected Results

Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0

X = input [0] - 2 Y = input [0] - 2 If (X == Y) print 1 else print 0

Actual Results

Pass/Fail Status

1 Pass 1 Fail

Motivating Example

Expected Results

Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0

X = input [0] - 2 Y = input [1] / 2 If (X == Y) print 1 else print 0

Actual Results

Pass/Fail Status

0 Fail 1 Fail

Motivating Example

Expected Results

Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0

X = input [0] * 2 Y = input [1] / 2 If (X == Y) print 1 else print 0

Actual Results

Pass/Fail Status

1 Pass 0 Pass

Our Experiment Flex Gzip Replace Schedule Sed Space Total

1 Fault 21 20 25 20 20 33 139

2 Faults 189 182 185 165 188 200 1109

3 Faults 502 505 598 500 660 200 2965

4 Faults 1000 1000 700 500 950 200 4350

5 Faults 1000 1000 700 500 950 200 4350

17 Faults 1000 1000 700 500 950 200 4350

Total Versions 14,712 14,707 10,608 7,685 14,168 3,233 65,113

Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733

Number of Executions

7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433

Flex Gzip Replace Schedule Sed Space Total

1 Fault 21 20 25 20 20 33 139

2 Faults 189 182 185 165 188 200 1109

3 Faults 502 505 598 500 660 200 2965

4 Faults 1000 1000 700 500 950 200 4350

5 Faults 1000 1000 700 500 950 200 4350

17 Faults 1000 1000 700 500 950 200 4350

Total Versions 14,712 14,707 10,608 7,685 14,168 3,233 65,113

Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733

Number of Executions

7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433

Total Versions 65,113

Number of Executions 138,239,433

Our Experiment

Results

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

a

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults

0.0

0.2

0.4

0.6

0.8

1.0

Perc

enta

geof

Inte

ract

ion

Obfuscation

Synergy Independence

Multi Type

Impact of Fault Interaction

Fault Localization

Failure Clustering

Regression Testing

Run Test

Suite

Cluster

FailuresAssign Clusters

to Developers

Fix Faults

in Clusters

Pass

Two Faults Fix a Fault One Fault

Test 1

Test 2

Test 3

Pass

Fail

Pass

Pass

Fail

Time

Results Continued Gzip Flex Replace Schedule Sed Space

LoEC 2013 4020 242 150 2253 3651

Branches 1727 2680 182 66 2554 1190

Methods 514 1261 63 41 416 717

Data Dep 45,291 143,075 2005 1327 94,614 29,883

Control Dep 26,766 61,279 1444 1074 21,991 25,047

Total Dep 72,057 204,354 3449 2401 116,605 54,930

LoEC: Branches

1.165 1.5 3.818 2.273 1.893 3.068

LoEC: Methods 3.916 3.188 3.841 3.659 5.416 5.092

LoEC: Data Dep

0.044 0.028 0.121 0.113 0.024 0.122

LoEC: Control Dep

0.075 0.066 0.168 0.140 0.103 0.146

LoEC: Total Dep

0.028 0.020 0.070 0.062 0.019 0.067

Conclusions

Defined Fault Interaction Demonstrated that Fault Interaction is prominent (primarily fault obfuscation) Demonstrated that Fault Interaction can have a real impact on practice and research Demonstrated that the size of a program affects Fault Interaction

Take Home Message

Fault Interaction is prevalent Fault Obfuscation is prevalent

We need to adapt our techniques

Conclusions

Thank You!

This work was partially funded by: University of California, Irvine