Automatically Generated Patches as Debugging Aids: A Human Study (FSE 2014)

Automatically Generated Patches as Debugging Aids: A Human Study

Yida Tao, Jindae Kim, Sunghun Kim

Dept. of CSE, The Hong Kong University of Science and Technology

Chang Xu

State Key Lab for Novel Software Technology, Nanjing University

• Promising research progress• ClearView1: Prevent all 10 Firefox exploits

• GenProg2: Fix 55/105 real bugs

[1] Automatically Patching Errors in Deployed Software. Perkins et al. SOSP’09[2] A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. Le Goues et al. ICSE’12

2

Automatic Program Repair

3

Automatic Program Repair

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code

4

“It won't get your bug patched any quicker. You’ll just have shifted the coders' attention away from their own app's bugs, and onto the repair tool’s bugs.”

Automatic Program Repair

#what-could-possibly-go-wrong

• Blackbox repair

• Increasing maintenance cost

• Vulnerable to attack

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13

5

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13

#program-out-of-control

6

#what-could-possibly-go-wrong

• Blackbox repair

• Increasing maintenance cost

• Vulnerable to attack

Use automatically generated patches as debugging aids

7

Use automatically generated patches as debugging aids

Our Human Study

• Investigate the usefulness of generated patches as debugging aids

• Discuss the impact of patch quality on debugging performance

• Explore practitioners’ feedback on adopting automatic program repair

8

Methodology

9

BugsParticipantsDebugging aid

10

Debugis given to

BugsParticipantsDebugging aid 11

BugsParticipantsDebugging aid 12

Low-quality generated patch

BugsParticipantsDebugging aid 13

Low-quality generated patch

High-quality generated patch

BugsParticipantsDebugging aid 14

Low-quality generated patch

High-quality generated patch

Buggy method location

BugsParticipantsDebugging aid 15

Grad: 44

Engr: 28

MTurk: 23

95 Participants

CS graduate students

Industrial software engineers

Amazon Mechanical Turk workers

BugsParticipantsDebugging aid 16

BugsParticipantsDebugging aid 17

44 Graduate students• Between-group design

14 students

15 students

15 students

BugsParticipantsDebugging aid 18

44 Graduate students• Between-group design

Low-quality generated patch

High-quality generated patch

Buggy method location

14 students

15 students

15 students

BugsParticipantsDebugging aid 19

44 Graduate students• Between-group design• Onsite setting

• Eclipse IDE• Supervised session

Low-quality generated patch

High-quality generated patch

Buggy method location

14 students

15 students

15 students

BugsParticipantsDebugging aid 20

Low-quality generated patch

High-quality generated patch

Buggy method location

Remote participants(28 Engr + 23 MTurk)

• Within-group design

BugsParticipantsDebugging aid 21

Remote participants(28 Engr + 23 MTurk)

• Within-group design• Online debugging system

Low-quality generated patch

High-quality generated patch

Buggy method location

BugsParticipantsDebugging aid 22

BugsParticipantsDebugging aid 23

Bug Selection Criteria

• Real bugs

• The bug has accepted patches written by developers

• Proper number of bugs

• The bug has generated patches with different quality

BugsParticipantsDebugging aid 24

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

BugsParticipantsDebugging aid 25

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)if(sub!=null){

args[i+1] = sub.toString();}

}

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)args[parenCount+1] = new Integer(reImpl.leftContext.length);

}

BugsParticipantsDebugging aid 26

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

avg. ranking from 85 devs and students

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)if(sub!=null){

args[i+1] = sub.toString();}

}

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)args[parenCount+1] = new Integer(reImpl.leftContext.length);

}

1.6

2.8

BugsParticipantsDebugging aid 27

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

avg. ranking from 85 devs and students

High-Quality Patch Low-Quality patch

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)if(sub!=null){

args[i+1] = sub.toString();}

}

for (int i=0; i<parenCount; i++)SubString sub = (SubString)parens.get(i)args[parenCount+1] = new Integer(reImpl.leftContext.length);

}

1.6

2.8

BugsParticipantsDebugging aid 28

BugsParticipantsDebugging aid 29

Participants submit 337 patches as their debugging outcome

BugsParticipantsDebugging aid 30

Participants submit 337 patches as their debugging outcome

Location109

LowQ112

HighQ116# submitted patches

w.r.t debugging aid

BugsParticipantsDebugging aid 31

Participants submit 337 patches as their debugging outcome

Location109

LowQ112

HighQ116# submitted patches

w.r.t debugging aid

Bug166

Bug274

Bug359

Bug476

Bug562

# submitted patches w.r.t bugs

Evaluation of debugging performance

32

Patch CorrectnessCorrectness

33

Patch Correctness

• Passing test casesCorrectness

34

Patch Correctness

• Passing test cases

• Matching the semantics of original accepted patches

Correctness

35

Patch Correctness

• Passing test cases

• Matching the semantics of original accepted patches

• 3 evaluators

Correctness

36

Debugging Time

• Eclipse Plug-in

• Website Timer

Correctness

Debugging time

37

Correctness

Debugging time

• Independent variables• Debugging aids

• Bugs

• Participant types

• Programming experience

38

Multiple Regression AnalysisCorrectness

Debugging time

• Independent variables• Debugging aids

• Bugs

• Participant types

• Programming experience

correctness = α0 + α1 ∙ x1 + α2 ∙ x2 + α3 ∙ x3 + α4 ∙ x4

debugging time = β0 + β1 ∙ x1 + β2 ∙ x2 + β3 ∙ x3 + β4 ∙ x4

39

Post-study Survey

• Helpfulness of debugging aids

• Difficulty of bugs

• Opinions on using generated patches as debugging aids

Correctness

Debugging time

Survey feedback

40

Results

41

High-quality patches significantly improve debugging correctness

1

48%

33%

71%

42

High-quality patches significantly improve debugging correctness

1

48%

33%

71%

43

Location LowQ HighQ

% of correct patches

48%

71%

Location LowQ HighQ

% of correct patches

High-quality patches significantly improve debugging correctness

1

Positive Coefficient = 1.25

p-value= 0.00 < 0.05 48%

71%

44

Location LowQ HighQ

% of correct patches

Low-quality patches slightly undermine debugging correctness

2

48%

33%

71%

45

Location LowQ HighQ

% of correct patches

Low-quality patches slightly undermine debugging correctness

2

Negative Coefficient = -0.55

p-value= 0.09 48%

33%

71%

46

Location LowQ HighQ

% of correct patches

Low-quality patches can undermine debugging correctness

2

Negative Coefficient = -0.55

p-value= 0.09 48%

33%

71%

47

High-quality patches are more useful for difficult bugs3

48

High-quality patches are more useful for difficult bugs3

49

2

3

4

5

Bug Difficulty

Bug1Math-280

Bug2Rhino-114493

Bug3Rhino-192226

Bug4Rhino-217379

Bug5Rhino-76683

High-quality patches are more useful for difficult bugs3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Bug1 Bug2 Bug3 Bug4 Bug5

% of correct patches

Location LowQ HighQ

50

2

3

4

5

Bug Difficulty

Bug1Math-280

Bug2Rhino-114493

Bug3Rhino-192226

Bug4Rhino-217379

Bug5Rhino-76683

4The type of debugging aid does not affect debugging time

51

4The type of debugging aid does not affect debugging time

0

20

40

60

80

Location LowQ HighQ

Debugging time (min)

52

5Other factors’ impact on debugging performance

Difficult bugs significantly slow down debugging

Engr and MTurk are more likely to debug correctly

Novices tend to benefit more from HighQ patches

53

Helpfulness of debugging aidsVery helpful

Helpful

Medium

Slightly Helpful

Not Helpful

54

Participants consider high-quality generated patches much more helpful than low-quality patches

Low-quality generated patch

High-quality generated patch

Mann-Whitney U test

p-value = 0.001

6

Feedback

55

56

Quick starting point

• Point to the buggy area

• Brainstorm

“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”

57

Quick starting point

• Point to the buggy area

• Brainstorm

Confusing, incomplete, misleading

• Wrong lead, especially for novices

• Require further human perfection

“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”

58

“Generated patches would be good at recognizing obvious problems”

“…but may not recognize more involved defects.”

59

“Generated patches would be good at recognizing obvious problems”

“…but may not recognize more involved defects.”

60

“Generated patches simplify the problem”

“…but they may over-simplify it by not addressing the root cause.”

“I would use generated patches as debugging aids, as they provide extra diagnostic information”

61

“I would use generated patches as debugging aids, as they provide extra diagnostic information”

“…along with access to standard debugging tools.”

62

Threats to Validity

63

Threats to Validity

• Bugs and generated patches may not be representative

• Quality measure of generated patches may not generalize

• May not generalize to domain experts

• Possibility of blindly reusing generated patches• Remove patches that are submitted less than 1 minute

64

Takeaway

65

• Auto-generated patches can be useful as debugging aids• Participants fix bugs more correctly with auto-

generated patches

• Quality control is required• Participants’ debugging correctness is

compromised with low-quality generated patches

• Maximize the benefits• Difficult bugs

• Novice developers