8/9/2019 HelpMeOut Talk - CHI2010

1/70

Bjrn HartmannUniversity of California, Berkeley

EECS, Computer Science Divisionbjoern@eecs.berkeley.edu

Daniel MacDougall, Joel Brandt,

Scott R. KlemmerStanford University HCI Group

Suggesting solutions to error messages

What would other

programmers do?

mailto:bjoern@eecs.berkeley.edumailto:bjoern@eecs.berkeley.edumailto:bjoern@eecs.berkeley.edu

8/9/2019 HelpMeOut Talk - CHI2010

2/70

End-user programmers outnumberprofessional software developers.

To write code, end users often find and

modify existing examples.

8/9/2019 HelpMeOut Talk - CHI2010

3/70

End-user programmers outnumberprofessional software developers.

To write code, end users often find and

modify existing examples.This strategy does not work well fordebugging. (but it should)

8/9/2019 HelpMeOut Talk - CHI2010

4/70

8/9/2019 HelpMeOut Talk - CHI2010

5/70

8/9/2019 HelpMeOut Talk - CHI2010

6/70

8/9/2019 HelpMeOut Talk - CHI2010

7/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 7

8/9/2019 HelpMeOut Talk - CHI2010

8/70

Research Idea:

Help programmers fix errors by showingexamples how other programmers have

fixed similar errors in the past.(In real-time, automatically)

8/9/2019 HelpMeOut Talk - CHI2010

9/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 9

8/9/2019 HelpMeOut Talk - CHI2010

10/70

images: processing.org

8/9/2019 HelpMeOut Talk - CHI2010

11/70

photo: d. shiffman

8/9/2019 HelpMeOut Talk - CHI2010

12/70

MIT Media Lab, Sensable Cities

8/9/2019 HelpMeOut Talk - CHI2010

13/70

8/9/2019 HelpMeOut Talk - CHI2010

14/70

8/9/2019 HelpMeOut Talk - CHI2010

15/70

8/9/2019 HelpMeOut Talk - CHI2010

16/70

Collecting example fixes

Finding relevant fixes

Presenting found fixes

8/9/2019 HelpMeOut Talk - CHI2010

17/70

Collecting example fixes

Finding relevant fixes

Presenting found fixes

8/9/2019 HelpMeOut Talk - CHI2010

18/70

Goal:Build a database of example bug fixes that

take code from a broken into a fixed state.(With minimal user intervention.)

8/9/2019 HelpMeOut Talk - CHI2010

19/70

Possible approaches:

Web search

8/9/2019 HelpMeOut Talk - CHI2010

20/70

Possible approaches:

Web search

Mining version control repositoriesBugMem, Kim, SIGSOFT06;

Dynamine, Livshits, SIGSOFT05

8/9/2019 HelpMeOut Talk - CHI2010

21/70

Possible approaches:

Web search

Mining version control repositoriesBugMem, Kim, SIGSOFT06;

Dynamine, Livshits, SIGSOFT05

Instrumenting IDEsinGimp, Terry, CHI08CommunityCommands, Matejka, UIST09Photoshop Tutorials by Demonstration, Grabler, SIGGRAPH09

8/9/2019 HelpMeOut Talk - CHI2010

22/70

Approach:Instrument programmers IDEs to

automatically collect errors and sourcecode changes that correct those errors.

(Caveat: no logic bugs)

8/9/2019 HelpMeOut Talk - CHI2010

23/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 23

8/9/2019 HelpMeOut Talk - CHI2010

24/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 24

8/9/2019 HelpMeOut Talk - CHI2010

25/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 25

8/9/2019 HelpMeOut Talk - CHI2010

26/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 26

8/9/2019 HelpMeOut Talk - CHI2010

27/70

8/9/2019 HelpMeOut Talk - CHI2010

28/70

Save source and error message at each compilation

Look for transitions from broken to error-free state

8/9/2019 HelpMeOut Talk - CHI2010

29/70

Progress heuristicIf the code makes progress past the

previous point of error,mark error as resolved

8/9/2019 HelpMeOut Talk - CHI2010

30/70

Implementation1. At runtime, count how often each line is executed

2. On error, store lines execution count

3. On next run, if count for error line exceeds previouscount, mark as resolved.

Important assumptions

1. Errors cause uncaught exception

2. Identical input across executions

8/9/2019 HelpMeOut Talk - CHI2010

31/70

Collecting example fixes

Finding relevant fixes

Presenting found fixes

8/9/2019 HelpMeOut Talk - CHI2010

32/70

Goal:Given a users broken code, find mostrelevant examples in the fix database.

8/9/2019 HelpMeOut Talk - CHI2010

33/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 33

8/9/2019 HelpMeOut Talk - CHI2010

34/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 34

8/9/2019 HelpMeOut Talk - CHI2010

35/70

Approach:Use both error message and source codestructure to search for relevant fixes.

8/9/2019 HelpMeOut Talk - CHI2010

36/70

Users code

(broken)

Example

(broken)

Example

(fixed)

patch

8/9/2019 HelpMeOut Talk - CHI2010

37/70

Users code

(broken)

Example

(broken)

similar to ?

8/9/2019 HelpMeOut Talk - CHI2010

38/70

Naive implementation:Look at code surrounding error andcompute string edit distance.

8/9/2019 HelpMeOut Talk - CHI2010

39/70

Better:

Match on code structure, not text.

8/9/2019 HelpMeOut Talk - CHI2010

40/70

Better: match on structure, not text

Replace identifiers with lexical analyzer.

Assumption: Locality (i.e., fix is near error)

8/9/2019 HelpMeOut Talk - CHI2010

41/70

Can we use the same approach for codewith runtime errors?

No, because such errors lacklocality.

String s = null;//...

//...System.out.println(s);

root cause

exception

8/9/2019 HelpMeOut Talk - CHI2010

42/70

Collecting example fixes

Finding relevant fixes

Presenting found fixes

8/9/2019 HelpMeOut Talk - CHI2010

43/70

Goal:Present fixes to users in a way thatfacilitates reasoning about relevance &applicability to their own code

8/9/2019 HelpMeOut Talk - CHI2010

44/70

Example

(broken)

Example

(fixed)

:Users code

(broken)

?:

8/9/2019 HelpMeOut Talk - CHI2010

45/70

Approach:

1. Show before-after comparison

2. Offer explanations

3. Attempt to patch users codewhere possible

8/9/2019 HelpMeOut Talk - CHI2010

46/70

8/9/2019 HelpMeOut Talk - CHI2010

47/70

8/9/2019 HelpMeOut Talk - CHI2010

48/70

Where do these explanationscome from?

8/9/2019 HelpMeOut Talk - CHI2010

49/70

Crowdsource themfrom experts.

8/9/2019 HelpMeOut Talk - CHI2010

50/70

How can we best encourageexperts to contribute?

8/9/2019 HelpMeOut Talk - CHI2010

51/70

We ensure their efforts havemaximum impact.

8/9/2019 HelpMeOut Talk - CHI2010

52/70

8/9/2019 HelpMeOut Talk - CHI2010

53/70

8/9/2019 HelpMeOut Talk - CHI2010

54/70

Example

(broken)

Example

(fixed)

Users code

(broken)

?patch patch

8/9/2019 HelpMeOut Talk - CHI2010

55/70

float [] A = new float[];

float [] Z = new float[10];

8/9/2019 HelpMeOut Talk - CHI2010

56/70

8/9/2019 HelpMeOut Talk - CHI2010

57/70

8/9/2019 HelpMeOut Talk - CHI2010

58/70

What size database is needed to returnuseful suggestions?

How useful are suggestions collectedthrough instrumentation?

Which types of errors are covered?

Which arent?

8/9/2019 HelpMeOut Talk - CHI2010

59/70

2 workshops on Processing at California College of Art

8 participants; 39 person-hours of usage

8/9/2019 HelpMeOut Talk - CHI2010

60/70

8/9/2019 HelpMeOut Talk - CHI2010

61/70

image: arkadian.eu

8/9/2019 HelpMeOut Talk - CHI2010

62/70

274 queries (7 per person/hour)

84% returned at least one suggestion

8/9/2019 HelpMeOut Talk - CHI2010

63/70

Manual analysis of queries and returnedsuggestions:

Does at least one of the top 3 returnedsuggestions lead to a solution?

8/9/2019 HelpMeOut Talk - CHI2010

64/70

8/9/2019 HelpMeOut Talk - CHI2010

65/70

HelpMeOut gave users a helpful

suggestion half of the time.

8/9/2019 HelpMeOut Talk - CHI2010

66/70

8/9/2019 HelpMeOut Talk - CHI2010

67/70

Structured & syntax directed editors canprevent syntax errors...Cornell Programm Synthesizer, Teitelbaum

Scratch, Resnick, CAMC09

...but they increase the viscosity of code.

Constant background compilation with

error hinting aids syntax error correction.

8/9/2019 HelpMeOut Talk - CHI2010

68/70

dynamic languagessuggest improvements

other authoring domains

8/9/2019 HelpMeOut Talk - CHI2010

69/70

HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 69

8/9/2019 HelpMeOut Talk - CHI2010

70/70

bjoern@eecs.berkeley.edubid berkeley edu

http://hci.berkeley.edu/mailto:bjoern@eecs.berkeley.eduhttp://hci.berkeley.edu/mailto:bjoern@eecs.berkeley.edumailto:bjoern@eecs.berkeley.edu