Backstop: A Tool for Debugging Runtime Errors

Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon

Columbia University

Observation Novice/CS1 Java programmers have

difficulty dealing with uncaught exceptions and the resulting stack traces

java.lang.NumberFormatException: For input string: "4.5" at java.lang.NumberFormatException.forInputString (NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:456) at java.lang.Integer.parseInt(Integer.java:497) at Test.main(Test.java:124)

Problems Students are often confused by runtime

errors in Java

Stack traces were not developed with the introductory-level CS student in mind!

Few, if any, tools exist to help novice programmers interpret and debug runtime errors

Our Solution: Backstop A command-line tool to produce detailed

and helpful error messages when an uncaught runtime error (exception) occurs

Also provides debugging support by displaying the lines of code that are executed as a program runs, as well as the values of any variables on those lines

Hypothesis “Backstop allows students to understand

and debug runtime errors more easily than just the Java stacktrace”

Backstop is part of Columbia University’s ongoing efforts to develop tools to help CS1 students work with Java

Overview Related Work Handling of Runtime Errors Usability Study Debugging Support Conclusions and Future Work

Related Work Programming environments with easier-to-

understand compiler errorsBlueJ, ProfessorJ, DrJava, …

Identify most common compiler errorsGauntlet, Expresso, HiC, …

Identifying logical errors InSTEP, DEBUG, …

DebuggingCMeRun

Handling of Runtime Errors Designed to catch any uncaught

exceptions in the student’s program

Once the exception (or any other Java runtime error) is caught, interpret its meaning and provide a user-friendly message that also seeks to provide some enlightenment as to what caused the error

Handling of Runtime Errors BackstopRuntime “wraps” the student’s

(compiled) Java programUses reflection to invoke the “main” methodCatches any uncaught exception and

dispatches it to the appropriate handler

StackTraceElements and source code are used to produce a friendlier, more helpful error message that is appropriate for the type of exception

********************************************************************************* ERROR ***********************************************************************************An error occurred at line 124 of Test.java in the method"main". I'm sorry, the program cannot continue.

The line of code that caused the error is int myNum = Integer.parseInt(arg);

It seems that the code wants to convert a String to aninteger.

However, the String "4.5" appears to be a floating pointvalue, not an integer. Don’t forget, an integer can’tcontain a decimal point. You may want to use a differentdatatype to store the value, like float.

This is called a NumberFormatException.

The full error is as follows:java.lang.NumberFormatException: For input string: "4.5" at java.lang.NumberFormatException.forInputString (NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:456) at java.lang.Integer.parseInt(Integer.java:497) at Test.main(Test.java:124)

Refer to “the code”, not

“you”

Make a suggestion

Reinforce what is already

known

Also use Java terms

Usability Study Research Question: “Is the information

provided by Backstop helpful for understanding runtime errors?”

17 students (8 male, 9 female) who had just completed a CS1 summer course

Students were asked to find/fix a runtime error in code they did not write

After completion, were asked subjective questions about the experience

Usability Study 1. Subject was shown the code for a

program that reads a String and displays the number of occurrences of each letter

2. Subject was then told how the algorithm works

3. It was then suggested that the code would produce an error for some inputs

4. Subject was suggested to try a sentence like “hello world”

5. Backstop error appeared

Problem Code

// loop through and increment the values for each charfor (int i = 0; i < s.length(); i++){ // get the character at index i char c = s.charAt(i); // increment the value in the array of occurrences // figure out its index by subtracting 'a' from it occur[(int)(c - 'a')]++;}

************************************************************************** ERROR ***************************************************************************An error occurred at line 30 of StringCount.java in themethod "main". I'm sorry, the program cannot continue.

The line of code that caused the error is occur[(int)(c - 'a')]++;

It seems that the code tried to use an illegal value asan index in an array.

The code was trying to access an element at index -65of the array called "occur".

The expression "(int)(c - 'a')" had the value -65 whenthe error occurred.

Remember, you cannot have a negative index. Be surethat the index is always positive.

This error is called an ArrayIndexOutOfBoundsException.The full error is as follows:java.lang.ArrayIndexOutOfBoundsException: -65 at StringCount.main(StringCount.java:30)

Questions Was it more useful than the stacktrace?

All said “yes” Did the output mislead you?

Two said “yes”; one admitted he didn’t read it Describe the tone (vs. stack trace)

Six said it was “friendly”Four said “too long”Two said “too pedantic for a general audience”Three did not notice it

Observations Most common complaint was that the

output is too longOne possibility is giving the option to see

more details or user-adjusted verbosity levels

Average time to complete the task with Backstop was around 5.8 minutesUnderstanding the cause took 1-2 minutes

General Debugging Support Display each line of code as it gets executed

File nameLine numberSource code

Also display the values of any variables

Output at runtime

System.out.print("\nFib.java:27: sum = x + y;");System.out.print(" (x was " + x);System.out.print(" , y was" + y );System.out.print(")"); sum = x + y;System.out.print(" (sum is now " + sum + ")");

System.out.print("\nFib.java:29: y = x;");System.out.print("(x was " + x);System.out.print(")");y = x;System.out.print(" (y is now " + y + ")");

Fib.java:27: sum = x + y; (x was 2 , y was 1) (sum is now 3)Fib.java:29: y = x; (x was 2) (y is now 2)

Modified source code

Original source code

sum = x + y; y = x;

Everything on one line to retain original

line number

Verb tense indicates change of variable

values

Line number printed before code is executed, in case of exception

StringCount.java:27: char c = s.charAt(i); (i was 0) (c is now h)StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1)StringCount.java:27: char c = s.charAt(i); (i was 1) (c is now e)StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1)StringCount.java:27: char c = s.charAt(i); (i was 2) (c is now l)StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1)StringCount.java:27: char c = s.charAt(i); (i was 3) (c is now l)StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 2)StringCount.java:27: char c = s.charAt(i); (i was 4) (c is now o)StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now 1)StringCount.java:27: char c = s.charAt(i); (i was 5) (c is now !)StringCount.java:30: occur[(int)(c - 'a')]++;********************************************************************************************************** ERROR **********************************************************************************************************An error occurred at line 30 of StringCount.java in the method "main".I'm sorry, the program cannot continue.

The line of code that caused the error is occur[(int)(c - 'a')]++;

It seems that the code tried to use an illegal value as an index to an array.

The code was trying to access an element at index -64 of the array called "occur".The variable "(int)(c - 'a')" had the value -64 when the error occured.Remember, you cannot have a negative index. Be sure that the index is always positive.

This error is called an ArrayIndexOutOfBoundsException.

The full error is as follows:java.lang.ArrayIndexOutOfBoundsException: -64

at StringCount.main(StringCount.java:30)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Questions Was it helpful?

All but one said “yes”

Did the output mislead you? Five said “yes”, but four said “only at first”

Describe the appearance Five said more spacing would make it easier to read Four said more spacing would make it harder to read Three said the spacing would make no difference

Evaluation Backstop helped students identify and fix

errors in code they did not write

Backstop automates/mimics the way that many CS1 students debug a Java program on the command line

Still need a study involving the use of both parts of Backstop to debug a runtime error

Conclusion and Future Work Our contribution: Backstop

Makes the error messages produced by uncaught Java exceptions easier to understand (compared to the Java stacktrace)

Provides friendlier and more useful information about how to fix the error

Automates simple debugging approach

Future work could include integration with an IDE, and further studies to measure Backstop’s effectiveness

Backstop: A Tool for Debugging Runtime Errors

Chris Murphy, Eunhee Kim, Gail Kaiser, Adam CannonColumbia University

http://www.cs.columbia.edu/~cmurphy/backstopcmurphy@cs.columbia.edu