Safety as a Software Metric

Matthias Felleisen and Robert Corky CartwrightRice University

Why Safety as a Metric?

• Measuring Software: Syntax versus Semantics

• What is Programming Language Safety ?

• What Makes an Individual Program Safe ?

• How about Teaching Program Safety?

Why Measure Software?

• correct and efficient software

• maintainable software

• extensible software

What do Metrics Measure?

• lines of code

• number of procedures, gotos, loops, modules, statements versus expressions, …

• in short: Syntactic Attributes of software

What should Metrics Measure?

• correctness

• extensibility

• maintainability

• in short: semantic and organizational attributes

Measuring Correctness is Difficult

• …

• goal: measure certain aspects of correctness

• specifically: assume the programming language is safe, what kind of problems can we predict?

Safe Programming Languages

Safety -- A High-Level View (1)

“Close the valve by 10 degrees!” “Turned the valve by

10 degrees!”

Safety -- A High-Level View (2)

“Close the valve by 10 degrees!” “Turned the valve by

15 degrees!”

Safety -- A High-Level View (3)

“Close the valve by 10 degrees!”

“OUCH!”

Safety -- A High-Level View (4)

----------------------------------------------------

Safety -- A High-Level View (5)

----------------------------------------------------

ERROR!

C and C++ are NOT Safe!

int f(int n, int m) { int r = n % m;

if (0 == r) return m; else return f(m,r); }

main() { char a = 'a'; char b = 'b'; int mn[2] = {24,6}; char c = 'c'; char d = 'd';

printf("%d\n",f(mn[0],mn[1])); printf("%d\n",f(mn[0],c)); printf("%d\n",f(mn[0],mn[2])); }

Safety in Programming Languages

• a safe language protects every computational primitive, e.g., +, *, if, vector-lookup, record dereference, …

• protection is implemented with a mixture of compile-time and run-time checks

• safety guarantees errors are caught

• safety greatly increases effectiveness of debugging

Safety

• … is NOT just TYPE checking!

Examples

• Fortran

• C

• C++

• Perl

• ML

• Eiffel

• Java

• Scheme (untyped, but safe)

UNSAFE Languages SAFE Languages

Safe Programs and Measuring Safety

Measuring the Safety of Programs

• programs in safe languages signal errors

• programs should not signal errors

• determine whether any computational primitive might signal an error

• make programmers explain potential faults

MrSpidey: Measuring the Safety of Scheme Programs

• Scheme is a dialect of Algol and LISP

• lexical scope, first-class functions (“mini-objects”)

• LISP’s syntax (parentheses) and primitives (cons, car, and cdr)

some function call, somewhere in the program

SYMBOLS are bad for +

general input shapes

Measuring Safety is More than Checking Types

• check general “data shapes”

• lists with at least N items

• vector references

• …

list with at least one NUMBER

NIL is not okay

An Elaborate Example from the Scheme Front-end

S-expression S-expression

(let (<var> <rhs:exp>) <body:exp>)

((lambda (<var>) <body:exp>) <rhs:exp>)

weak invariant

… yields many checks

stronger invariant yields stronger results

Teaching with Safety Metrics

Program Construction: Rice University, Fall 1998

• course on program safety

• understanding

• measuring

• based on Scheme and Java

On Safety of Languages and Programs

• programming language safety

• program safety

• theory and tools for “measuring” program safety– logics that conservatively approximate

semantics– logics that extend the logic of type checking

The Pragmatics of MrSpidey

• using MrSpidey:– checking – understanding potential fault sites:

• data set• data flow

– is it a problem with the program? – is it a problem with the theory/tool?– if the latter, can a re-organization help?

Hands-on Work

• homework assignments– sets of problems for each bullet– increasing complexity – theory and practice

• project: implement sequential subset of Java– modules and data invariants that cross

boundaries– exploring large pieces of code

Evaluation (1)

• course evaluation: excellent

• targeted questions: – understanding of language safety – understanding of program safety – understanding of measuring safety with

theorem provers– effectiveness of homeworks versus project

Evaluation (2)

• Positives: – appreciate safety– appreciate tools – appreciate theory– understand the above

based on homework

• Negatives– project too large

Summary

• new, semantics-based thinking about “metrics”

• extensions: measuring stronger invariants (numeric constraints, polyvariant); measuring organization (patterns?)

• teaching: a good approach to have students understand partial correctness

Thank You

Matthew Flatt

Shriram Krishnamurthi

Robby Findler

Mike Fagan (92)

Andrew Wright (94)

Cormac Flanagan (96)