Software Quality: Testing and Verification II

Software Quality:Testing and Verification II

2

1.A failure is an unacceptable behaviour exhibited by a system

— The frequency of failures measures software reliability Low failure rate = high reliability

— Failures result from violation of a requirement

2.A defect is a flaw that contributes to a failure— It might take several defects to cause one

failure

3.An error is a software decision that leads to a defect

Software Flaws are identified at three levels

3

Eliminating Failures: Testing vs Verification

Testing = running the program with a set of inputs to gain confidence that the software has few defectsGoal: reduce the frequency of failures When done: after the programming is completeMethodology: develop test cases; run the program with

each test caseVerification = formally proving that the software has no defects

(in this case, the program is said to be “correct”)Goal: eliminate failuresWhen done: before, during and after the programming is

completeMethodology: write separate specifications for the

code; prove that the code and the specifications are mathematically equivalent

4

Program Correctness

The correctness of a program is based on a specific standard.

That standard is called a specification.

int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}

E.g., a specification for the above program might be that it “finds the maximum value of any two integers.”

5

Formalizing a Specification

A Formal specification is written as a logical expression called an assertion.

An assertion describes the state of the program’s variables.Two key assertions are the program’s precondition and its

postcondition.

int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}

A domain is a set of values over which a variable is well defined.

The primitive types (int, float, boolean, etc.) and standard Java classes (String, Vector, HashMap, etc.) provide domains for reasoning about programs.

Postcondition Q = what’s ensured tobe true when the program finishes

Precondition P = what’s requiredfor the program to begin its run.

6

Pre- and Postconditions

Postconditions describes what it will compute. For max, a postcondition is Q = m = max(a, b), where max is a mathematical function that delivers the larger of two integers.

Preconditions describe minimum requirements for the program to run.For max, a and b can be any integers, so the precondition is P = true.

Before proving a program’s correctness, we first write its specifications:{true}int max (int a, int b) { int m; if (a >= b) m = a; else m = b; return m;}

{m = max(a, b)}

7

Another Example – computing a factorial (n!)

{n > 1} int Factorial (int n) { int f = 1; int i = 1; while (i < n) { i = i + 1; f = f * i; } return f;}

{f = n!}

This example raises two issues. What happens if:1. the program has a loop that never terminates?2. The program terminates abnormally (e.g., an exception is raised)?

Precondition P

Postcondition Q

8

Defining away the problem: “partial correctness”

Let’s assume that nothing bad ever happens…• programs always terminate after a finite number of steps, and

• termination is always normal.

Then such a program is partially correct if, for every set of input values that satisfies precondition P, the program computes a result that satisfies postcondition Q.

E.g., Factorial is partially correct if for every value of n that satisfies n > 1, it computes f =n!

9

(Partial) Correctness Proofs

Let’s think generally about any program or sequence of statements s, whose pre- and postconditions are P and Q.

A “Hoare triple” is a predicate of the form {P} s {Q}

which asserts that “execution of statements s, beginning in a state that satisfies P, results in a state that satisfies Q.”

If we can prove that this Hoare triple is valid, (i.e., it is true for all assignments of values to variables in P, Q, and s) then the program s is said to be correct.

But how can we construct such a proof?

10

Constructing a Correctness Proof

If the program is a series of statements: s1; s2; …; sn

We start with the Hoare triple {P} s1; s2; …; sn {Q}, and • use inference rules for programs to • derive an intermediate triple {Pi} si {Pi+1} for every statement si in this sequence.

When done, we also ensure that

Note: This process is similar to a direct proof in logic, where we• use inference rules to• derive a series of assertions that logically link the premises to the conclusion.

So, what are the inference rules for programs?

€

P ⇒ P1 and Pn+1⇒ Q.

11

Inference Rules are tied to program statement types

€

12

Formal methods and real programs

Major question: Where do formal methods fit in the software process? That is,

1. How do we integrate them into object oriented programs?

2. How do we write pre- and postconditions P and Q for

– methods?– classes?– systems?

3. Once written, how are these used to ensure that software is correct?

13

Recall the spiral model for software design

Requirements

Specifications(use cases)

Design

Coding/Integration

Testing andVerification

Deployment (v 1.0)Review

Prototype

v 1.1v 1.0

14

Formal methods in the object oriented design process

In OO design, we focus on classes and objectsMethods and messages are subordinate

Collections of objects have state, which is the set of all active objects and the values of their variables at any moment of run time.

Formal specifications P and Q are therefore logical expressions about each object’s state.

Tools for the formal design process• Specifications : Java Modeling Language (JML)• Design: Unified Modeling Language (UML and JML)

• Coding: Java and JML• Verification: JML

15

Correctness of OO systems

Where?• Method level: pre- and post-conditions, loop invariants

• Class level: class invariant (class state)• System level: intra-class invariants (system state)

When?• Specification and design phases:

Write specifications for all classes and methods (UML/JML)

• Coding phase:Develop code from the specifications (UML/JML/Java)

• Validation phase:Prove (mathematically) that specifications and code are logically equivalent (JML <==> Java)

16

What is JML? (www.jmlspecs.org)

History– Emerged in early 2000s out of ESC/Java2

Goals— Integration of formal methods throughout the software process

— Formal specification accessible to programmers— Direct support for design by contract— Integration with a real language (Java)

JML allows us to mix specifications directly with the Java code— Preconditions— Postconditions— Loop invariants— Class invariants

17

JML Basics

JML specifications are special comments in a Java program:

//@ for one-liners

/*@ …. @*/ for multiple-liners

The Hoare triple

{P} s1; s2; …; sn {Q}

is written in JML/Java as

(P and Q are written as Javaboolean expressions, and useparameters, local, and classvariables as arguments.)

/*@ requires P ; ensures Q ;@*/type method (parameters) { local variables s1; s2; …; sn

}

18

JML Language Summary

Note: p and e are also good old fashioned Java boolean expressions, possibly augmented by JML-specific operations.

19

Here’s a simple example, first as a Hoare triple:

{n > 1} P (precondition)int Factorial (int n) { int f = 1; int i = 1;

{1 < i i < n f = i!} R (loop invariant) while (i < n) { i = i + 1; f = f * i; } return f;}

{f = n!} Q (postcondition)

20

And again with JML expressions for P, Q, and R:

/*@ requires 1 <= n ;

ensures \result == (\product int i; 1<=i && i<=n; i) ;

@*/

static int Factorial (int n) {

int f = 1;

int i = 1;

/*@ loop_invariant i <= n && f == (\product int j; 1 <= j && j <= i; j);

@*/

while (i < n) {

i = i + 1;

f = f * i;

}

return f;

}

P Q R

21

JML-based software tools (1 and 3 are built into Eclipse)

1. Compiling (use jmlc instead of javac)• Does syntactic and type checking, and byte code generation for all JML assertions and Java code

2. Static checking (ESC/Java2)3. Runtime assertion checking (use jmlrac instead of java)

• Checks truth of precondition P at entry to every call• Checks truth of postcondition Q at exit from every call• Checks truth of loop invariant R before every iteration• Issues a Java Exception when any of these is not true

Note: this is not formal verification (Checking truth for one instance of a call is not the same as checking truth for all instances. The latter is proof of correctness!)

4. Proof assistance tools (Daikon, LOOP)

22

JML Eclipse Environment

3. Runtime assertion checking

1. Compiling

23

Continuing our JML example, let’s wrap Factorial inside a simple class:

public class myFactorial {/*@ requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; i); @*/static int Factorial (int n) {…}public static void main(String[] args) { int n = Integer.parseInt(args[0]); System.out.println("Factorial of " + n + " = " +

Factorial(n));}}

24

… compile it with jmlc, and run it twice with jmlrac:

% jmlc -Q myFactorial.java

% jmlrac myFactorial 3

Factorial of 3 = 6

% jmlrac myFactorial -5

Exception in thread "main”

org.jmlspecs.jmlrac.runtime.JMLEntryPreconditionError:

by method myFactorial.Factorial regarding specifications at

File "myFactorial.java", line 3, character 15 when

'n' is -5

at myFactorial.checkPre$Factorial$myFactorial(myFactorial.java:240)

at myFactorial.Factorial(myFactorial.java:382)

at myFactorial.main(myFactorial.java:24)

normal run

abnormal run(throws a JML exception)

25

JML Exceptions

26

JML helps identify errors

Example 1: Suppose we change the while loop from while (i < n)

to while (i <= n) so that n! will be computed

incorrectly. Here’s the result:

% jmlrac myFactorial 3Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 9, character 24 when 'n' is 3 at myFactorial.internal$Factorial(myFactorial.java:102) at myFactorial.Factorial(myFactorial.java:575) at myFactorial.main(myFactorial.java:211)

invariant not satisfied

27

JML Example 2

Suppose we change the while loop from while (i < n) to while (i <=n )

and also remove the JML loop invariant. Now we get:

% jmlrac myFactorial 3Exception in thread "main" org.jmlspecs.jmlrac.runtime.JMLNormalPostconditionError: by method myFactorial.Factorial regarding specifications at File "myFactorial.java", line 4, character 23 when 'n' is 3 '\result' is 24 at myFactorial.checkPost$Factorial$myFactorial(myFactorial.java:321) at myFactorial.Factorial(myFactorial.java:392) at myFactorial.main(myFactorial.java:24)

postcondition not satisfied

28

JML Example 3

Disagreement between a JML specification and a program may signal an error in the specification.

E.g., if the loop invariant had specified j <= i rather than j < i the following outcome would occur:

% jmlrac myFactorial 3

Exception in thread "main"

org.jmlspecs.jmlrac.runtime.JMLLoopInvariantError: LOOP INVARIANT:

by method myFactorial.Factorial regarding specifications at

File "myFactorial.java", line 9, character 24 when

'n' is 3

at myFactorial.internal$Factorial(myFactorial.java:101)

at myFactorial.Factorial(myFactorial.java:573)

at myFactorial.main(myFactorial.java:209)

invariant not satisfied

29

But beware… JML is no panacea

jmlrac doesn’t trap all errors… here are two “normal” runs:

% jmlrac myFactorial 21Factorial of 21 = -1195114496% jmlrac myFactorial 32Factorial of 32 = -2147483648

Recall: (1) Java has no ArithmeticOverflow exception, but (2) Factorial(n) for n > 12 should give a result > 231-1

Note: jmlrac computes the same wrong result when it checks the postcondition as the Factorial method computes, so that this error goes undetected.

Conclusion: the program and its specifications are both wrong.

wrong results!

30

Exception Handling in JML

/*@ requires P ; ensures Q ; signals (exception) expression;@*/type method (parameters) { locals s1; s2; …; sn

(includes “throw new exception ;”)}

We can throw Java Exceptions, and then validate their circumstances in JML whenever they occur.

whenever this happens

JML executes this

Two outcomes: 1) expression is true and normal Java exception handling proceeds, or 2) expression is false and JMLExceptionalPostconditionError is reported.

31

JML Example 4: Throwing and Checking Exceptions

/*@ requires 1 <= n; ensures \result == (\product int i; 1<=i && i<=n; i); signals (ArithmeticException) n > 12; @*/ static int Factorial (int n) { if (n > 12) throw new ArithmeticException(); else { …

% jmlrac myFactorial 22Exception in thread "main" java.lang.ArithmeticException at myFactorial.internal$Factorial(myFactorial.java:9) at myFactorial.Factorial(myFactorial.java:610) at myFactorial.main(myFactorial.java:213)

Normal Java exceptionhandling occurs, sincen > 12 is true.

32

Additional Points about JML

1. We can sometimes avoid a signals clause by strengthening the precondition. E.g., for Factorial, we could have said:

requires 1 <= n && n < 13 ;2. Specifications are always declarative; they never

affect the state of the program.3. Runtime assertion checking is not proof, but it

does provide a rigorous framework for debugging.4. JML provides a language in which formal methods

and Java programs can be integrated.5. There’s a lot more to JML. We are particularly

interested in:a. class level specifications?b. Tools for static checking of specifications?c. Tools for proving correctness?

33

Readings (at the course web site)

Hoare, An axiomatic basis for computer programming, Communications of the ACM 12(10):576-580.

Leino, Hoare-Style Program Verification ILeino, Hoare-Style Program Verification IIGordon, Specification and Verification I Dwyer et al., Software SpecificationsLeavens and Cheon, Design by Contract with JMLPoll, Kiniry, and Cok, Introduction to JMLBurdy et al., Overview of JML ToolsJML Reference Manual