Notes on: Is Proof More Cost-Effective Than Testing?

by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor

Prepared by

Stephen M. Thebaut, Ph.D.University of Florida

UK Defense Standard 00-55

• For procurement of safety critical software in defense equipment

• Perceived emphasis on formal methods (formal specification and design, proofs of correctness)

• Purpose of this paper: report on work showing that using formal methods on a large scale is practical and effective at both the specification and code level

SHOLIS

• Ship Helicopter Operating Limits Information System – safety critical system aiding safe operation of helicopters on naval vessels

• Z-based formal specification and SPARK programming language (subset of Ada)– Omitted features: gotos, aliasing, default parameters

for subprograms, side-effects in functions, recursion, tasks, user-defined exceptions, exception handlers, & generics

– simplification of other features

Proofs

• Specification-Based– Approx. 150 proofs in 500 pages– Consistency of global variables and constants,

existence of initial states and checking of preconditions

• Code-Based– Approx. 9,000 verification conditions verified– Predicate transforms used for non-looping programs– Invariants used to prove partial (weak) correctness of

looping programs + separate arguments for termination

Some Observations

• Skills needed: knowledge of Z, proof by cases and contradiction, understanding of imperative programming constructs, familiarity with proof concepts such as loop invariants (not that much!)

• Proof validation: formal peer + IV&V reviews; proofs were not inspected by the customer

• Z proof appeared to be substantially more efficient at finding faults than the most efficient testing phase.

Some Observations (cont’d)

• Code proofs appeared to be more efficient than unit testing, despite the fact that substantial amounts of unit testing were completed before the bulk of code proof started.

Specification using “Z”

• Z is a mature and expressive notation for model-based specification. It combines formal and informal descriptions and incorporates graphical highlighting.

• The basic building blocks of Z-based specifications are schemas.

• Schemas identify state variables and define constraints and operations in terms of those variables.

Schemas

• Schemas are comprised of three parts: a name, a signature, and a predicate.

– The signature declares (state) entities introduced in a schema and their type.

– Signature declarations are given in the form identifier: type.

– The predicate defines logical relationships between entities in the declaration.

• Z is based on the predicate calculus + typed sets.

Graphical Layout of a Schema

Schema Name

Declarations

Predicates

Signature

Predicate

Schemas (Cont.)

• Schemas describe state entities or operations on state entities.

– To describe state entities, declared variables comprise the entities while predicates specify their invariant properties.

– To describe operations, declarations consist of initial and final state variables, inputs, and outputs, while predicates specify the relations among them.

Example 1

Container

contents: Ncapacity: N

contents capacity

Specification of a generic container entity:modeled as natural numbers

invariant property

Example 1 (Cont.)

Indicatorlight: {off, on}reading: Ndanger_level: N

light = on reading danger_level

Specification of a generic indicator entity:

Example 1 (Cont.)

Storage_tankContainerIndicator

reading = contentscapacity = 500danger_level = 50

Specification of a storage tank entity:

Predicates on separate lines are separated by an implicit “AND”.

Example 1 (Cont.)

Storage_tankcontents: Ncapacity: Nlight: {off, on}reading: Ndanger_level: N

contents capacitylight = on reading danger_levelreading = contentscapacity = 500danger_level = 50

Expanded specification of a storage tank entity:

Example 1 (Cont.)

Fill-OK Storage-tankamount?: N

contents + amount? capacitycontents’ = contents + amount?

Specification of a nominal storage tank fill operation:

“dash” indicates the value of contents AFTER the operation

“?” indicates that amount is an INPUT.

“Delta” indicates that values of one or more variables will be changed.

Example 1 (Cont.)

OverFill Storage-tankamount?: Nr!: seq CHAR

capacity < contents + amount?r! = “Insufficient tank capacity – Fill cancelled”

Specification of an exceptional storage tank fill operation:

“Xi” indicates that NO variables will be changed by the operation.

“!” indicates that r is an OUTPUT.

Example 1 (Cont.)

Fill

Fill-OK V OverFill

Complete specification of the storage tank fill operation:

Key Points

• Model-based specification relies on a state model of the system using mathematical entities such as the predicate calculus and typed sets.

• Functions / operations may be specified by defining their effect on system state.

• It is normal to specify functions / operations incrementally and then combine the fragments to produce a complete specification.

Key Points (Cont.)

• Z specifications are comprised of a number of schemas that introduce typed names and define predicates over those names. They are distinguished from surrounding text by graphical highlighting.

• Schemas are building blocks which may be combined and used in other schemas. The effect of including schema A in schema B is that schema B inherits the names and predicates of schema A.

Notes on: Is Proof More Cost-Effective Than Testing?

by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor

Prepared by

Stephen M. Thebaut, Ph.D.University of Florida