Hampshire BCS – February 2002 1 Logic vs. Magic in Critical Systems Ian O’Neill why wizards are not enough or :

Hampshire BCS – February 2002 1

Logic vs. Magic in Critical Systems

Ian O’Neill

why wizards are not enoughor:


Agenda

• Introduction: a historical magic

• The software process

• What are “Critical Systems”?

• The need to “reason”

• Some obstacles

• Advantages of logic — project experience

• Summary


A Historical Magic

• Go to statement considered harmful

• Structured programming

• “De-spag” tools


Add up the first N (positive) values of the array

A Specification

traverse the array until N positive values have been found and sum them

traverse N elements of the array and sum any positive ones (ignoring any negative ones) ?

or


function Sum(A : Atype; Count : Aindex) return Integer is Accumulator, Value : Integer; Index : Aindex;

begin Accumulator := 0; Index := 1; <<Label1>> Value := A(Index); if Value <= 0 then goto Label2; end if; Accumulator := Accumulator + Value;

<<Label2>> if Index = Count then goto Label3; end if; Index := Index + 1; goto Label1;

<<Label3>> return Accumulator; end Sum;


function Sum(A : Atype; Count : Aindex) return Integeris IC : Counter; Accumulator, Value : Integer; Index : Aindex;begin IC := 1; Accumulator := 0; Index := 1; while IC < 6 loop case IC is when 1 => Value := A(Index); IC := 2; when 2 => if Value <= 0 then IC := 4; else IC := 3; end if; when 3 => Accumulator := Accumulator + Value; IC := 4; when 4 => if Index = Count then IC := 6; else IC := 5; end if; when 5 => Index := Index + 1; IC := 1; when 6 => Null; end case; end loop; return Accumulator;end Sum;


Engineering:

The use of science and mathematics to solve practical problems


Agenda





• Some obstacles


• Summary


The software process

• Progressive refinement:

– a need

– requirements

– specification

– design

– source code

– object code




– a need

– requirements

– specification

– design

– source code

– object code




– a need

– requirements

– specification

– design

– source code

– object code




– a need

– requirements

– specification

– design

– source code

– object code


Add up the first N (positive) values of the array

NZ seq:nonNegSum

ssxsx

sxsxsx

nonNegSum

nonNegSumnonNegSumseq:;\:

nonNegSumnonNegSumseq:;:

0

^

^

ZNZ

ZN

snnsns

)..1nonNegSum(sum:;seq:

seq:sum

ZZ

NZZ

A Specification




– a need

– requirements

– specification

– design

– source code

– object code




– a need

– requirements

– specification

– design

– source code

– object code


Language Insecurity

a tiny example

procedure Init2(X, Y : out Integer)

is

begin

X := 1;

Y := 2;

end Init2;

What is the meaning of:

Init2(A, A);




– a need

– requirements

– specification

– design

– source code

– object code


Formality at last!

1010011010010000100010001001000110101000111101010100001010101010101010101001010010101010100001010101001001010100110101010101010100000101000101010010101001010010010101010101010101010111101001010010101010101010010.......


A typical development process

Informal, opaque black box

Precise object code


Agenda





• Some obstacles


• Summary


What are critical systems?

• Some important objectives:

– efficiency

– cost

– time to market

– functionality


What are critical systems?

• Systems where reliability is more important than:

– efficiency

– cost

– time to market

– functionality

Robert Dewar - Ada Core Technologies


Contributors to Lifecycle Costs

0102030405060708090

100

design code integrate IV&V

Cost of error correction

<50% >50%Time spent: <20% >80%


Producing Safety-Critical Software

• Not just a question of “being more careful”

• The need to be able to show, before there is any service experience, that a system will be safe enough requires a qualitatively different approach

• We can only achieve this by logical reasoning.


Agenda





• Some obstacles


• Summary


Foundations of Reasoning

• We reason about information

• But what about “information hiding”?

• We need to hide detail not information

• We do this through abstraction

• We can’t just magic complexity away

Example: Program “State”


Swap Algorithm

procedure Swap(X, Y : in out T) is

begin

Temp := X;

X := Y;

Y := Temp;

end Swap;

Temp : T;


Swap Algorithm


begin

Temp := X;

X := Y;

Y := Temp;

end Swap;

Temp : T;


A Store Object

package Storeis

procedure Put(X : in T);

function Get return T;

end Store;


Swap Algorithm


begin

Store.Put(X);

X := Y;

Y := Store.Get;

end Swap;


Store

Swap

Heap


Store

Swap ????


• The location of state is the single biggest influence on coupling and cohesion.

• It is probably the most important design decision we must make

yet

• OOP regards it as an “implementation detail”; and

• UML does not even have a notation to express it!


OOP - The Biggest Magic of them All

• We hide all state• We disguise the hierarchical relationships between

objects• We hide the control flow by use of messages and

dynamic dispatch• In extreme cases, we even deny there is a software

design process at all (Shlaer Mellor)

Of course all this hiding makes things easier - at least until all the hidden complexity emerges during integration testing!


An OOP Array Sum

State HandlerTotal Handler

Count Handler

Data Handler



“There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.”

Professor C.A.R. HoareThe 1980 Turing award lecture


Agenda





• Some obstacles


• Summary


Some obstacles

• Fashion• Low expectations• Poor contract writing• The difficulty of saying “no”• Gurus• Belief that:

– Software engineering is the same as knowing tool or product X

– Knowing tool or product X is more important than domain or engineering knowledge


Fashion

• We must use X, everyone else is• The staff want to use X, we won’t be able to recruit or

keep them if we don’t• All the tool vendors are supporting X• Y, which we know is better, “is dead”• Progress is the same as the number of lines of code

written• The most important thing is to get to test quickly


Low Expectations

• Software is always buggy• Even Microsoft “the best software company in the

world” produces buggy software• Vendors never offer warranties• Since software is always a load of rubbish we might

as well buy the cheapest load of rubbish




Poor Contract Writing

• Fixed-price contracting

– sets producer and customer in opposition– even mutually beneficial changes are difficult

• Time and material contracting

– removes any incentive to do a more efficient job


The Difficulty of Saying No

• Engineers have a duty to say “No” where safety is being compromised; however,

• There is often someone else who can be persuaded to say “Yes” (and charge you less!)

• The Chinook FADEC saga:– the safety assessors have been cast as the villains

unreasonably holding up entry to service– the software is being treated as safe because it is

so badly written it is not possible to prove that it’s dangerous


Gurus

• Close associates of wizards and magicians• Know all 123 operator precedence levels of C• Don’t make the kinds of mistakes that mere mortals

make so don’t see the benefit of techniques that help avoid them

• Have a strong incentive to maintain the status quo because they fear losing their guru status

• Don’t often make good engineers


Misunderstanding the Nature of

Knowledge• C programmers are easy to hire so we should use C

– riveters are easy to hire but it hasn’t stopped aircraft companies adopting composites

• Learning a new tool or programming language is hard but learning a new problem domain is easy – so people experienced in banking systems should

write flight control systems• We should use techniques that are widely known

rather than ones that are appropriate– like driving Ford Mondeos to the moon!


Agenda





• Some obstacles


• Summary


Contributors to Lifecycle Costs

0102030405060708090

100

design code integrate IV&V

Cost of error correction

<50% >50%Time spent:


The Need to Reason

• To bring error detection forward we must be able to

reason about source code

• We can’t reason about source code unless it has a

precise meaning

• Too often the meaning of code is defined by its test

results!


“… one could communicate with these machines in any language provided it was an exact language …”

“… the system should resemble normal mathematical procedure closely, but at the same time should be as unambiguous as possible.”

Alan Turing, 1948


What is SPARK?

• A sub-language of Ada with particular properties that make it ideally suited to the most critical of applications:– Completely unambiguous– All rule violations are detectable– Formally defined– Tool supported

• SPARK facilitates Correctness by Construction


A disclaimer

• SPARK is not the world’s only logic

• SPARK is not a superior magic


A Store Object

package Store

is procedure Put(X : in T);

function Get return T; end Store;


A Store Object

package Store--# own State;is procedure Put(X : in T); --# global out State;

function Get return T; --# global State;end Store;


procedure Swap(X, Y : in out T) --# global out Store.State;isbegin Store.Put(X); X := Y; Y := Store.Get;end Swap;


SHOLIS - Ship Helicopter Operating

Limits Instrumentation System

• MOD’s first Def Stan 00-55 project

• 27,000 lines of SIL4 code

• Z specification

• SPARK code and formal proof

• Information flow analysis used to show separation of critical and non-critical functions

• Zero defects in acceptance and trials



Defence Standard 00-55: Example

Requirements

• Formal specification– must be possible to verify formally that S/W meets spec

• Formal design– small on SHOLIS: most code proved directly against spec

• Proof obligations at all stages– e.g. formal design against spec, code against formal design

• Resource modelling– to show S/W meets spec of available hardware resources

• Static analysis of code– including: control, data and information flow analysis


Z Proof

• Consistency of global variables and constants• Existence of initial states• Checking of preconditions

– above proofs show specification is well-defined• Key safety properties

– e.g. In ; Calc ; Out gives the correct warning• Proofs constructed by “rigorous argument”

– largely manual (tool for mechanical schema expansion etc.)

– arguments contain the main proof steps, but omit detailed symbol manipulation a tool requires for complete proofs


SHOLIS on test...


Code Proof

• Full partial correctness - formally specified “SIL4” units

• Proof of safety invariants at “main loop” level

• Freedom from exceptions - all code

• 9000 VCs from code– 3100 for partial correctness and safety invariants– 5900 for exception freedom

• 6800 discharged by Simplifier


Faults found vs. Effort

0

10

20

30

40

50

60

70

80

90

100

Sp

ecif

icat

ion

Z P

roo

f

Hig

h-l

evel

Des

ign

Co

de

Un

it T

est

Inte

gra

tio

n T

est

Co

de

Pro

of

Sy

stem

Val

idat

ion

Acc

epta

nce

Oth

er

No

. o

f F

au

lts

Fo

un

d

0

0.1

0.2

0.3

0.4

0.5

0.6

Eff

icie

ncy


Lockheed C130J


C130J• 130,000 lines of safety related code in mission computer• Process designed to

– reduce V&V costs (and consequent delays)– meet certification requirements, UK MoD and FAA

• Based on rigorous specification and design– CoRE– Parnas tables– SPARK

• Lockheed benefits– ‘Correctness by construction’– Errors trapped early in process– V & V during coding– Accelerated validation & verification– Cost reduction


Lockheed on SPARK• Some errors immediately uncovered by formal analysis, such as conditional

initialization errors may only emerge after very extensive testing.

• The technology for generating and discharging the proof obligations, based on the SPARK components of Ada, was crucial, in binding the code to the initial requirements.

• SPARK provides an extremely robust and efficient basis for formal verification.

• The process has proven effective with typical software developers and did not necessitate and inordinate amount of additional training.

• Experience has shown that SPARK coding occurs at near typical Ada rates.

• Code written in SPARK is deterministic and inherently statically analysable.

• Very few errors have been found in the software during even the most rigorous levels of FAA testing, which is being successfully conducted for less than a fifth of the normal cost in industry.

• Correctness by construction is no longer a theoretical abstraction; it is now a practical way to develop software that exceeds its technical goals while delivering sterling business performance.


Lockheed on SPARK• Some errors immediately uncovered by formal analysis, such as conditional

initialization errors may only emerge after very extensive testing.

• The technology for generating and discharging the proof obligations, based on the SPARK components of Ada, was crucial, in binding the code to the initial requirements.

• SPARK provides an extremely robust and efficient basis for formal verification.

• The process has proven effective with typical software developers and did not necessitate and inordinate amount of additional training.

• Experience has shown that SPARK coding occurs at near typical Ada rates.

• Code written in SPARK is deterministic and inherently statically analysable.

• Very few errors have been found in the software during even the most rigorous levels of FAA testing, which is being successfully conducted for less than a fifth of the normal cost in industry.

• Correctness by construction is no longer a theoretical abstraction; it is now a practical way to develop software that exceeds its technical goals while delivering sterling business performance.

Very few errors have been found in the software during even the most rigorous levels of FAA testing, which is being successfully conducted for less than a fifth of the normal cost in industry.


Aerosystems’ IV&V Conclusions

• Significant, safety-critical errors were found by static analysis in code developed to DO-178B Level A

• Proof of SPARK code was shown to be cheaper than other forms of semantic analysis performed

• SPARK code was found to have only 10% of the residual errors of full Ada and Ada was found to have only 10% of the residual errors of C

• No statistically significant difference in residual error rate could be found between DO-178B Level A and Level B code


Agenda





• Some obstacles


• Summary


“Any sufficiently advanced technology is indistinguishable from magic."

Arthur C. Clarke, 1962

But this doesn’t mean that all magical illusions are underpinned by advanced technology!


Logic vs Magic

• Magic is attractive because life would be so much easier if it worked!

• Magic is transient, fashion driven and ultimately futile• Software engineering is hard because we are trying

to solve hard problems• The engineering response to solving hard problems is

to use clever people and good mathematics• Being the best screwdriver user in the world might

make you a mechanic but not an engineer!


and finally

“Real life problems are those that remain after you have systematically failed to apply all the known solutions”

Edsger Dijkstra, 1973

Documents

Hampshire BCS – February 2002 1 Logic vs. Magic in Critical Systems Ian O’Neill why wizards are not enough or :