1 Formal Methods Quick Tutorial Ricky W. Butler Oct 22, 2003

1

Formal MethodsQuick Tutorial

Ricky W. Butler

Oct 22, 2003

http://shemesh.larc.nasa.gov/fm

2

Outline

• Motivation Motivation (3 minutes)(3 minutes)

• Formal Methods by way of exampleFormal Methods by way of example

— Theorem Proving Theorem Proving (12 (12 minutes)minutes)

— Model Checking state machines Model Checking state machines (12 (12

minutes)minutes)

• A Few More Thoughts A Few More Thoughts (3 minutes)(3 minutes)

3

“There are no stupid questions”

4

Motivation

Say, I think I see where we went off. Isn’t eight times seven fifty-six?Say, I think I see where we went off. Isn’t eight times seven fifty-six?

5

• Increased Capabilities– Almost any breakthrough in any field will depend on advanced

computer systems– And software is where most of the complexity goes

• Software Costs Dominate– Development costs (More than half of the non-recurring development

of the Boeing 777)– Integration (only way of testing it is all-up simulation)– Certification (based on process, not product)

• Major Safety Concerns– Reliability (dealing with hardware failures)– Correctness (no faults in its own design)– Man-machine interaction (issue in majority of recent crashes)– Certification (complexity forces FAA to rely on DERs)

Software Underpins Everything In Aviation

6

x: VAR nat B: VAR bool f(x,B): nat = IF x > 30 AND B THEN x*x ELSE 90*x ENDIF

g(x): nat = IF x <= 30 THEN 2*x ELSE 0 ENDIF

h(B): nat = IF B THEN 60 ELSE 0 ENDIF

x

B

f(x,B)

g(x)

h(B)

System

safe_prop: THEOREM f(x,B) >= g(x) * h(B)

Example System

7

Design Verification

In the beginning, there was Simulation:

•Build a model•Feed IT inputs•Analyze results

8





Verification by Simulation

x B f g h safe_prop ----------- ---------------------------- -------- 0 T 0 0 60 TRUE 10 F 1200 20 0 TRUE 50 T 2500 0 60 TRUE 30 F 3600 60 0 TRUE 30 T 2700 60 60 FALSE

9

x: VAR nat B: VAR bool f(x,B): nat = IF x > 30 AND B THEN x*x ELSE 90*x ENDIF g(x): nat = IF x <= 30 THEN 2*x ELSE 0 ENDIF h(B): nat = IF B THEN 60 ELSE 0 ENDIF

Formal Verification


IF x > 30 AND B THEN x*x ELSE 90*x ENDIF >= g(x) * h(B)

BNOT B

IF x > 30 THEN x*x ELSE 90*x ENDIF >= g(x) * h(B) 90*x >= g(x) * h(B)

90*x >= g(x) * 0

90*x >= 0

10

x: VAR nat B: VAR bool f(x,B): nat = IF x > 30 AND B THEN x*x ELSE 90*x ENDIF g(x): nat = IF x <= 30 THEN 2*x ELSE 0 ENDIF h(B): nat = IF B THEN 60 ELSE 0 ENDIF

Formal Verification (cont.)

B

IF x > 30 THEN x*x ELSE 90*x ENDIF >= g(x) * h(B)

x > 30 x <= 30

x*x >= g(x) * h(B) 90*x >= g(x) * h(B)

x*x >= 0 * h(B)

x*x >= 0

90*x >= 2*x * h(B)

90*x >= 2*x * 60

90*x >= 120*x

11

demo_incorrect: THEORYBEGIN





END demo_incorrect

safe_prop :

|-------{1} FORALL (B: bool, x: nat): f(x, B) >= g(x) * h(B)

Rule? (grind)

f rewrites f(x, B) to IF x > 30 AND B THEN x * x ELSE 90 * x ENDIFg rewrites g(x) to IF x <= 30 THEN 2 * x ELSE 0 ENDIFh rewrites h(B) to IF B THEN 60 ELSE 0 ENDIFTrying repeated skolemization, instantiation, and if-lifting,this simplifies to: safe_prop :

{-1} x!1 >= 0{-2} B!1 |-------{1} x!1 > 30{2} 90 * x!1 >= 120 * x!1

Rule?

PVS SPECIFICATION OUTPUT FROM PVS THEOREM PROVER

Using a “Mechanical” Proof Checker Using a “Mechanical” Proof Checker (PVS)(PVS)

12

demo_correct: THEORYBEGIN




safe_prop: LEMMA f(x,B) >= g(x) * h(B)

END demo_correct

safe_prop :

|-------{1} FORALL (B: bool, x: nat): f(x, B) >= g(x) * h(B)

Rule? (grind)

f rewrites f(x, B) to IF x > 30 AND B THEN x * x ELSE 90 * x ENDIFg rewrites g(x) to IF x <= 30 THEN 2 * x ELSE 0 ENDIFh rewrites h(B) to IF B THEN 40 ELSE 0 ENDIFTrying repeated skolemization, instantiation, and if-lifting,Q.E.D.

Run time = 0.35 secs.Real time = 3.33 secs.

CORRECTED PVS SPECIFICATION

OUTPUT FROM PVS THEOREM PROVER

Using a “Mechanical” Proof Checker (PVS)Using a “Mechanical” Proof Checker (PVS)

13

The Key Difference

• Simulation/Testing only explores a small part of the state space

• Formal Verification explores the entire state space

The only way you can assure yourself that there are no safety-relevant bugs hidden in some dark corner of your software is to explore the entire state space.

14

Can We Automate Verification?

Program

SafetyProps

Verified!

Buggy!

The Big V or

Counter Example

15

Kurt Goedel (1906-1978)

Answer: NO!

16

• Algebra (in general) is undecidable(i.e. There is no terminating algorithm that can

determine whether a formula is true) But some sub-theories are decidable (e.g.

Presburger arithmetic): develop automated decision procedures

Why Can’t We Automate Verification?

• While loops in a program lead to induction proofs(Discovery of the induction invariant involves creativity: n: P(n) requires P(n) Q1(n) … Qj(n))

• Modeling the environment that the program interacts with often involves continuous mathematics (e.g. calculus, trig) hybrid models

• Finite-state models can be automatically analyzed via model-checking (execution times can be exorbitant)

17

Proof CheckersProof Checkers

But we do use Proof Checkers! But we do use Proof Checkers! (aka Theorem Provers)(aka Theorem Provers)

-- because we make too many -- because we make too many mistakesmistakes

Why then do you call it a Why then do you call it a “theorem prover” if it can only “theorem prover” if it can only check? check?

Because it does help, it just Because it does help, it just can’t do the whole job.can’t do the whole job.

18

State Machine Analysis

Event From Toex x A - 10 ( x+10 , y )ey y B - 10 ( x , y+10)em x > y AND y > 0 ( x – y , y )em y > x AND x > 0 ( x , y - x )es ( y , x )

State = (x: nat, y: nat)

Events = {ex, ey, em , es}

19

State Machine Analysis:

0, 0

ex

10,0 20,0 30,0

10,10 20,100,10 30,10

ex ex

exexex

ey ey ey ey

0,20

es

es

em em

PROVE: For all reachable states (x,y) from (0,0):

x A and y B

(Proving Invariants)

20


• Theorem proving– deduction– human directed

• Model checking– exhaustive search– automatic– state space must be finite


21


0, 0ex

10,0 20,0 30,0

10,10 20,100,10 30,10

ex ex

exexex

ey ey ey ey

0,20

es

es

em em

PROVE: for all reachable states (x,y) from (0,0):

x A and y B

Counter-example: A = 30, B = 20

(0,0) ----> (10,0) ----> (20,0) ----> (30,0) ----> (0,30)ex ex ex es


22


INVARIANT: For all reachable states (x,y) from (0,0) that

x max(A,B) and y max(A,B)

Event From Toex x A - 10 ( x+10 , y )ey y B - 10 ( x , y+10)em x > y AND y > 0 ( x – y , y )em y > x AND x > 0 ( x , y - x )es ( y , x )

Proof Approach: Assume invariant holds in current state. Show eachtransition preserves the invariant. By induction true for all reachable states.


23

MODULE mainVAR x : 0..50; y : 0..50;

event: {ex,ey,em};

ASSIGN

init(x) := 0; init(y) := 0;

next(x) := case event = ex & x <= AA - 10 : x + 10; event = ey & y <= BB - 10 : x; event = em & x > y & y > 0 : x - y; event = em & y > x & x > 0 : x; 1 : y; esac; next(y) := case event = ex & x <= AA - 10 : y; event = ey & y <= BB - 10 : y + 10; event = em & x > y & y > 0 : y; event = em & y > x & x > 0 : y - x; 1 : x; esac;

SMV Version of State Machine

24

0

50

100

150

200

250

300

0..50 0..100 0..150 0..200 0..250

SMV Execution Times

Range of State Space Variables

Exe

cutio

n T

imes

(se

cs)

25

26

DEFINE AA := 40; BB := 50;

invariant_1 := AG (x <= AA & y <= BB); invariant_2 := AG (x <= max_AB & y <= max_AB);

-- specification invariant_1 is false-- as demonstrated by the following execution sequence

state 1.1:x = 0y = 0event = em

state 1.2:event = ey

state 1.3:y = 10

state 1.4:y = 20

state 1.5:y = 30

state 1.6:y = 40

state 1.7:y = 50event = em

state 1.8:x = 50y = 0

-- specification invariant_2 is true

Output From SMV Model Checker

27

Different perspectives on the Joys of Theorem Proving

Ms. Miller, tell the Ms. Miller, tell the verification team I will be a verification team I will be a little late with that proof.little late with that proof.

Just two more lemmas and the proof of the KB3D CD&R algorithm will be complete!

28

What Does It Look Like In a Theorem Prover?

x: VAR posreal % positive real numbern: VAR posnat % positive natural number

Y = IF n > 15 THEN x ELSE n * x ENDIF

Z = IF x 10 AND n 10 THEN 2 * x / n ELSE 10 / x ENDIF

x

n

YSystem

safe_prop: THEOREM Y * Z <= 200

Z

29

Executable

Code

Low-level design

High-level design

Requirements

Test cases

Coverage

Reviews

Traditional V&V

FM in development process

sizesize

101066 statesstates

10101010 statesstates

1010100100 statesstates

1010100100 100100

statesstates (i.e. 10 million lines code)(i.e. 10 million lines code)

proofsproofs

deeper,conceptual

shallow,thousands of cases

Programming Programming language such as language such as C++ have C++ have complex, ill-complex, ill-defined semantics defined semantics

30

Formal Methods Does Not Bring Perfection!

Requirements

proof

proof

proof

High Level Design

Detailed Design

Implementation

intendedfunctionality

physical devices

• peer review• simulation• testing

• peer review• simulation• testing

Are the component models accurate?

31

You guys are both my witnesses…He insinuated that ZFC set theory is superior to type theory!

Warning: some of our team members are a little Warning: some of our team members are a little sensitivesensitive

32

The Research Challenge

• Formal Methods are not a silver bullet:– Must be applied early in life cycle---not a band aid– Involves a significant learning curve– Each new application requires the development of new

models and analysis techniques– Cost of using formal methods still high for some

applications.• The software engineering community is dominated by the

process management viewpoint (i.e., that careful management of software development process is the key to quality)

• Historically, disasters must occur before major changes are made.

33

EXTRA SLIDESEXTRA SLIDES

34


sys_lem:

{-1} Y = IF n > 15 THEN x ELSE n * x ENDIF{-2} Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF |-------[1] Y * Z <= 200

Rule? (case "n > 15")

Case splitting on n > 15, this yields 2 subgoals

35


sys_lem.1

{-1} n > 15[-2] Y = IF n > 15 THEN x ELSE n * x ENDIF[-3] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF |-------[1] Y * Z <= 200

Rule? (assert)Simplifying, rewriting, and recording with decision procedures,

This completes the proof of sys_lem.1

What Happened? Well, assuming n > 15 we have:

Y = xZ = 10 / x

36


sys_lem.2

[-1] Y = IF n > 15 THEN x ELSE n * x ENDIF[-2] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF{-3} n <= 15 |-------[1] Y * Z <= 200

Rule? (assert)Simplifying, rewriting, and recording with decision procedures,

give us the other subgoal give us the other subgoal

37


sys_lem.2 {-1} Y = n * x[-2] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF[-3] n <=15 |-------[1] Y * Z <= 200

Rule? (replace -1) (hide -1)

sys_lem.2

[-1] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF[-2] n <=15 |-------[1] n*x* Z <= 200

38


Rule? (case "x <= 10 AND n <= 10")

Case splitting on x <= 10 AND n <= 10, this yields 2 subgoals

sys_lem.2.1

{-1} x <= 10 AND n <= 10[-2] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF[-3] n <=15 |-------[1] n * x * Z <= 200

Rule? (replace -1) Replacing using formula -1,

this simplifies to

39


sys_lem.2.1

{-1} Z = 2 * x / n[-2] n <= 10[-3] x <= 10|-------[1] n * x * Z <= 200


[-1] n <= 10[-2] x <= 10 |-------{1} n * x * (2 * x / n) <= 200

Rule? (assert) hit it with the hammer

40


sys_lem.2.1

[-1] n <= 15[-2] x <= 10 |-------{1} n * x * (2 * x / n) <= 200

Rule? (mult-ineq -2 -2)

{-1} x * x <= 10 * 10[-2] n <= 15[-3] x <= 10 |-------{1} n * x * (2 * x / n) <= 200

Rule? (assert)Simplifying, rewriting, and recording with decision procedures,This completes the proof of sys_lem.2.1.

But nothing happens!

hit it with the hammer again

41


[-1] Z = IF x <= 10 AND n <= 10 THEN 2 * x / n ELSE 10 / x ENDIF{-2} NOT (x <= 10 AND n <= 10){-3} n <= 15 |-------[1] n * x * Z <= 200


{-1} x >10 OR n > 10{-2} n <= 15 |-------[1] n * x * 10 / x <= 200

Rule? (assert)

This completes the proof of sys_lem2.2.This completes the proof of sys_lem.2.Q.E.D.

42

Different Verification Challenges

c(x,y) = y /[Sin2(x) + Cos2(x)]

x

yc(x,y)System

trigonometry

c(x,y) =

y

i

i

y

x

0

Need to formally reason about

series

c(x,y) = d/dx (x3y + y5)/xcalculus

Different Domains -->

Documents

1 Formal Methods Quick Tutorial Ricky W. Butler Oct 22, 2003