Formal Methods in the Real World: Specification ...sseshia/219c/spr...Kress-Gazit, and G. J. Pappas,...

Formal Methods in the Real World: Specification, Verification & 

Synthesis

Sanjit A. SeshiaProfessor

UC Berkeley

EECS 219CMay 1, 2019

Synthesis of Controllers/Plans for Cyber‐Physical Systems

S. A. Seshia 2

[Shoukry et al., HSCC 2017, CDC 2017, Proc. IEEE 2018;Desai et al. ICCPS 2017; Saha et al., IROS 2014]

Joint work with:Ankush Desai, Shromona Ghosh, Pierluigi Nuzzo, Indranil Saha,               Yasser Shoukry, Vijay Kumar, George Pappas, Shaz Qadeer,               

Alberto Sangiovanni‐Vincentelli, Paulo Tabuada

TerraSwarm Research Center & NSF ExCAPE project

Goal: Correct‐by‐Construction Motion Planning for Robotics

3

Declarative Task Specification (Temporal Logic)[+ Examples]

Safe, Correct Executable Software

ComponentLibrary

CompilerDrone executing plan in ROS/Gazebo Simulator

Challenges

• Scalable synthesis of motion plans blending high‐level (discrete) and low‐level (continuous) control

• Implementation in software on top of networked robotics platforms

• Dealing with untrusted components, uncertainty in environment, and ML‐based perception

S. A. Seshia 4

Motion Planning Problem

5

• Problem: Compute an obstacle‐free trajectory that is feasible with the given robot dynamics within the target workspace.

• Core problem for autonomous vehicles

• Challenges:

– Handling high‐level tasks (captured by formalisms such as Linear Temporal Logic)

– Handling distributed team of robots

– Handling uncertainty

[Shoukry et al., HSCC 2017, CDC 2017, Proc. IEEE 2018;Saha et al., IROS 2014]

Related Work: Abstraction Based Techniques

6

Discrete Continuous

X

Scales poorly as the number of continuousstates increases

• P. Tabuada and G. J. Pappas, “Linear time logic control of discrete-time linear systems,” TAC 2006.

• M. Kloetzer and C. Belta, “Temporal Logic Planning and Control of Robotic Swarms by Hierarchical Abstractions,” TAC 2007.

• G. E. Fainekos, A. Girard, H. Kress-Gazit, and G. J. Pappas, “Temporal logic motion planning for dynamic robots,” Automatica 2009.

• …

Related Work: Optimization Based Techniques

7

Discrete Continuous• Alberto Bemporad and Manfred

Morari, “Control of systems integrating logic, dynamics, and constraints,” Automatica, 35(3), 1999.

• E. M. Wolff, U. Topcu, and R. M. Murray, “Optimization-based trajectory generation with linear temporal logic specifications,” ICRA 2014.

• V. Raman, A. Donze, D. Sadigh, R. Murray, S. Seshia, “Reactive synthesis from signal temporal logic specifications,” HSCC 2015.

• …

Scales poorly as the number of discretestates increases

Two Extremes

8

Discrete Continuous(Abstraction based)

Discrete Continuous(Optimization based)

X

Scales poorly as the number of continuousstates increases

Scales poorly as the number of discretestates increases

Boolean Constraints

Convex Constraints

Satisfiability Modulo Convex Programming

9

• SAT Solvers: central tools in formal methods to reason about discrete dynamics.

• Convex Optimization: central tools in control theory to reason about continuous dynamics.

• Approach for hybrid dynamics: SAT Solving + Convex Programming

Convex Optimization

Mixed IntegerProgramming

SAT + ConvexSAT Solvers SMT

Solvers

[Shoukry et al., HSCC 2017]

Motivating Example: Obstacle Avoidance

10

• Given:• Robot dynamics (linear)• Input constraints• Initial and final states

• Generate the input sequence (controller)

Motivating Example: Obstacle Avoidance

11

• Given:• Robot dynamics (linear)• Input constraints• Initial and final states

• Generate the input sequence (controller)

Motivating Example: Obstacle Avoidance

12

Motivating Example: Obstacle Avoidance

13

Motivating Example: Obstacle Avoidance

14

SMC FormulaMonotoneDefinition:

Applications: Controller Synthesis

15

Obstacle Avoidance

LTL Motion Planning

Multi-robotMotion Planning

Secure Traffic Routing

Applications: CPS Security

16

Secure Localization

Secure State Estimation

Satisfiability Modulo Convex Programming

17

Monotone SMC Formula

The satisfiability of the monotone SMC formula can always be cast as a feasibility problem for a finite disjunction of convex constraints.

Any feasibility problem for a finite disjunction of convex programs can be posed as a satisfiability problem for a monotone SMC formula.

Theorem:

Complexity of Solving SMC

18

Reduce the number of iterations?

Monotone SMC Formula

Solving SMC

19

Key idea: abstraction-based search

Monotone SMC Formula

• Step 0:• Given a monotone SMC formula , construct its

Boolean expansion:

where is obtained from by replacing each convex constraint with a Boolean variable

Boolean Expansion

20

Lemma:and are equi-satisfiable.

Solver Operation

21

• Step I: Solve the “Booleanabstraction” of the problem.

• Step II: Extract which convexconstraints are active.

• Step III: Check the satisfiability of:

• Step IV: Generate UNSAT certificate:

Solver Operation

22

Performance?

Minimal UNSAT Certificate

23

• Finding a minimal UNSAT certificate is equivalent to finding an Irreducible Infeasible Set (IIS) of:

• NP-hard in general.

• However, our problem has more structure than general IIS problems.

• Can we exploit the structure of the Boolean constraintsto help find the IIS of the convex program?

Summary of UNSAT certificates

24

UNSAT Certificate Minimal

Complexity(number of convex

problems)

Trivial No Constant

IIS Yes Exponential

Sum of Slacks Yes* Linear

Minimum Prefix Yes* Constant

* under reasonable technical assumptions

Monotone SMC Formula

Minimum Prefix Certificates

25

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

26

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

27

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

28

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

29

Key idea: Find the “shortest” UNSAT certificate (e.g., the shortest prefix witness of a safety property violation).

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

30

Key idea: Find the “shortest” UNSAT certificate (e.g., the shortest prefix witness of a safety property violation).

Example: switched linear/convex systems, motion planning

Minimum Prefix Certificates

31

Definition: Positively Ordered Unate (POU) FunctionA function is said to be POU with respect to an ordering of its variables if for all values of bi, we have

Theorem:Consider a monotone SMC formula and its Boolean abstraction . If: (1) is positively ordered unate(2) the domain of the real variables is boundedthen minimal UNSAT certificates exist and can be computed in constant time.

Theorem:Let s* be the solution of the above optimization problem. If k* is the minimal index such that |s*|> 0, then the certificate: is the minimum prefix certificate.

Minimum Prefix Certificates

32

Summary of UNSAT certificates

33

UNSAT Certificate Minimal

Complexity(number of convex

problems)

Trivial No Constant

IIS Yes Exponential

Sum of Slacks Yes* Linear

Minimum Prefix Yes* Constant

* under reasonable technical assumptions

Monotone SMC Formula

Scalability Results

34

Increase the number of Boolean constraints#Boolean variables = 4800#Real variables = 100

Increase the number of Real variables#Boolean variables = 4800#Boolean constraints = 7000

10000 x

http://yshoukry.bitbucket.io/SatEX

Results(1): Single Robot, Reach-Avoid

10000 x

Syclop (Synergistic Combination of Layers Of Planning):- High level planner + low level RRT/EST- outperform traditional sampling-based algorithms by orders of magnitude

Z3 and MILP (LTL OPT) times outtime out =1 hour

SMC generated trajectories (blue) aresmoother than Syclop generated traje

Result (2): LTL Motion Planning

Result (3): Multi-Robot, LTL

Result (4): Secure CPS

38

Under attack - no protection Under attack - with protection

Specification & Verification of Industrial CPS via Formal Inductive Synthesis

S. A. Seshia 39

[Jin, Donze, Deshmukh, Seshia, HSCC 2013, TCAD 2015;Yamaguchi et al. FMCAD 2016;Jha & Seshia, Acta Inf. 2017]

Joint work with:Jyotirmoy Deshmukh, Alex Donze, Susmit Jha, Xiaoqing Jin, 

Tomoyuki Kaga, Tomoya Yamaguchi

Typical Closed‐Loop CPS Model/Verification

S. A. Seshia 40

Controller Plant

Environment

u

ego

adv

Challenges for Verification of Closed‐Loop Automotive Control Systems

Closed‐loop setting very complex software + physical artifacts  nonlinear dynamics large look‐up tables large amounts of switching

Requirements Incomplete/Informal Specifications often created concurrently with the design!

Designers often only have informal intuition about what is “good behavior” “shape recognition” 

Mining Requirements from Closed‐Loop Models 41

Experimental Engine Control Model

Industry Problem: Applying Formal Methods to  Legacy Systems

It’s working, but I don’t understand why!

Our Solution: Requirement Mining

Value added by mining:

Mined Requirements become useful 

documentation

Use for code maintenance and revision

Use during tuning and testing

Mining Requirements from Closed‐Loop Models 42

Designer reviews mined requirements  “Settling time is 6.25 ms” “Overshoot is 100 units” Expressed in Signal Temporal Logic                          

[Maler & Nickovic, ‘04]

Tool extracts properties of closed‐loop design using a Simulator

Designer’s View of Our Solution

6.25ms

100

Mining Requirements from Closed‐Loop Models 43

CounterExample Guided Inductive Synthesis (CEGIS)

Find “Tightest” Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 5 msOvershoot is 5 KPaUpper Bound on x is 3.6

1.

Mining Requirements from Closed‐Loop Models 44

Experimental Engine Control Model

[Jin, Donze, Deshmukh, Seshia, HSCC’13; TCAD’15]

Idea of CEGIS goes back to ASPLOS 2006 paperby Solar‐Lezama et al.

Settling Time is 5.3 msOvershoot is 5.1 KPaUpper Bound on x is 3.8

Settling Time is … msOvershoot is … KPaUpper Bound on x is …

CounterExample Guided Inductive Synthesis (CEGIS)

Find “Tightest” Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 45

Experimental Engine Control Model

CounterExample Guided Inductive Synthesis

Find "Tightest" Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

NO

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

Mined Requirement

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 46

Experimental Engine Control Model

CounterExample Guided Inductive Synthesis

Find "Tightest" Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

NO

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

Mined Requirement

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 47

Experimental Engine Control Model

Parametric Signal Temporal Logic 

(PSTL)

Parameter Synthesis (exploits monotonicity)

Optimization‐based Falsification

CounterExample Guided Inductive Synthesis

Find "Tightest" Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

NO

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

Mined Requirement

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 48

Experimental Engine Control Model

Parametric Signal Temporal Logic 

(PSTL)

Parameter Synthesis (exploits monotonicity)

Optimization‐based Falsification

Signal Temporal Logic (STL)• Extension of Linear Temporal Logic (LTL) and Metric Temporal Logic (MTL)– Quantitative semantics: satisfaction of a property over a trace given real‐valued interpretation

– Greater value more easily satisfied – Non‐negative satisfaction value  Boolean satisfaction

• Example: “For all time points between 60 and 100, the absolute value of x is below 0.1”

S. A. Seshia 49

0 100

1

-0.1 +0.1

60

x

t

[Maler & Nickovic, 2004]

Quantitative Satisfaction Function  for STL

• Function  that maps STL formula  and a given trace (valuation of signals) to a numeric value 

• Example: is  inf[60,100] (0.1 - |x|)

• Quantifies “how much” a trace satisfies a property– Large positive value: trace easily satisfies – Small positive value: trace close to violating – Negative value: trace does not satisfy 

Mining Requirements from Closed‐Loop Models 50

Parametric Signal Temporal Logic (PSTL)

• Constants in STL formula replaced with parameters– Scale parameters– Time parameters

• Examples:

S. A. Seshia 51

Between some time and 10 seconds, x remains greater than some value

After transmissionshifts to gear 2, itremains in gear 2 for at least secs

CounterExample Guided Inductive Synthesis

Find "Tightest" Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

NO

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

Mined Requirement

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 52

Experimental Engine Control Model

Parametric Signal Temporal Logic 

(PSTL)

Parameter Synthesis (exploits monotonicity)

Optimization‐based Falsification

Parameter Synthesis = Find ‐tight values of params (for suitably small )

0 100

3

2.9

1 000 000

Find "Tightest" Properties

Mining Requirements from Closed‐Loop Models 53

Too loose

Want the value of  corresponding to the “tightest” satisfaction over a set of traces

x

Parameter Synthesis

• Non‐linear optimization problem

– Satisfaction function for STL is non‐linear in general

• Naïve (“strawman”) approach: 

– grid parameter space to  precision

– evaluate satisfaction value at each point

– pick valuation with smallest satisfaction value

• Problem: Exponential number of grid points (in 

#parameters)

Find "Tightest" Properties

Mining Requirements from Closed‐Loop Models 54

If upper bound of all signals is 3, any number > 3 is also an upper bound

• Satisfaction function monotonic in parameter value• Example:

• (, x) = inft (  ‐ x(t) )• For all x, (, x) is a monotonic function of • Advantage: If monotonic, use binary search over parameter space, otherwise exhaustive search

Satisfaction Monotonicity

0 10050

34

Find "Tightest" Properties

Mining Requirements from Closed‐Loop Models 55

• Need to decide whether:For all x, (, x) is a monotonic function of 

• Theorem: Deciding monotonicity of a PSTL formula is undecidable

• Use an encoding to satisfiability modulo theories (SMT) solving – Quantified formulas involving uninterpreted functions, and arithmetic over reals  linear arithmetic if PSTL predicates are linear

– Solved easily with Z3

Deciding Satisfaction MonotonicityFind "Tightest" Properties

Mining Requirements from Closed‐Loop Models 56

CounterExample Guided Inductive Synthesis

Find "Tightest" Properties

Settling Time is ??Overshoot is ??Upper Bound on x is ??

Are there behaviors that do NOT satisfy theserequirements?

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

NO

Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1

Mined Requirement

Counterexamples

1.

Mining Requirements from Closed‐Loop Models 57

Experimental Engine Control Model

Parametric Signal Temporal Logic 

(PSTL)

Parameter Synthesis (exploits monotonicity)

Optimization‐based Falsification

Black‐Box Falsification Procedure

u S(u)

Falsification Tool

\

\

Mining Requirements from Closed‐Loop Models

Are there behaviors that do NOT satisfy theserequirements?

58

Falsification as Optimization

• Solve * = minu (, S(u))– Leverages quantitative semantics of STL– Relies on standard numerical optimization methods (e.g. 

Nelder‐Mead)

• If * < 0, found falsifying trace!

Nonlinear Optimization Problem, No exact solution, Limited theoretical guarantees

Are there behaviors that do NOT satisfy theserequirements?

Mining Requirements from Closed‐Loop Models 59

Experimental Evaluation Summary   [details in TCAD’15 paper]• Defined Templates for Common Requirements in Automotive Control – all monotonic PSTL!!– Dwell‐Time requirements– Timed/Untimed Safety properties– Timed Inevitability (bounded liveness)– Input Profiles: assumptions on shape of input signals– Control‐theoretic requirements on output signals (bounded overshoot/undershoot, settling time, error from reference signal, etc.)

• Three Benchmarks– Simple Simulink Automatic Transmission Model– Toyota HSCC’14 Challenge  – Air‐Fuel Ratio controller– Toyota Experimental Diesel Engine Airpath controller

S. A. Seshia 60

Results on Industrial Airpath Controller

• Found max overshoot with 7000+ simulations in 13 hours• Attempt to mine maximum observed settling time: 

– stops after 4 iterations– gives answer tsettle = simulation time horizon (shown in trace below)

Experimental Engine Control Model

Mining Requirements from Closed‐Loop Models 61

[Jin, Donze, Deshmukh, Seshia, HSCC 2013]

Mining can expose deep bugs

• Uncovered a tricky bug– Discussion with control designer revealed it to be a real bug

– Root cause identified as wrong value in a look‐up table, bug was fixed

• Why mining could be useful for bug‐finding:– Can uncover subtle relations that should not hold– Looking for bugs Mine for negation of bug

Experimental Engine Control Model

Mining Requirements from Closed‐Loop Models 62

Bug fixed  Settling time successfully mined 

S. A. Seshia 63

OLD

NEW

Industrial Case Studies with Toyota

• Work with group @ Toyota Japan on enabling software verification by mining specifications on the closed‐loop system

• Useful in a production setting:– Finds “issues” where previous methods fell short!– Reduced 70% of human effort

S. A. Seshia 64

[Yamaguchi et al., FMCAD’16]

Toyota Unit’s Experience with Model Checking 

65

Total Work hour

560min

1 trial

70min

40min

7 trials

Making modelMaking property

Revising property

Revising model

Executingmodel checking

Mappingcounterexample

Making/revising property: 110 minMapping counterexample: 280 min for just 1 module

[Yamaguchi et al., FMCAD’16]

66

Overview of Methodology

in out

1. Pre-condition mining

Pre-condition forsoftware module

2. Softwaremodel checking

Module levelcounterexample

System levelcounterexample

3. Simulation-BasedVerification

Breach

Breach

SLDV/CBMC

[Yamaguchi et al., FMCAD’16]

Requirement Mining In Toyota Case Study  

67

▼ Founded false case

Violation area of post-condition

▼

▼▼

Find system level violationactuator output < 150

[Yamaguchi et al., FMCAD’16]

Summary: Specification & Verification of Industrial CPS

• Formal Specification often not available• Inductive Synthesis from Models and Data can generate useful specifications and find bugs– Counterexample‐Guided Inductive Synthesis (CEGIS)

• Simulation‐based Temporal Logic Falsification is a scalable, industrially‐applicable method

• Requirement Mining + Falsification can make Software Verification tools more effective

S. A. Seshia 68