SDRL & RTG University of Pennsylvania 5/24/01 1 Run-time Monitoring and Checking Based on Formal Specifications Insup Lee Department of Computer and Information

5/24/01

1

SDRL & RTGUniversity of Pennsylvania

Run-time Monitoring and Checking Based on Formal Specifications

Insup LeeDepartment of Computer and Information Science

University of Pennsylvania

24 May 2001

Joint work with M. Kim, M. Viswanathan, S. Kannan, and O. Sokolsky

5/24/01

2


Objectives• Specification and verification

– complete analysis, all behaviors are correct– gap between abstract model and

implementation• Testing

– tested behaviors are correct– not complete

• Run-time behavior checking– consistency between abstract model and

implementation• To provide a framework for automatic generation

of monitors and checkers

5/24/01

3


Fundamental Issues

• How does a monitor gather information from a running system?

• How does the monitor relate to requirements?

• How do we integrate dynamic monitoring with static analysis?

• Can monitor be used to steer a system?• What mathematical guarantees do monitors

provide?

5/24/01

4


Monitorable Properties• Run-time monitoring and checking checks whether

properties are violated or not by observing only finite traces execution at run-time

• A class of monitorable properties is a Turing computable subset of safety properties – Safety property: finite trace reveals violations.– Monitorable property: violations Turing

computable.• Equivalent to class 1 in the Arithmetic Hierarchy

Liveness

Safety

MEDL = Monitorable property

Properties on Traces

Safety closure of the halting problem

5/24/01

5


System SpecSystem

Spec

RequirementSpec

RequirementSpec

Formal verification

Design

SystemImplementation

SystemImplementation

MonitoringScript

MonitoringScript

Implementation

Checker/CorrectorChecker/Corrector

SystemSystem FilterCommunication

Run-time Check

MaCS Framework

EventHandler

EventHandler

CorrectorCorrector

CheckerChecker

5/24/01

6


Design Issues

• Filter– passive versus active– when to take snapshot

• Event Handler– mapping between concrete state and

abstract event• Checker

– inclusion based on trace, ready semantics, bisimulation

• Corrector– how to provide feedback

5/24/01

7


Overview of MaCS framework• Based events and conditions• Two types of abstraction

– time abstraction by instrument filter (IF)– data abstraction by event recognizer (ER)

• Scripting languages– MEDL (Meta Event Definition Language):

property specification language– PEDL (Primitive EDF): Implementation-language

dependent• Goals: Automatic generation of IF, ER, Checker,

Steerer

5/24/01

8


Events and Conditions

• Must be able to reason about both time instants and durations in a program execution.– Events and conditions are a natural division,

which is also found in other formalisms like SCR.

• Need temporal operators combining events and conditions in order to reason about traces.

Crecy Poitiers Agincourtstart(war) end(war)

1341 145714121344 1348

5/24/01

9


Two-Sorted Logic

• Conditions interpreted over 3 values: true, false and undefined.

• [E,E) pairs a couple of events to define an interval.• start and end define the events corresponding to

the instant when conditions change their value.• Primitive events and conditions

212121 | | | ),[ | )(defined :: CCCCCEECC

CE

EEEECCeE

when

| | | )(end | )(start | :: 2121

5/24/01

10


An example

• Example: correct = oldDateUsed => (f().num-retries = 1)

– If old data is used then number of retries must be 1

– f().num-retries is a local variable of function f()

invoke f() f().num-retries = 1 oldDataUsed return oldDataNotUsed

• correct is true, undefined, and true again.

5/24/01

11


MaCS prototype architecture

Program(Java byte code)

Monitoring Script(PEDL)

Requirements(MEDL)

PEDLPEDLCompilerCompiler

MEDLMEDLCompilerCompiler

Instrumented Code

Filter Generator(JTREK)

InstrumentationInformation

Compiled PEDL

Compiled MEDL

Event Recognizer Checker

Steering Script(SADL)

SADLSADLCompilerCompiler

InstrumentationInformation

Injector class(Java byte code)

5/24/01

12


MEDL

• Meta Event Definition Language (MEDL)• Expresses requirements using the events and conditions,

sent by event recognizer.

• Describes the safety requirements of the system, in terms of conditions that must always be true, and alarms (events) that must never be raised.– property periodic = (period == 1000)

– alarm staleData = oldDataUsed when connected

• Auxiliary variables may be used to store history.– request_info -> { num_hits’ = num_hits+1; hit_ratio’ = num_hits/num_fails; }

• Used to automatically create checker

5/24/01

13


Structure of MEDL scriptReqSpec HexPattern

// imported events, conditions, and actionsimport event MAValert, initimport action collapse, restore// variable declarationsvar int numMAValert, prevCount1, prevCount2var float prevAverage, currAverage// definitions of eventsevent endPeriod = start(time(MAValert)-periodStart > 3000)alarm NoPattern = start(currAverage > prevAverage*1.1+10) // guarded variable updates and action invocationsupdateAvg -> {

currAverage’= (prevCount2+prevCount1+numMAValert)/3;}NoPattern -> { invoke collapse; }

End

5/24/01

14


PEDL

• Primitive Event Definition Language (PEDL)• PEDL describes a thin interface between low-level

implementation and high-level requirement– The language maps the low-level state

information of the system to high-level events used in describing the requirements.

– Fast event recognition is key point• Provides primitives to refer to values of variables

and to certain points in the execution of the program• Depends on system implementation language• Used to create filter and event recognizer

5/24/01

15


Primitive Event Definition Language (PEDL)

• Information about the system comes in two different forms:– Conditions, which are true or false for a finite duration of time

(e.g., is variable x >5?), and– Events, which are either present or absent at some instant of

time (e.g., is the control right now at the end of method f?)• Provides primitives to refer to values of variables and to

certain points in the execution of the program.– condition IC = (50<train_position) && (train_position<100);– Event endGD = startM(Gate.gu());

• Provides primitive “time” to refer to time when events happen – condition slowTrain = (time(endIC)-time(startIC)) > 3000;

5/24/01

16


Structure of PEDL ScriptMonScr MAVpattern

// exported events and conditionsexport event MAValert, init;

// Overhead Reduction Options[timestamp;][valueabstract;][deltaabstract;][multithread;]

// monitored object declarationsmonmeth void Console.createMAVs(int);monobj double MAV.run().distance;

// definitions of events and conditionsevent init = startM( Console.createMAVs(int) );event MAValert = start(MAV.run().distance > 40 &&

MAV.run().distance < 120) ;End

5/24/01

17


The Current MaCS Prototype System

• MaCS instruments Java bytecode, not a source code.

• Filter resides in the host of target program as a separate thread.

• The filter sends updated value and time stamp to the event recognizer.

• The event-recognizer (ER) evaluates condition and event description

• ER sends evaluation result to the run-time checker.• MaCS works on multi-threaded applications

5/24/01

18


MaCSware Version 0.99

• Components– Static components

• PEDL/MEDL/SADL compiler• Bytecode Instrumentor

– Dynamic components• Filter/Injector, event recognizer and run-time

checker• GUI

• http://www.cis.upenn.edu/~rtg/macs

5/24/01

19


Related Work

• Monitoring in process-level by input/output/message interface instrumentation – Supervisor [T.Savor98], JEM[G.Liu99] ,MOTEL[Log00]

• Monitoring in statement-level by target system instrumentation– ANNA [S.Sankar93]

• checks assertion for annotated Ada program

– Java Runtime Timing Constraint Monitor [A.Mok97]• monitors Java program using specification language RTL

– ALAMO [C.Jeffery98]• instruments C source code and monitors it

• Monitoring in instruction-level by environment instrumentation– Dynascope [R. Sosic95], JPDA [JVM99]

5/24/01

20


Current and future work• Extend MaCS framework

– Monitoring of hybrid systems– Distributed monitoring

• Extend MaC prototype implementation– Monitoring and checking frequency optimization– Add program checking capability– Add steering capability

• Develop MaC formal foundations • Integrate with other tools

– Network simulator – Test generator

– Charon• http://www.cis.upenn.edu/~rtg/macs

5/24/01

21


Automatic Test Generation from Formal Specifications

Insup LeeDepartment of Computer and Information Science

University of Pennsylvania

24 May 2001

Joint work with H.S. Hong and O. Sokolsky

5/24/01

22


Specification-based Testing

Specification Spec(I)

• Determines whether an implementation conforms to its specification – Hardware and protocol conformance testing– FSM and LTS

• For each test case (I, Spec(I)),– Apply I to the implementation– Observe the set of output sequences Imp(I)– Check Imp(I) Spec(I)

Implementation

I

Imp(I)

5/24/01

23


Specifications

• Specification-based test generation for reactive, real-time, and hybrid systems– EFSM

• FSM + data variables– Statecharts, hierarchical reactive modules

• EFSM + hierarchy + concurrency + communication – CHARON, hybrid systems

• EFSM + hierarchy + concurrency + communication + differential equations

5/24/01

24


Test Coverage Criteria for EFSMs• Structural information in EFSMs (application-independent)

– Control flow: state or transition coverage– Data flow: all-definition, all-use, or all-def-use-path

• Properties to be validated (application-dependent)– MEDL scripts, Scenarios, MSCs, temporal logics

empty nonempty

t1: increment/money:=1

t4: decrement[money=1]/money:=0

t2: increment[money<MAX]/money:=money+1

t3: decrement[money>1]/money:=money-1

5/24/01

25


Test Generation from EFSM

• Model checking based approach to generate tests– Finding counterexamples during the model checking of

EFSMs– State coverage: !EF nonempty– Transition coverage: !EF t4– All-definition coverage: !EF (t1 & EF (!(t1|t2|t3|t4) U t4))

• Explore other approaches

empty nonempty

t1: increment/money:=1

t4: decrement[money=1]/money:=0

t2: increment[money<MAX]/money:=money+1

t3: decrement[money>1]/money:=money-1

5/24/01

26


Test Generation for CHARON

• Adapt EFSM-based test generation to hybrid systems

• Transforming differential equations into EFSMs– Qualitative reasoning and predicate abstraction

techniques for CHARON• Determining the executability of test cases

– Symbolic execution techniques for CHARON

5/24/01

27


Other issues

• Test generation– How to meet coverage criteria– Test suite optimization

• Test execution– Integration of testing with MaCS

• Effectiveness of coverage criteria in practice• Apply to real examples

5/24/01

28


Q&A

Documents

SDRL & RTG University of Pennsylvania 5/24/01 1 Run-time Monitoring and Checking Based on Formal Specifications Insup Lee Department of Computer and Information