Reusable Models for Timing and Liveness Analysis of Middleware for Distributed Real-

Time and Embedded SystemsVenkita Subramonian and Christopher Gill*

{venkita,cdgill}@cse.wustl.eduDept. of Computer Science and

EngineeringWashington University, St. Louis, MO

EMSOFT 2006October 23-25, 2006, Seoul, Korea

*Research supported in part by NSF CAREER award CCF-0448562+Research supported in part by NSF grants CCR-01-21403, CCR-02-20134, CCR-02-09237, CNS-0411363, and CCF-0430102, ARO grant DAAD19-01-1-

0723, and NAVY/ONR contract N00014-03-1-0939

Thanks also to Joseph Sifakis, Marius Bozga and Iulian Ober for valuable advice about the IF tool set,

and to Doug Schmidt and the hundreds of ACE+TAO+CIAO developers/users world-wide

César Sánchez and Henny Sipma+

{cesar, henny}@cs.stanford.eduDepartment of Computer ScienceStanford University, Stanford, CA

2

A Motivating Example: Real-Time Image Transmission

low (and variable) bandwidth radio link

virtual folder,images

adaptationmiddleware

transmissionmiddleware

displays

serverside

clientside

imagesource

Camera

Console

Soft real-time distributed system (over wireless)

Many end-to-end activities must be managed at once» image (de-)compression,

transmission, analysis, display Needs adaptive enforcement

» Task scheduling, rate adaptation, feedback control

Many applications are like this» Is correct reuse of middleware

possible?[Gill et al., “Integrated Adaptive QoS

Management in Middleware: An Empirical Case Study” (RTAS ‘04)]

[Wang et al., “CAMRIT: Control-based Adaptive Middleware for Real-time Image Transmission” (RTAS ‘04)]

3

Motivation for Formal Models of Middleware

…what's ultimately most important is to determine how things perform for applications that are representative of actual user configurations, rather than micro-benchmarks alone…

ORB configurations have a substantial impact on performance, e.g., TAO's fastest concurrency model is thread-per-connection, rather than thread pool…

Doug Schmidt, ICE/CORBA newsgroup discussion 3/29/05

…a main obstacle to the application of rigorous development techniques is the lack of methodology for relating application software and functional design to physical architecture and implementation…

Joseph Sifakis, “Modeling real-time systems-challenges and work directions”, EMSOFT 2001

The real-time embedded systems community needs formal ways to evaluate the semantics of different middleware configurations

The formal verification community needs accurate middleware models

4

Vanderbilt University and UC Irvine» GME, CoSMIC, PICML, DREAM

UC Berkeley» Ptolemy, E-machine, Giotto

Kansas State University and University of Nebraska» Bogor, Cadena

University of Pennsylvania» VERSA

University of Brest» Cheddar

A Brief Survey of Other Related Work

5

ACE+TAO+CIAO COTS Middleware Architecture

A “Distributed System Software Stack”

EMSOFT (Wednesday, session B9)

modeling and verification focus

CODES+ISSS (Tuesday, session A10)

system software engineering focus

EMSOFT (Monday, session B1)

protocol design and proof focus

TAO+CIAO

ACE

6

Goal: Correct (Re-)use of DRE Middleware

Thesis: formal models of fine-grain middleware building blocks provide a principled approach to analyze timing and liveness properties of middleware-based DRE systems

CADENADREAMRMAPtolemyGMECoSMICUPPAALIF-toolkitSPINBogor

System ModelingSystem Implementation

Formal approach

tocomposition

Reusablesoftwarebuilding blocks

Bridging the Gap:composable, reusable and executable timed automata models of ACE building blocks

7

Another Gap: Between High Level Models and Lower-Level Consequences of their Configuration Choices

Modeling at a high-level of abstraction is helpful» Focus on nuances of the application

Low-level design choices matter too» May invalidate (implicit) assumptions

We focus on low-level frameworks» Capture fundamental concurrency

and timing details of the middleware

CORBA

EJB

CCM

Implementation Platform

OS

Application Models

Collocation strategiesBuffer flushing strategiesCommunication strategiesConnection strategiesConcurrency strategies

Pattern-based frameworks

8

Model Architecture in IF for ACE

Network/OS layer: interprocess communication abstractions Middleware layer: ACE pattern-oriented abstractions Application layer: application-specific semantics within ACE event handlers

9

Modeling Threads Challenge

» No native constructs for threads in model checkers that currently support timed automata

Option 1: model all thread actions as a single automaton» Suitable for high level modeling of application semantics

Option 2: model a thread as multiple interacting automata » Interactions model the flow of control » This option better abstracts the nuances of ACE-level mechanisms

Foo

output method_request

Bar

input method_request

input method_result output method_result

10

Modeling Thread Scheduling Semantics (1/4)

Easy to achieve with one automaton per thread» Specify to model checker directly

E.g., using IF priority rules More difficult with more than one

automaton per thread » Thread of control spans interactions

among automata» Need to express thread scheduling in

terms of execution control primitives provided by the model checker

prio_rule: pid1 < pid2 if pid1 instanceof Activity1 and pid2 instanceof Activity2

Update Display

Control Flow Rate

1 automaton per thread

Activity1 Activity2

Foo

output m_req

Bar

input m_req

input m_result output m_result

2 automata per thread

11

Modeling Thread Scheduling Semantics (2/4) Solution

» Introduce a thread id that is propagated along automata interactions

» Thread id acts as an index to a storage area which holds each thread’s scheduling parameters

1

1

2

2

Foo1 Bar1

Foo2 Bar2

Prio=5 Prio=8

thread_schedule: pid1 < pid2ifpid1 instanceof Foo1 and pid2 instanceof Bar1 and({Foo1}pid1).threadid <> ({Bar1}pid2).threadid and({Thread}(({Foo1}pid1).threadid)).prio <({Thread}(({Bar1}pid2).threadid)).prio )

Give higher preference to the automaton whose “thread” (pointed to by thread id) has higher priority

Foo1 Bar1

Bar2Foo2 Bar1

Foo1

Hint to the model checker

Resulting Behavior

12

Modeling Thread Scheduling Semantics (3/4) What if two threads have the same priority? In an actual implementation, run-to-completion

(SCHED_FIFO) may control the possible interleavings How can we model run-to-completion?

Foo1 Foo2 Foo3

Bar1 Bar2 Bar3

Bar1

Foo1

Foo2

Foo3

Bar1

Bar2

Bar3

Bar1

Bar1

Bar2

Bar3

Foo1

Foo2

Foo3

Foo1

How do we pruneout this space?

13

Modeling Thread Scheduling Semantics (4/4)

Current=nil

Bar1

Foo1

Foo2

Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil

Current=1 Current=2

Current=1

Bar1

Bar1

Solution» Record id of

currently executing thread

» Update when executing actions in each automaton

Current=1

Bar1

Foo3

Current=2

Bar2

Current=2

Bar3

Current=2

Hint to the model checker

14

Problem: Over-constraining Concurrency

Current=nil

Bar1

Foo1

Foo2

Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil

Current=1 Current=2

Current=1

Bar1

Bar1

Current=1

Bar1

Foo3

Current=2

Bar2

Current=2

Bar3

Current=2

Hint to the model checker

Current=2

Time progresses

Bar3

Foo3

Bar3 always chosen to run

15

Solution: Idle Catcher Automaton

Key idea: lowest priority “catcher” runs when all others are blocked» E.g., catcher thread in

middleware group scheduling (RTAS ‘05)

Here, idle catcher automaton» runs when all other automata

are idle (not enabled), but before time progresses

» Resets value of current id to nil

Current=2

Current=nil

Time progress

Bar3Foo3

Foo3, Bar3 blockedat this point

Current=nil

Idle catcher runs

Foo3 or Bar3 could be chosen to run. Over-constraining eliminated

16

Problem: Tractability

Model checking can suffer from state space explosion» State space reduction, live variable analysis can help» But even good model checkers don’t fully solve this

Need to think of modeling as a design issue, too» Does the model represent what it needs to represent?» Can the model be re-factored to help the checker?» Can domain specific information help avoid unnecessary checking?

“right away”“in a minute”“get coffee”“go for an espresso”“maybe tomorrow?”

17

Optimization 1: Leader Election Leader/Followers concurrency

» Threads in a reactor thread pool take turns waiting on the reactor

» One thread gets the “token” to access the reactor - leader

» All other threads wait for the token – followers It does not matter which thread gets selected

as leader in a threadpool Model checker not aware of this domain

specific semantics For BASIC-P protocol example, saved factor of

~50 in state space, and factor of ~20 in time

Token to access thereactor is available

Prune this out

T1T2

T3

T2 T3

T3

18

Optimization 2: System Initialization Similar idea, but different technique Iff ok to establish initial object relations

in any order, can optimize away» E.g., 2 server automata, each of which

creates a reactor automaton Useful when modeling object systems

in model checkers with dynamic automaton creation capability (e.g., IF)

State space reduction depends on application» Factor of ~250 for a deadlock scenario

with 2 reactors and 3 threads in each reactor

S1 R1 S2 R1

S1 R1

S2 R2

S2 R1

S1 R2

Prune this out

S1 creates R S2 creates R

S2 creates RS1 creates R

19

Verification of a Real-Time Gateway

An exemplar of many realistic ACE-based applications We modified the Gateway example to add new

capabilities» E.g., Real time, Reliability, Control-Push-Data-Pull» Value added service in Gateway before forwarding a to

consumer E.g. Using consumer specific information to customize data stream

Different design, configuration choices become important » E.g., number of threads, dispatch lanes, reply wait strategies

Gateway

Supplier1

Supplier2

Consumer1

Consumer3

Consumer2

Consumer4

20

Model Checking/Experiment Configuration

Gateway is theoretically schedulable under RMA» Utilization = 80%» Schedulable utilization = 100% for harmonic

periods » Assumption – Messages from 50ms supplier is

given higher preference than 100ms supplier ACE models let us verify scheduling enforcement

» IN THE ACTUAL SYSTEM IMPLEMENTATION

Gateway

S1

C1

S2

C2

C3

C4

20

20

10

10

100ms

50ms

100ms

100ms

50ms

50ms

Period

Relative Deadline

Deadline Exec time

C1 100ms 20ms

C2 100ms 20ms

C3 50ms 10ms

C4 50ms 10ms

Value-addedexecution

(and its cost)

21

Real-time Gateway – Single Thread

Single reactor thread dispatches incoming events I/O (reactor) thread same as dispatch thread I/O thread responsible for value added service

Gateway

SupplierHandler

SupplierHandler

Reactor

ConsumerHandler

ConsumerHandler

ConsumerHandler

ConsumerHandler

Supplier Consumer

22

Real-time Gateway – Dispatch Lanes

Gateway

SupplierHandler

SupplierHandler

Reactor

ConsumerHandler

ConsumerHandler

ConsumerHandler

ConsumerHandler

Supplier Consumer

Single reactor thread again dispatches events to gateway handlers I/O (reactor) thread puts message into dispatch lanes Lane threads perform value added service, dispatch to consumers DO QUEUES HELP OR HURT TIMING PREDICTABILITY?

23

Model/Actual Traces for Real-time Gateway

C1

C1

C2

C2

C3 C4

C3 C4

20 40 6010 30 50

S1,S2 S2

Model

Actual

Single threaded GatewayExecution in the context of reactor thread

Deadline miss for Consumer4because of blocking delay at reactor

Gateway with dispatch lanes

Execution in the context of lane threads

20 40 6010 30 50

C3 C4 C1 C2 C3 C4 C2

C3 C4 C1 C2 C3 C4 C2

70 80 90100Time

C1, C2, C3, C4

C3 C4 C1 C2 C3 C4 C2

20 40 6010 30 50 70 80 90100

Period

Exec time

Deadline

C1

100ms

20ms

100ms

C2

100ms

20ms

100ms

C3

50ms 10ms

50ms

C4

50ms 10ms

50ms

Expected execution timeline with RMS

Time

C3, C4

S1,S2 S2

24

Concluding Remarks Timed automata models of middleware building blocks

» Can be used to verify middleware concurrency and timing semantics

Domain specific model checking optimizations» Can achieve significant reductions in state space

We have run multiple case studies using these models» Allow comparison of model checking & software execution

traces» Demonstrate reusability and fidelity of our models

Future work» Continue to develop reusable models and optimizations» Incorporate domain information directly into model checking

tools » Extend our approach beyond real-time concerns

Model mechanism faults and failure modes Power usage (yet another ~real valued “clock”)

25

Questions?

26

Backup Slides

27

Model checking & actual timing tracesshow BASIC-P protocol’s regulation ofthreads’ use of resources (no deadlock)

EH33

EH23EH13

Timing Traces for BASIC-P Protocol

EH22EH12

R1 R2

EH32

Flow2

R1 R2

Flow3

EH31

EH21EH11

R1 R2

Flow1

28

BASIC-P Blocking Delay ComparisonActual ExecutionModel Execution

Blocking delayfor Client2

Blocking delayfor Client3

29

Overhead of ACE TP/DA reactor with BASIC-P

Negligible overhead with no DA protocol

Overhead increases linearly with # of event handlers due suspend/resume actions on handlers at BASIC-P entry/exit