Reusable Models for Timing and Liveness Analysis of Middleware for Distributed Real-
Time and Embedded SystemsVenkita Subramonian and Christopher Gill*
{venkita,cdgill}@cse.wustl.eduDept. of Computer Science and
EngineeringWashington University, St. Louis, MO
EMSOFT 2006October 23-25, 2006, Seoul, Korea
*Research supported in part by NSF CAREER award CCF-0448562+Research supported in part by NSF grants CCR-01-21403, CCR-02-20134, CCR-02-09237, CNS-0411363, and CCF-0430102, ARO grant DAAD19-01-1-
0723, and NAVY/ONR contract N00014-03-1-0939
Thanks also to Joseph Sifakis, Marius Bozga and Iulian Ober for valuable advice about the IF tool set,
and to Doug Schmidt and the hundreds of ACE+TAO+CIAO developers/users world-wide
César Sánchez and Henny Sipma+
{cesar, henny}@cs.stanford.eduDepartment of Computer ScienceStanford University, Stanford, CA
2
A Motivating Example: Real-Time Image Transmission
low (and variable) bandwidth radio link
virtual folder,images
adaptationmiddleware
transmissionmiddleware
displays
serverside
clientside
imagesource
Camera
Console
Soft real-time distributed system (over wireless)
Many end-to-end activities must be managed at once» image (de-)compression,
transmission, analysis, display Needs adaptive enforcement
» Task scheduling, rate adaptation, feedback control
Many applications are like this» Is correct reuse of middleware
possible?[Gill et al., “Integrated Adaptive QoS
Management in Middleware: An Empirical Case Study” (RTAS ‘04)]
[Wang et al., “CAMRIT: Control-based Adaptive Middleware for Real-time Image Transmission” (RTAS ‘04)]
3
Motivation for Formal Models of Middleware
…what's ultimately most important is to determine how things perform for applications that are representative of actual user configurations, rather than micro-benchmarks alone…
ORB configurations have a substantial impact on performance, e.g., TAO's fastest concurrency model is thread-per-connection, rather than thread pool…
Doug Schmidt, ICE/CORBA newsgroup discussion 3/29/05
…a main obstacle to the application of rigorous development techniques is the lack of methodology for relating application software and functional design to physical architecture and implementation…
Joseph Sifakis, “Modeling real-time systems-challenges and work directions”, EMSOFT 2001
The real-time embedded systems community needs formal ways to evaluate the semantics of different middleware configurations
The formal verification community needs accurate middleware models
4
Vanderbilt University and UC Irvine» GME, CoSMIC, PICML, DREAM
UC Berkeley» Ptolemy, E-machine, Giotto
Kansas State University and University of Nebraska» Bogor, Cadena
University of Pennsylvania» VERSA
University of Brest» Cheddar
A Brief Survey of Other Related Work
5
ACE+TAO+CIAO COTS Middleware Architecture
A “Distributed System Software Stack”
EMSOFT (Wednesday, session B9)
modeling and verification focus
CODES+ISSS (Tuesday, session A10)
system software engineering focus
EMSOFT (Monday, session B1)
protocol design and proof focus
TAO+CIAO
ACE
6
Goal: Correct (Re-)use of DRE Middleware
Thesis: formal models of fine-grain middleware building blocks provide a principled approach to analyze timing and liveness properties of middleware-based DRE systems
CADENADREAMRMAPtolemyGMECoSMICUPPAALIF-toolkitSPINBogor
System ModelingSystem Implementation
Formal approach
tocomposition
Reusablesoftwarebuilding blocks
Bridging the Gap:composable, reusable and executable timed automata models of ACE building blocks
7
Another Gap: Between High Level Models and Lower-Level Consequences of their Configuration Choices
Modeling at a high-level of abstraction is helpful» Focus on nuances of the application
Low-level design choices matter too» May invalidate (implicit) assumptions
We focus on low-level frameworks» Capture fundamental concurrency
and timing details of the middleware
CORBA
EJB
CCM
Implementation Platform
OS
Application Models
Collocation strategiesBuffer flushing strategiesCommunication strategiesConnection strategiesConcurrency strategies
Pattern-based frameworks
8
Model Architecture in IF for ACE
Network/OS layer: interprocess communication abstractions Middleware layer: ACE pattern-oriented abstractions Application layer: application-specific semantics within ACE event handlers
9
Modeling Threads Challenge
» No native constructs for threads in model checkers that currently support timed automata
Option 1: model all thread actions as a single automaton» Suitable for high level modeling of application semantics
Option 2: model a thread as multiple interacting automata » Interactions model the flow of control » This option better abstracts the nuances of ACE-level mechanisms
Foo
output method_request
Bar
input method_request
input method_result output method_result
10
Modeling Thread Scheduling Semantics (1/4)
Easy to achieve with one automaton per thread» Specify to model checker directly
E.g., using IF priority rules More difficult with more than one
automaton per thread » Thread of control spans interactions
among automata» Need to express thread scheduling in
terms of execution control primitives provided by the model checker
prio_rule: pid1 < pid2 if pid1 instanceof Activity1 and pid2 instanceof Activity2
Update Display
Control Flow Rate
1 automaton per thread
Activity1 Activity2
Foo
output m_req
Bar
input m_req
input m_result output m_result
2 automata per thread
11
Modeling Thread Scheduling Semantics (2/4) Solution
» Introduce a thread id that is propagated along automata interactions
» Thread id acts as an index to a storage area which holds each thread’s scheduling parameters
1
1
2
2
Foo1 Bar1
Foo2 Bar2
Prio=5 Prio=8
thread_schedule: pid1 < pid2ifpid1 instanceof Foo1 and pid2 instanceof Bar1 and({Foo1}pid1).threadid <> ({Bar1}pid2).threadid and({Thread}(({Foo1}pid1).threadid)).prio <({Thread}(({Bar1}pid2).threadid)).prio )
Give higher preference to the automaton whose “thread” (pointed to by thread id) has higher priority
Foo1 Bar1
Bar2Foo2 Bar1
Foo1
Hint to the model checker
Resulting Behavior
12
Modeling Thread Scheduling Semantics (3/4) What if two threads have the same priority? In an actual implementation, run-to-completion
(SCHED_FIFO) may control the possible interleavings How can we model run-to-completion?
Foo1 Foo2 Foo3
Bar1 Bar2 Bar3
Bar1
Foo1
Foo2
Foo3
Bar1
Bar2
Bar3
Bar1
Bar1
Bar2
Bar3
Foo1
Foo2
Foo3
Foo1
How do we pruneout this space?
13
Modeling Thread Scheduling Semantics (4/4)
Current=nil
Bar1
Foo1
Foo2
Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil
Current=1 Current=2
Current=1
Bar1
Bar1
Solution» Record id of
currently executing thread
» Update when executing actions in each automaton
Current=1
Bar1
Foo3
Current=2
Bar2
Current=2
Bar3
Current=2
Hint to the model checker
14
Problem: Over-constraining Concurrency
Current=nil
Bar1
Foo1
Foo2
Give higher preference to the automaton whose thread is the currently running thread. Non-deterministic choice if Current is nil
Current=1 Current=2
Current=1
Bar1
Bar1
Current=1
Bar1
Foo3
Current=2
Bar2
Current=2
Bar3
Current=2
Hint to the model checker
Current=2
Time progresses
Bar3
Foo3
Bar3 always chosen to run
15
Solution: Idle Catcher Automaton
Key idea: lowest priority “catcher” runs when all others are blocked» E.g., catcher thread in
middleware group scheduling (RTAS ‘05)
Here, idle catcher automaton» runs when all other automata
are idle (not enabled), but before time progresses
» Resets value of current id to nil
Current=2
Current=nil
Time progress
Bar3Foo3
Foo3, Bar3 blockedat this point
Current=nil
Idle catcher runs
Foo3 or Bar3 could be chosen to run. Over-constraining eliminated
16
Problem: Tractability
Model checking can suffer from state space explosion» State space reduction, live variable analysis can help» But even good model checkers don’t fully solve this
Need to think of modeling as a design issue, too» Does the model represent what it needs to represent?» Can the model be re-factored to help the checker?» Can domain specific information help avoid unnecessary checking?
“right away”“in a minute”“get coffee”“go for an espresso”“maybe tomorrow?”
17
Optimization 1: Leader Election Leader/Followers concurrency
» Threads in a reactor thread pool take turns waiting on the reactor
» One thread gets the “token” to access the reactor - leader
» All other threads wait for the token – followers It does not matter which thread gets selected
as leader in a threadpool Model checker not aware of this domain
specific semantics For BASIC-P protocol example, saved factor of
~50 in state space, and factor of ~20 in time
Token to access thereactor is available
Prune this out
T1T2
T3
T2 T3
T3
18
Optimization 2: System Initialization Similar idea, but different technique Iff ok to establish initial object relations
in any order, can optimize away» E.g., 2 server automata, each of which
creates a reactor automaton Useful when modeling object systems
in model checkers with dynamic automaton creation capability (e.g., IF)
State space reduction depends on application» Factor of ~250 for a deadlock scenario
with 2 reactors and 3 threads in each reactor
S1 R1 S2 R1
S1 R1
S2 R2
S2 R1
S1 R2
Prune this out
S1 creates R S2 creates R
S2 creates RS1 creates R
19
Verification of a Real-Time Gateway
An exemplar of many realistic ACE-based applications We modified the Gateway example to add new
capabilities» E.g., Real time, Reliability, Control-Push-Data-Pull» Value added service in Gateway before forwarding a to
consumer E.g. Using consumer specific information to customize data stream
Different design, configuration choices become important » E.g., number of threads, dispatch lanes, reply wait strategies
Gateway
Supplier1
Supplier2
Consumer1
Consumer3
Consumer2
Consumer4
20
Model Checking/Experiment Configuration
Gateway is theoretically schedulable under RMA» Utilization = 80%» Schedulable utilization = 100% for harmonic
periods » Assumption – Messages from 50ms supplier is
given higher preference than 100ms supplier ACE models let us verify scheduling enforcement
» IN THE ACTUAL SYSTEM IMPLEMENTATION
Gateway
S1
C1
S2
C2
C3
C4
20
20
10
10
100ms
50ms
100ms
100ms
50ms
50ms
Period
Relative Deadline
Deadline Exec time
C1 100ms 20ms
C2 100ms 20ms
C3 50ms 10ms
C4 50ms 10ms
Value-addedexecution
(and its cost)
21
Real-time Gateway – Single Thread
Single reactor thread dispatches incoming events I/O (reactor) thread same as dispatch thread I/O thread responsible for value added service
Gateway
SupplierHandler
SupplierHandler
Reactor
ConsumerHandler
ConsumerHandler
ConsumerHandler
ConsumerHandler
Supplier Consumer
22
Real-time Gateway – Dispatch Lanes
Gateway
SupplierHandler
SupplierHandler
Reactor
ConsumerHandler
ConsumerHandler
ConsumerHandler
ConsumerHandler
Supplier Consumer
Single reactor thread again dispatches events to gateway handlers I/O (reactor) thread puts message into dispatch lanes Lane threads perform value added service, dispatch to consumers DO QUEUES HELP OR HURT TIMING PREDICTABILITY?
23
Model/Actual Traces for Real-time Gateway
C1
C1
C2
C2
C3 C4
C3 C4
20 40 6010 30 50
S1,S2 S2
Model
Actual
Single threaded GatewayExecution in the context of reactor thread
Deadline miss for Consumer4because of blocking delay at reactor
Gateway with dispatch lanes
Execution in the context of lane threads
20 40 6010 30 50
C3 C4 C1 C2 C3 C4 C2
C3 C4 C1 C2 C3 C4 C2
70 80 90100Time
C1, C2, C3, C4
C3 C4 C1 C2 C3 C4 C2
20 40 6010 30 50 70 80 90100
Period
Exec time
Deadline
C1
100ms
20ms
100ms
C2
100ms
20ms
100ms
C3
50ms 10ms
50ms
C4
50ms 10ms
50ms
Expected execution timeline with RMS
Time
C3, C4
S1,S2 S2
24
Concluding Remarks Timed automata models of middleware building blocks
» Can be used to verify middleware concurrency and timing semantics
Domain specific model checking optimizations» Can achieve significant reductions in state space
We have run multiple case studies using these models» Allow comparison of model checking & software execution
traces» Demonstrate reusability and fidelity of our models
Future work» Continue to develop reusable models and optimizations» Incorporate domain information directly into model checking
tools » Extend our approach beyond real-time concerns
Model mechanism faults and failure modes Power usage (yet another ~real valued “clock”)
25
Questions?
26
Backup Slides
27
Model checking & actual timing tracesshow BASIC-P protocol’s regulation ofthreads’ use of resources (no deadlock)
EH33
EH23EH13
Timing Traces for BASIC-P Protocol
EH22EH12
R1 R2
EH32
Flow2
R1 R2
Flow3
EH31
EH21EH11
R1 R2
Flow1
28
BASIC-P Blocking Delay ComparisonActual ExecutionModel Execution
Blocking delayfor Client2
Blocking delayfor Client3
29
Overhead of ACE TP/DA reactor with BASIC-P
Negligible overhead with no DA protocol
Overhead increases linearly with # of event handlers due suspend/resume actions on handlers at BASIC-P entry/exit