Patterns for providing End-to-End Real-time Guarantees in DOC Middleware

Wednesday, April 19, 2023

Patterns for providing End-to-End Real-time Guarantees

in DOC Middleware

Irfan [email protected]

PhD Final ExaminationAdvisors: Dr. Ron K. Cytron and

Dr. Douglas C. Schmidt

Washington University, St. Louis

2

Irfan Pyarali Real-time CORBA

Presentation Outline

1. Quick overview of RT-CORBA and thesis scope2. Trace an invocation end-to-end3. Identifying sources of unbounded priority

inversion4. Patterns for providing end-to-end real-time

guarantees5. RT-CORBA architecture6. Empirical evaluation of end-to-end behavior


3


Real-time CORBA Overview• RT CORBA adds QoS control to

regular CORBA improve the application predictability

– Bounding priority inversions– Managing resources end-to-end

• Policies & mechanisms for resource configuration/control in RT-CORBA include:Processor Resources– Thread pools– Priority models– Portable prioritiesCommunication Resources– Protocol policies– Explicit bindingMemory Resources– Request buffering

• These capabilities address some (but by no means all) important real-time application development challenges

ClientOBJREF

Object(Servant)

in argsoperation()

out args + return

IDLSTUBS

IDLSKEL

Object Adapter

ORB CORE GIOP

Protocol Properties

End-to-End PriorityPropagation

ThreadPools

StandardSynchronizersExplicit

Binding

Portable Priorities

SchedulingService


4


Thesis Scope• End-to-end QoS

guarantees require vertical and horizontal integration – Peer-to-Peer– Network adapter to

Application layer

• Thesis focus areas:– End-to-end priority

propagation– Demultiplexing– Dispatching– Concurrency


5


Related ResearchArea Research

Presentation Layer

Time/space tradeoff: compiled vs interpreted stubs [19]

Data Copying and

Memory Allocations

Using gather-write & scatter-read I/O calls to avoid excessive data copying; Using stack & TSS allocators to avoid heap allocations [43]

Transport Protocols

Principles for optimizing CORBA IIOP performance [18], and AITPM [39], synergistic combination of IP and ATM technologies to designed a highly scalable gigabit IP router

Manage Connections

Non-multiplexed connection model to avoid priority inversion [52]

OS Scheduling

Empirical evaluation of context switching overhead and priority inversions overhead for different real-time and general purpose operating systems [29]

I/O Subsystems

RIO [27], an extensible and predictable I/O framework that can integrate seamlessly with real-time middleware

Network Adapters

APIC [7], a high-performance ATM Port Interface Controller that supports efficient zero-copy buffer management that shares request buffers across OS protection domains


6


Reply Demultiplexer

Tracing an Invocation End-to-end

Connection Cache Memory Pool

ConnectorReactor

A B

CV1

CV2

A B SConnection Cache Memory Pool

AcceptorReactor

X Y

X Y C

POA

S

S

1. Lookup connection to Server (S)2. Lookup failed – make new connection to Server (S)

3. Add new connection to cache4. Add new connection to Reactor 5. Accept new connection from Client (C)

6. Add new connection to Cache7. Add new connection to Reactor

C

8. Wait for incoming events on the Reactor

9. Allocate memory to marshal data

10. Send data to Server – marking connection (S) busy

11. Wait in Reply Demuliplexer –some other thread is already leader, become follower

C

12. Read request header13. Allocate buffer for incoming request

14. Read request data15. Demultiplex request and dispatch upcall

16. Send reply to client17. Wait for incoming events

C

18. Leader reads reply from server

S

S

CV1

19. Leader hands off reply to follower

S

20. Follower unmarshals reply

CV1

Client ORB Server ORB


7


Reply Demultiplexer

Identifying Sources of Unbounded Priority Inversion


ConnectorReactor

A B

CV1

CV2

A B SConnection Cache Memory Pool

Acceptor

X Y C

Client ORB Server ORB

Connection cache– Time required to send request is dependent on availability of network

resources and the size of the request– Priority inheritance will help– Creating new connections can be expensive and unpredictable

Memory Pool– Time required to allocate new buffer depends on pool fragmentation and

memory management algorithm– Priority inheritance will help

Reply Demultiplexer– If the leader thread is preempted by a thread of higher priority before the

reply is handed-off (i.e., while reading the reply or signaling the invocation thread), then unbounded priority inversion will occur

– There is no chance of priority inheritance since signaling is done through condition variables

Reactor– No way to identify high priority client request from one of lower priority

Reactor

X Y CS

POA

POA– Time required to demultiplex request may depend on server organization– Time required to dispatch request may depend on contention on dispatching

table– Priority inheritance will help


8


ORB I/O

TiggerPooh

Root POA

Simpsons

Homer Lisa

Winnie the Pooh

Family

Mr. Burns Moe

Townspeople

Fox Disney

HankBobby

King of the Hill

Doh

!

Dri

nk B

eer

Pla

y S

axop

hone

Sel

l Pro

pane

Eat

Hon

ey

Bou

nce

Mak

e M

oney

Sel

l Bee

r

Pla

y P

oohS

tick

s

Run

Nuc

lear

Pla

nt

Impr

ess

Gir

ls

Wor

k at

Nuc

lear

Pla

nt

POA Layer

Servant Layer

Skeleton Layer

ORB Core LayerFox/Simpsons/FamilyPOA Id

Homer

ServantId

Doh!

OperationName

CORBA Request Demultiplexing

HashMap

HashMap

HashMap

HashMap

HashMap


9


Problems with Request Demultiplexing• Slow demultiplexing

– Five hash lookups required• Unacceptable worst-case time

– Hash map searches are O(n) worst-case• Lookup time dependent on

– Height of POA hierarchy– Number of POAs– Number of Servants– Number of Operations

• Required demultiplexing– Independent of server organization and configuration– Predictable, efficient and scalable


10


Skeleton DemultiplexingForces• Skeletons names known at

compile-time• Cannot change representation

– CORBA requires names to be transmitted as strings

Homer

Doh!

Drink Beer

Work at Nuclear Plant

Play trick on Ned

Bowl

Bowl

Doh!

Work at Nuclear Plant

Play trick on Ned

Drink Beer

Perfect Hash MapSolution• Use GPERF to generate Perfect

Hash function offline• Worst-case time O(1)


11


Servant DemultiplexingForces• Servants can be added and

removed on the fly• Can change representation

– Object reference is ORB specific

Family

Marge

Bart

Maggie

Lisa

Homer

Solution• Use Active Object Map• Worst-case time O(1)

Active Map

IndexGeneration

CountServant

0 5 Homer

1 2 Marge

2 1 Bart

3 10 Lisa

4 3 Maggie


12


Simpsons

Family

Townspeople

Fox

King of the Hill

POA Demultiplexing

Fox / Simpsons

Fox / Simpsons /

Family

Fox / Simpsons /

Townspeople

Fox

Fox / King of the Hill

Forces• POAs can be added and

removed on the fly• Can change representation

– Object reference is ORB specific

Solution• Flatten POA hierarchy

• Logically still the same

• Use Active Object Map

• Worst-case time O(1)

Active Map

IndexGeneration

CountPOA

0 5 Fox

1 2Fox /

Simpsons

2 1Fox /

Simpsons / Family

3 10Fox /

Simpsons / Townpeople

4 3Fox /

King of the Hill


13


ORB I/O

TiggerPooh

Fox / Simpsons

Homer Lisa

Disney / Winnie the Pooh

Fox / Simpsons /

Family

Mr. Burns Moe

Fox / Simpsons /

TownspeopleFox Disney

HankBobby

Fox / King of the Hill

Doh

!

Dri

nk B

eer

Pla

y S

axop

hone

Sel

l Pro

pane

Eat

Hon

ey

Bou

nce

Mak

e M

oney

Sel

l Bee

r

Pla

y P

oohS

tick

s

Run

Nuc

lear

Pla

nt

Impr

ess

Gir

ls

Wor

k at

Nuc

lear

Pla

nt

POA Layer

Servant Layer

Skeleton Layer

ORB Core Layer2:1

POA Id

0:5

ServantId

Doh!

OperationName

Revised Request Demultiplexing

Root POAActiveMap

ActiveMap

PerfectHash Map


14

Irfan Pyarali Real-time CORBASummary of Revised Request

DemultiplexingDemultiplexing Stage Absolute Time (μs)

1. Parsing Object Key 2

2. POA demux 2

3. Servant demux 3

4. Operation demux 3

5. Parameter demarshal Operation dependent

6. User upcall Servant dependent

7. Return value marshal Operation dependent

Pyarali, et al., Applying Optimization Principle Patterns to Real-time ORBs COOTS, May 1999


15


Request Dispatching Strategies

PatternNested Upcalls

Priority Inversion

Appropriate when

Serialized Dispatching

No UnboundedLittle/no contention Short-lived upcalls

Recursive Lock

Yes Unbounded Same as above

Readers/Writer Lock

Limited Unbounded Concurrent Upcalls

Required Yes BoundedPredictable Behavior


16


Reference Count during DispatchProblem• Satisfy MT applications with stringent real-time requirements

– Lock cannot be held for the duration of an upcall– The target element cannot be removed while an upcall is in progress– Other objects can be added or removed while an upcall is in process– Bound all priority inversions – Increase concurrency, allowing simultaneous upcalls

MONITOR LOCK

DIS PATCH ING TABLE

UPC AL L &MODIFI CATION

THREADS

1 2 10 0 0 000

Pyarali, et al., A Pattern Language for Efficient, Predictable, Scalable, and Flexible Dispatching Mechanisms for DOC MiddlewareISORC, March 2000

Solution• Count the current number of upcalls on each entry• Events are dispatched as follows:

1. Acquire lock, locate entry, increase reference count, release lock2. Perform upcall3. Re-acquire lock, decrement reference count, release lock

• Once reference count reaches zero, element can be safely removed– Reference count is greater than zero during upcall

Consequences• Upcalls can execute concurrently and can safely use the dispatching

table• Priority inversion independent of upcall duration• Can collaborate in object life-cycle management• It requires two locks per upcall


17


Half-Sync/Half-Async Thread Pool Design

Design Overview• Single Acceptor endpoint• One Reactor for each

priority level• Each lane has a queue• I/O and application

processing are done in different threads


18


Evaluation of Half-Sync/Half-Async Thread Pools

Criteria Evaluation

Feature SupportGood: supports request buffering and thread borrowing

Scalability Good: I/O layer resources shared

EfficiencyPoor: high overhead for data movement, context switches, memory allocations, & synchronizations

Optimizations Poor: cannot use stack and TSS memory Priority Inversion Poor: some unbounded, many bounded


19


Comparing Memory Management SchemesActions and Properties

Global Memory Pool

TSS Memory Pool Stack Memory Pool

Malloc 1. Acquire Lock 2. Manipulate Pool 3. Release Lock

1. TSS Access 2. Manipulate Pool

1. Stack Allocation

Free 1. Acquire Lock 2. Manipulate Pool 3. Release Lock

1. TSS Access 2. Manipulate Pool

Nothing

Share Memory Yes No No

Contention High None None

Size Variable Variable Fixed


20

Irfan Pyarali Real-time CORBAPerformance of Memory Management

Schemes

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4

Competing Threads

No

rmal

ized

Th

rou

gh

pu

t

Global Memory Pool TSS Memory Pool Stack Memory Pool

• Performance order: Stack > TSS > Global• Predictability order: Stack = TSS > Global• Contention greatly effects Global Memory Pool

• No effect on Stack and TSS Memory Pools


21


Leader/Followers Thread Pool Design

Design Overview• Each lane has its

own set of resources– Reactor/Acceptor

• I/O and application processing are done in the same thread


22


Evaluation of Leader/Followers Thread Pools

Criteria Evaluation

Feature SupportPoor: not easy to support request buffering or thread borrowing

Scalability Poor: I/O layer resources not shared

EfficiencyGood: little or no overhead for data movement, memory allocations, or synchronizations

Optimizations Good: can use stack and TSS memory Priority Inversion Good: little or no priority inversion


23


Performance of Thread Pool Implementations

Operations

Times called for HS/HA

Times called for LF

Malloc 2 0

Free 2 0

Enqueue 2 0

Dequeue 2 0

Signals 2 1

Locks 8 20

5000

10000

15000

20000

25000

30000

35000

40000

45000

1 2 3 4

Threads in Pool

Th

rou

gh

pu

t

Leader / Follow ers Half Sync / Half Async

Pyarali, et al., Optimizing Thread-Pool Strategies for Real-Time CORBA Optimizing Middleware ACM Workshop, June 2001


24


Leader/Followers

RT-CORBA Architecture


Connector

ReactorA B

CV1

CV2

A B S

ORB A

S

Acceptor

High Priority Lane

Leader/Followers


ReactorA B

CV1

CV2

A B S

S

Acceptor

Low Priority Lane

Leader/Followers


ReactorA B

CV1

CV2

A B S

ORB B

S

Acceptor

High Priority Lane

Leader/Followers


ReactorA B

CV1

CV2

A B S

S

Acceptor

Low Priority Lane

POA Connector POA


25


IOR Creation and Endpoint Selection

Model CLIENT PROPAGATED

Bands 0-15 and 45-55

0-15 ENDPOINT A

45-55 ENDPOINT C

10 ENDPOINT A

20 ENDPOINT B

50 ENDPOINT C

Model CLIENT PROPAGATED

Bands None

Model SERVER DECLARED

Bands Any

ENDPOINT B

ENDPOINT C

Model EITHER

Bands Any

ENDPOINT D


26


Motivation for Real-time Experiments• Illustrate RT, deterministic, and predictable ORB behavior• Demonstrate end-to-end predictability by utilizing the ORB to

– Propagate and preserve priorities– Exercise strict control over the management of resources– Avoid unbounded priority inversions

• End-to-end predictability of timeliness in fixed priority CORBA – Respecting thread priorities between client and server for

resolving resource contention during the request processing– Bounding the duration of thread priority inversions end-to-end– Bounding the latencies of operation invocations


27


Test Bed Description void method (in unsigned long work);

Network

Name hermes.doc.wustl.edu

OS Linux 2.4 (Redhat 7.1)Processor (1) Intel Pentium III 930 MHzMemory 500 MegabytesCPU Cache 256 KB

ORBSchedPolicy SCHED_FIFOORBScopePolicy SYSTEMORBPriorityMapping linear

High Priority Lane 32767Medium Priority Lane 21844Low Priority Lane 10922Best Effort Lane 0

Hardware Profile

Threads Profile

Priority Profile


28


1. Increasing Workload in Vanilla CORBA2. Increasing Workload in RT-CORBA With Lanes

• Increasing Priority Increasing Rate

3. Increasing Workload in RT-CORBA With Lanes• Increasing Priority Decreasing Rate

4. Increasing Best-effort Work in Vanilla CORBA5. Increasing Best-effort Work in RT-CORBA With

Lanes6. Increasing Workload in RT-CORBA Without Lanes

Description of Experiments


30


Experiment 1: Increasing Workload in Vanilla CORBA

Experiment• Measure the disruption caused by

Increasing Workload in Vanilla CORBA


Server• 3 threads

Client• 3 Rate-based Invocation threads

– High 75 Hertz– Medium 50 Hertz– Low 25 Hertz


31


Results: Increasing Workload in Vanilla CORBA

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Invo

cati

on

s / s

ec

25 Hertz (low priority) 50 Hertz (medium priority) 75 Hertz (high priority)


32

Irfan Pyarali Real-time CORBAConclusions: Increasing Workload in Vanilla

CORBA• As workload increases and system capacity

decreases, the high priority 75 Hertz client is effected first, followed by the medium priority 50 Hertz client, and finally by the low priority 25 Hertz client

• The above behavior is because all clients are treated equally by the server

• Behavior is unacceptable for a real-time system


33


Experiment 2: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Increasing Rate)


Increasing Workload in RT-CORBA With Lanes


Server• 3 thread lanes

– High / Medium / Low




34


Results A: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Increasing Rate)

(Client and Server on same machine)

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Inv

oc

ati

on

s /

se

c



35


Results B: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Increasing Rate)

(Client and Server on remote machines)

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Inv

oc

ati

on

s /

se

c

25 Hertz (low priority) 50 Hertz (m edium priority) 75 Hertz (high priority)


36


Conclusions: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Increasing Rate)

• As workload increases and system capacity decreases, the low priority 25 Hertz client is effected first, followed by the medium priority 50 Hertz client, and finally by the high priority 75 Hertz client

• The above behavior is because higher priority clients are given preference over lower priority clients by the server

• When client and server are on separate machines, the lower priority client threads are able to sneak in some requests between the time a reply is sent to the high priority thread and before a new request is received from it

• Behavior is consistent for a real-time system


37


Experiment 3: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Decreasing Rate)


Increasing Workload to RT-CORBA With Lanes

• Increasing Priority Decreasing Rate


– High / Medium / Low




38


Results: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Decreasing Rate)

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Invo

cati

on

s / s

ec

25 Hertz (high priority) 50 Hertz (medium priority) 75 Hertz (low priority)


39


Conclusions: Increasing Workload in RT-CORBA With Lanes (Increasing Priority Decreasing Rate)

• As workload increases and system capacity decreases, the low priority 75 Hertz client is effected first, followed by the medium priority 50 Hertz client, and finally by the high priority 25 Hertz client

• The above behavior is because higher priority clients are given preference over lower priority clients by the server



40

Irfan Pyarali Real-time CORBAExperiment 4: Increasing Best-effort Work in Vanilla

CORBAExperiment• Measure the disruption caused by the

Increasing Best-effort Work in Vanilla CORBA


Server• 4 threads



• Several Best-effort threads Continuous Invocations

Notes• System is running at capacity Any

progress made by Best-effort threads will cause disruptions


41


0

20

40

60

80

100

120

140

0 1 2 3 4 5 6 7 8 9 10

Continuous Workers

Invo

cati

on

s / s

ec

25 Hertz (high priority) 50 Hertz (medium priority)

75 Hertz (low priority) Continuous Workers

Results: Increasing Best-effort Work in Vanilla CORBA


42

Irfan Pyarali Real-time CORBAConclusions: Increasing Best-effort Work in Vanilla

CORBA• All three priority based clients suffer as the

number of best-effort clients are added to the system

• The above behavior is because all client threads are treated equally by the server

• Behavior is unacceptable for a real-time system


43

Irfan Pyarali Real-time CORBAExperiment 5: Increasing Best-effort Work in RT-CORBA With

LanesExperiment• Measure the disruption caused by

Increasing Best-effort Work in RT-CORBA With Lanes



– High / Medium / Low / Best-effort



• Several Best-effort threads Continuous Invocations

Notes• System is running at two levels

– At capacity Any progress by Best-effort threads will cause disruptions

– Just below capacity Best-effort threads should be able to capture any slack in the system


44


Results A: Increasing Best-effort Work in RT-CORBA With LanesSystem Running at Capacity (Work = 30)


0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6 7 8 9 10

Continuous Workers

Invo

cati

on

s / s

ec

25 Hertz (low priority) 50 Hertz (medium priority)

75 Hertz (high priority) Continuous Workers


45


Results B: Increasing Best-effort Work in RT-CORBA With LanesSystem Running Slightly Below Capacity (Work = 28)


0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6 7 8 9 10

Continuous Workers

Invo

cati

on

s / s

ec

25 Hertz (low priority) 50 Hertz (medium priority)



46


Results C: Increasing Best-effort Work in RT-CORBA With LanesSystem Running Slightly Below Capacity (Work = 28)

(Client and Server on remote machines)

0

10

20

30

40

50

60

70

80

0 1 2 3 4 5 6 7 8 9 10

Continuous Workers

Inv

oc

ati

on

s /

se

c

25 Hertz (low priority) 50 Hertz (m edium priority)



47

Irfan Pyarali Real-time CORBAConclusions: Increasing Best-effort Work in RT-CORBA With

Lanes

• Addition of best-effort client threads did not effect any of the three priority based clients

• Best-effort client threads were limited to picking up slack left in the system

• As the number of best-effort client threads increase, throughput per best-effort client thread decreases, but the collective best-effort client throughput remains constant

• When client and server are on separate machines, there is more slack in the system since all the client-side processing is done on another machine



48


Experiment 6: Increasing Workload in RT-CORBA Without Lanes


Increasing Workload in RT-CORBA Without Lanes


Server• 3 threads in pool



Notes• Server pool priority will be varied

– Low / Medium / High


49


Result A: Increasing Workload in RT-CORBA Without LanesServer Pool Priority = Low

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Invo

cati

on

s / s

ec



50


Result B: Increasing Workload in RT-CORBA Without LanesServer Pool Priority = Medium

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Invo

cati

on

s / s

ec



51


Result C: Increasing Workload in RT-CORBA Without LanesServer Pool Priority = High

0

10

20

30

40

50

60

70

80

25 30 35 40 45 50 55 60 65 70 75 80 85

Work

Invo

cati

on

s / s

ec



52


Conclusions: Increasing Workload in RT-CORBA Without Lanes

• When pool priority is low, a pool thread cannot preempt an upcall thread when a new request arrives from a client thread of higher priority. Therefore, all three client threads receive similar service from the server

• When pool priority is medium, a pool thread can only preempt an upcall thread running at low priority when a new request arrives from a client thread of medium or high priority. Therefore, the medium and high priority client threads receive similar service from the server

• When pool priority is high, a pool thread can preempt upcall threads running at low and medium priorities when a new request arrives from a client thread of high priority. Therefore, the high priority client thread receives the best service from the server

• Most desirable behavior when pool priority is high• Thread pool without lanes are more flexible than thread pool with lanes• However, thread pool without lanes can incur very high or unbounded

priority inversions in some cases


53


Concluding Remarks• End-to-end QoS guarantees requires careful

engineering of subsystems to ensure:– Predictability, scalability, and performance– Avoid unbounded priority inversion

• Demultiplexing, Dispatching, and Concurrency are key subsystems in the critical path

• Must be coupled with end-to-end priority propagation• Thesis presents:

– Patterns that address these challenges – Empirical evaluation of end-to-end behavior


54

Irfan Pyarali Real-time CORBAPattern identification goes beyond

CORBA• Distributed, real-time applications developers confront common challenges • Patterns can capture key design and performance characteristics of proven

mechanisms– Reuse successful strategies– Communicate design alternatives and decisions – Evaluate proposed designs and implementations– Careful application of patterns can yield efficient, predictable, scalable, and

flexible middleware• Create pattern handbook for developing real-time DOC middleware


55


TAO’s SuccessOrganization Domain

Boeing Aircraft Mission Control Computer

SAIC Distributed Interactive Simulation (HLA/RTI-NG)

ATD Automated Stock Trading

Raytheon Aircraft Carrier self-defense Systems

Cisco High-performance network switch control

US Army Joint Tactical Terminal

Contact Systems Surface mounted “pick-and-place” machines

Turkish Navy Shipboard Resource Management

Krones Process Automation

Siemens Hot Rolling Mill Control System

Lockheed-Martin Shipboard Resource Management

CUSeeMe Monitor H.323 Servers

Northrup-Grumman

Airborne Early Warning & Control

JPL/NASA SOFIA Airborne Telescope, Cassini Space Probe

Marconi Joint Tactical Radio System


56


Future Work• Integrate with Scheduling Services and Resource

Managers– Including Dynamic Scheduling (RT-CORBA 2.0)

• Integrate with Network QoS mechanisms– DiffServ and RSVP

• Integrate with higher level services– Real-time Event Service– Real-time Notification Service– CORBA Component Model

• Integrate with monitoring tools– TimeWiz, TotalView

• Integrate with fault-tolerance


57


Acknowledgements• Thanks to God Almighty for His help, blessings, and mercy which made

this long journey possible• Thanks to my family for giving me unparalleled encouragement and love• Thanks to my advisors, professors, and mentors for their guidance and

wisdom• Thanks to my friends and coworkers for their friendship and respect• And thanks to Washington University for a decade of enlightenment

Pyarali Family

Mohammad Hussain, Gulshan, Yasra, Shahid, Arif, and Asif

Advisors and

Mentors

Dr. Douglas C. Schmidt, Dr. Ron K. Cytron, Dr. David L. Levine, Dr. Christopher D. Gill, Dr. G. James Blaine, Dr. Douglas Niehaus, Dr. Robert B. Pless, Dr. Jeremy Buhler, Stephen M. Moore, Chris Cleeland, Fred Kuhns, Dr. Karlheinz Dorn, Dr. Ebrahim Moshiri, Dr. Richard Schantz, Dr. Joseph Loyall, Craig Rodrigues, Stephen T. Bachinsky, Dr. Russ Noseworthy, and Steve Huston

Friends

Balachandran Natarajan, Carlos O'Ryan, Marina Spivak, Angelo Corsaro, Pradeep Gore, Jeff Parsons, James Hu, Tim Harrison, Nanbor Wang, Aniruddha Gokhale, Yamuna Krishnamurthy, Prashant Jain, Joe Hoffert, Sumedh Mungee, Michael Kircher, Ossama Othman, Darrell Brunsch, Vishal Kachroo, Kirthika Parameswaran, Sharath Cholleti, and Krishnakumar Balasubramanian

Documents

Patterns for providing End-to-End Real-time Guarantees in DOC Middleware