The New Dyninst Event Model

University of Maryland

The New Dyninst Event Model

James Waskiewicz


Motivation: the series

Several years in the making: Paradyn separation requires… DyninstAPI support for threads

– Among other things

Supporting threads (a la Paradyn) requires…– Asynchronous event handling, which requires…

Solving the multi-event-source problem– Eg. Listen for and handle events coming from

• Socket / file descriptor• System call ( wait() )

– Solution: use threads• Just a can of worms …• Or Pandora’s box?


Overall Design Concerns

Keep thread stuff out of the way– Don’t want to “pollute” dyninst code with

synchronization primitives– Dyninst is already hard to fix when broken

• Keep threading issues modular Try to keep existing code thread-

context free– In other words, most dyninst code should be

able to run successfully no matter what thread executes it.

– There can be some tricky situations here (inferior RPCs for example)

– Obvious exceptions in signal handling


The Plan: The 3 thread model

3 threads– (1) User Interface Thread

• API user’s mutator– (1) Synchronous Event Thread

• Eg. Loop on waitpid()– (1) Asynchronous Event Thread

• Eg. Loop on select()

And a mailbox– Hide internal threads by forcing non UI

threads to “mail” events to the UI thread.• Eg. ensure that user-supplied callbacks

are only executed on the UI thread.



UI Thread Mutatee 1

Mai

lbox

Synchronous Event Thread

waitpid()

poll()

Asynchronous Event Thread

select()

Mutatee 2

Mutatee N


But there’s a problem (linux)

Ptrace() may only be called by one thread, period.– But we really want all threads to be able to

issue ptrace() commands• Process::pause()• Write/readDataSpace()

Solution: Add another thread– Solely responsible for accepting and

handling requests for calling ptrace()– And sometimes waitpid() and fork() …



UI Thread Mutatee 1

Mai

lbox

Synchronous Event Thread

waitpid()

poll()


select()

Mutatee 2

Mutatee N

Ptracer (DBI) Thread


But there’s another problem (windows)

Restrictive OS level debugging API– The thread that starts the mutatee process

is the only thread that is allowed to call WaitNextDebugEvent()

Solution: add N threads !– N = number of mutatees

This is actually a good idea– One “listener” thread per process

• Elegant in terms of code modularity• A nice simple concept: a thread that

starts a process and receives events from it until termination/detach

• Handles events as they occur.


The Plan: The (n+3) thread model

UI Thread

Mai

lbox

Synchronous Event Thread 1waitpid()Poll()


select()

Mutatee 2

Ptracer (DBI) Thread Mutatee 1

Mutatee N

Synchronous Event Thread 2waitpid()Poll()

Synchronous Event Thread Nwaitpid()Poll()


But there’s another problem: recursion

Dyninst is full of recursive event patterns– ie. An event that causes another event, and needs to wait

for its completion Best solution will involve encapsulating this behavior

into a “natural” structure– Either that or re-architect many functional elements of dyninst– And don’t forget: add more threads!

Solution: decouple event handling concept into:– Event generation

• One dedicated thread (per process)– Event handling

• Multiple possible threads (thread pool)• Number of threads depends on recursion depth

This is a bit nasty– Could probably be done using the mailbox’s recursive

callback features, but…• They’re complicated too, and need more thought


The Plan: The (2n+3+x) thread model

UI Thread

Mai

lbox

Signal Generator Thread 1


select()

Mutatee 2

Ptracer (DBI) Thread Mutatee 1

Mutatee N

SHSH SHSH SHSH


SHSH SHSH SHSH


SHSH SHSH SHSH


A simple example: full stop

UI Thread issues stopExecution()– Sends SIGSTOP to mutatee– Blocks inside waitForStop()

UI Thread

kill (SIGSTOP)

Mutatee

Signal GeneratorThread

Signal Handler Thread

Eve

nt G

ate

Wait for event:evtProcessStopblock


A simple example: full stop Mutatee Stops

– Notifes parent process of SIGSTOP– This is received by SignalGenerator, which is listening for mutatee events

UI Thread

kill (SIGSTOP)

Mutatee



SIGSTOP

Eve

nt G

ate



A simple example: full stop Signal Generator decodes SIGSTOP and assigns event

– Creates an Event with platform independent type evtProcessStop– Assigns Event to available SignalHandler Thread

UI Thread

kill (SIGSTOP)

Mutatee



SIGSTOP

evtProcessStop

Eve

nt G

ate



A simple example: full stop Signal Hander handles evtProcessStop

– Platform independent code for handling evtProcessStop– Signal waiting EventGates that evtProcessStop happened

UI Thread

kill (SIGSTOP)

Mutatee



SIGSTOP

evtProcessStop

Eve

nt G

ate

Wait for event:evtProcessStop

Signal event:evtProcessStop

block


A simple example: full stop Event Gate receives target event

– Release UI Thread from block

UI Thread

kill (SIGSTOP)

Mutatee



SIGSTOP

evtProcessStop

Eve

nt G

ate

Wait for event:evtProcessStop

Signal event:evtProcessStop

resume

block


A (slightly) more complex example User registers a DynLibrary Callback

– When a library is loaded, insert some instrumentation

UI Thread Mutatee


SH - 0

CB Manager

register callback

Mailbox

SH - 1


A (slightly) more complex example Continue mutatee execution

– Wait for it to exit– Mutatee runs

UI Thread Mutatee


SH - 0

Block until exit event

CB Manager

register callback

Continue execution Mailbox

kill (SIGCONT)

SH - 1

Event Gate


A (slightly) more complex example Eventually, mutatee executes dlopen()

– Stops, sending SIGTRAP to mutator

UI Thread Mutatee


SH - 0

SIGTRAP


CB Manager

register callback


kill (SIGCONT)

dlopen()

SH - 1

Event Gate


A (slightly) more complex example SignalGenerator Decodes SIGTRAP

– Analyzes runtime link maps, finds new library– Sends an evtLoadLibrary Event to a Signal Handler

UI Thread Mutatee


SH - 0

SIGTRAP

evtLoadLibrary


CB Manager

register callback


kill (SIGCONT)

dlopen()

SH - 1

Event Gate


A (slightly) more complex example SignalHandler handles evtLoadLibrary

– Adjusts Dyninst Internal State (platform indep)– Obtains relevant Callback from CBManager– “Executes” loadLibrary callback (zoom in to mailbox)

UI Thread Mutatee


SH - 0

SIGTRAP

evtLoadLibrary


CB Manager

register callback


dlopen()

SH - 1

Event Gate


A (slightly) more complex example Callback Execution (zoom in):

– Registers instance of callback with Mailbox (target = UI Thread)– Signals Event Gates that evtLoadLibrary happened– Blocks until completion of callback

UI Thread

SH - 0Block until exit event

Mailbox

Event Gate

CB

Block until CB complete

Signal evtLoadLibrary

Register CB


A (slightly) more complex example UI Thread, signalled, wakes up

– Checks mailbox and finds callback with target_thread == UIThread– But the Event does not match target– Executes callback (zoom back out… )

UI Thread

SH - 0Block until exit event

Mailbox

Event Gate

CB

Block until CB complete

Signal evtLoadLibrary

Register CB

Wake up, Execute Callbacks

CB


A (slightly) more complex example But the callback triggers another event

– InsertSnippet needs to allocate memory before insertion– Issues inferiorMalloc RPC and blocks for its completion

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()

SH - 1Inferior Malloc RPC

Insert:

resumeblock

Event Gate

Event Gate

block


A (slightly) more complex example Mutatee issues malloc() and traps SignalGenerator receives SIGTRAP

– Discovers that we just completed a RPC

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

Event Gate

malloc()

block

SIGTRAP


A (slightly) more complex example SignalGenerator sees that SH-0 is busy (blocked)

– Assigns evtRPCSignal event to SH-1

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

Event Gate

malloc()

block

SIGTRAP

evtRPCSignal


A (slightly) more complex example SH -1 handles RPC completion event

– Manages Dyninst internal state (RPC Manager)– Signals Event Gates that evtRPCSignal happened

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

Event Gate

malloc()

block

SIGTRAP

evtRPCSignal

evtRPCSignal


A (slightly) more complex example EventGate waiting for evtRPCSignal wakes up

– Releases UI thread from RPC block

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

Event Gate

malloc()

block

SIGTRAP

evtRPCSignal

evtRPCSignal


A (slightly) more complex example Callback finishes executing

– Inserts instrumentation in newly malloc()’d space– SH-0 is released from blocking state

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

Event Gate

malloc()

resume

SIGTRAP

evtRPCSignal

evtRPCSignal


A (slightly) more complex example UI Thread resumes blocking for process exit

– Inserts instrumentation in newly malloc()’d space– SH-0 is released from blocking state

UI Thread Mutatee


SH - 0


CB Manager

register callback


dlopen()


Insert:

resumeblock

Event Gate

malloc()


A (slightly) more complex example Finally, Mutatee Exits

– evtProcessExit propagates through Event Handling– UI Thread is released from block

UI Thread Mutatee


SH - 0resume


CB Manager

register callback


dlopen()

exit()SH - 1

Event Gate

malloc()

evtProcessExit

Signal event: evtProcessExit


Thread Inflation aside…

Good things have emerged too… Unified Dyninst event concept

– All event oriented subsystems operate on the same generic event concept• Well, its really just a new c++ class

Unified Dyninst event handling– Event handling framework is now platform

independent– Dyninst takes uniform actions when specific

events occur, across platforms• More or less• But more than before


Implications (future features)

Finer grained Callback APIs– We currently have callbacks for events like

• LoadLibrary• Process Exit• Thread Creation

– Now possible (and easy) to expose generic event callback to user• Eg. Execute func() when mutatee

receives signal S• Eg. Execte func() when mutatee issues

dlclose()


Implications (future features)

Spin-off “Debugger API” library– Separate out code for process control,

signal handling, other debugger-like operations into shared library

– Requires solving some complex problems:• Eg. How to deal with RPCs

How much gets spun off?– This is a hard question to answer– Related RPC and shared-object subsytems

are heavily intertwined.– Where to draw the line?

Documents

The New Dyninst Event Model