Mago DebuggerInner Workings
An implementation overviewBy Aldo Núñez
Mago Debugger
What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger
What is debugging?
Run/Attach Control Inspect Why?
Find out the cause of a problem (bug)
Run/Attach
Kick off a process Attach to an already running process
Control
Breakpoints Stepping Changing instruction pointer Suspend and resume threads
Inspect
Callstack Loaded modules Threads Memory Registers Variables Expressions
Mago Debugger
What is debugging?What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger
What is a debugger?
A process that runs, controls, and inspects another process
Special relationship between debugger and debuggee
System notifies debugger of events taking place in debuggee
OS APIs v. Hardware
OS• Debug events
• Break/Run mode
• Change state
HW• Single Step• Breakpoints
• Registers
Break and Run Mode
Break Run
Create or Attach
Break Event
Continue
A Windows Debugger
Loopevent ← WaitForDebugEvent( timeout )if got event
ContinueDebugEvent( event.pid,event.tid,
DISCARD_EXCEPTION )Until event.code = EXIT_PROCESS
Debug Events
Start Process Exit Process Start Thread Exit Thread Load Module Unload Module Exception Message
Windows API for Debugging
Debug Events• WaitForDebugEvent• ContinueDebugEvent
Launch• CreateProcess• TerminateProcess
Attach• DebugActiveProcess• DebugActiveProcessS
top
Registers• GetThreadContext• SetThreadContext
Memory• ReadProcessMemory• WriteProcessMemory
Threads• SuspendThread• ResumeThread
OS and HW Co-op (x86): Single Step
Enable SS
• GetThreadContext• context.Eflags or
0x100• SetThreadContext• ContinueDebugEvent
SS Event
• event ← WaitForEvent• event.code =
EXCEPTION• event.exception.code
= EXCEPTION_SINGLE_STEP
OS and HW Co-op (x86): Breakpoint
Enable BP
• ReadProcessMemory• WriteProcessMemory
• data = 0xCC (int 3)• FlushInstructionCache• ContinueDebugEvent
BP Event
• event ← WaitForEvent• event.code =
EXCEPTION• event.exception.code
= EXCEPTION_BREAKPOINT
• Except. address = original
• EIP = original + 1
Mago Debugger
What is debugging? What is a debugger?What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger
What is Mago?
A debugger for D programs A set of independent libraries A Visual Studio plug-in
History
Interest in debuggers since 2005 Started September 2009 Source code released August 2010 Integrated into Visual D September
2010
Libraries v. Visual Studio plug-in Benefits to making separate
components Targeted testing Mix and match for different purposes Use with any shell program
Benefits to making VS plug-in Well tested shell program already
written High level debug programming model
What does Mago look like?
Debug Engine
Exec Expr Syms
Component Responsibilities
Exec•Control debuggee•Read and change state
Expr•Evaluate D Expressions•Formatting
Syms•Read debug info
DE•Combine other components•Expose AD7 interface
Mago Debugger
What is debugging? What is a debugger? What is Mago?Execution Agent Expression Evaluator Symbol Reader Debug Engine D and the debugger
Execution Agent
Abstracts run, control, and inspection services
Built first to make it as solid as possible
Many APIs are locked to thread that started debuggee Because of underlying Windows API
Services
WaitForEvent, Continue from event Launch, Terminate Attach, Detach Read, Write Memory Set, Remove Breakpoint Step, Cancel Step Async Break
Breakpoint Management
Software breakpoint abstraction Hardware breakpoint abstraction Breakpoint sharing Resuming from breakpoint
Breakpoint Lifecycle
inc
mov
inc
mov
inc
mov
incinc
inc
Patch Run/Hit BP
TemporaryUnpatch
Single Step/Restore BP
Unpatch
IP
IP
IP
int
int
int
Multithreaded Single-Step Stepping over a single instruction Can easily step over most
instructions with native single step (SS)
Others require setting a BP after the instruction REP string instructions
Single-step with SS
A
B
A
B
Q
RSS
SS event
SS TargetThread
Hardware is set up to single step in this thread only.When that SS is over, hardware fires SS event for this thread only.
Single-step with BP: Bad
A
B
A
B
Q
Rrun
BP event
SS TargetThread
At hardware level, BPs are scoped to memory, not threads.So, any thread can hit the BP.
Single-step with BP: Good
A
B
A
B
Q
Rrun
BP event
SS TargetThread
Force scoping BP to target thread by suspending all others.
Steppers
State machines for complex stepping In, Over, Out, Go/Resume Instruction, Statement
Control low-level SS and BP Receive notification of SS and BP
events Can be canceled
Stepping Scenarios
Instruction steppers handle 18 scenarios 3x Instruction type: (simple, call, REP) 2x At a BP 3x Movement: (Go, Step In, Step Over)
Range stepper uses instruction steppers over an address range
Step Out stepper runs to a BP at return address
Example: Call Instruction, At BP
Go/Resume
IP
Step In
IP
Step Over
IP
IP
IP SS=1
SS=1
SS
SS
IP SS=1
BP
BP
BP
XA
XA
XA
BP
BP
BP
SS
BP
BP
BPBP BP BP
IPXA
BPBP
BPIP
BP
Threading
Debugger Thread
Main Thread
Run Command
Dispatch Event
Command
EventsDebuggee
Mago Debugger
What is debugging? What is a debugger? What is Mago? Execution AgentExpression Evaluator Symbol Reader Debug Engine D and the debugger
Expression Evaluator
Evaluates D expressions Input is textual expression Output is a result value record Declarations, symbols, and input values
come from outside IValueBinder, IDeclaration
Handles formatting values Enumerates children of values Based on DMD front end
EE Data Flow
chars tokensScanner Parser Value Binder
Process Debug Info
Value
Node Tree
EE Usage
MakeTypeEnv( &typeEnv );MakeNameTable( &nameTable );ParseText( L”a[2] + 3”, typeEnv,
nameTable, &expr );expr->Bind( options, binder );expr->Evaluate( options, binder, &result );
Resulting Node Tree
Parser
Type Env
String Table
Binder
Add
Int 2ID "a"
Index Int 3Find,Make Type
Add
Parse "a[2] + 3"
Make
Make
Build
debuginfoFind,
Make Type
Find Object Find Symbol
Debuggee Memory
9 3 5 11A334
Read
Variable "a"Array of intat 1A334
Get value at addr 1A33C
EvalBind Value = 8
Mago Debugger
What is debugging? What is a debugger? What is Mago? Execution Agent Expression EvaluatorSymbol Reader Debug Engine D and the debugger
Symbol Reader
Reads debug info for a program Maps of source files to lines Maps of source code lines to addresses Functions – address and scopes Symbols – name, type, value, storage Types
Reads specific formats Currently, CodeView 4.10, output by
DMD
Compare to DWARF
CODEVIEW
Fixed record fields Numeric constant
compression Common type
encoding Sorted symbols Nested Lexical blocks
DWARF
Flexible Attributes: key-value Explicit base type
definition Location expressions
Compression Flatten tree Abbreviations Byte code for tables
CodeView Sample
12S_GDATA320x00000310
10x1003
a
IntArray1
LF_ARRAYT_INT4 (0x0074)
T_UINT4 (0x0075)0x0010
18
record lengthrecord type
offsetsegment@typename
name
record type@elemtype@idxtypebyte count
record length2
22**
2
2
422*
2
Spec Actual
1
9
Size
Mago Debugger
What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol ReaderDebug Engine D and the debugger
Debug Engine
A plug-in to the VS Debugger package (vsdebug.dll)
Standalone DLL doesn’t depend on any other package
Expected to implement AD7 interface Knows how to debug one kind of
program DEs are multiplexed during a debug
session
AD7 Interface
A programming model for debugging processes
Single-threaded calls from VS Debugger to DE Simplifies design
COM interfaces Debug Engine is a COM co-class
Programming Model
IDebugEngine2 IDebugThread2 IDebugBoundBreakpoint2 IDebugExpression2 IDebugStackFrame2 IDebugDisassemblyStream2 IDebugEvent2
Threading
Event
HandleEvent
Event
HandleEvent
Main
SDM DE Event Monitor AD7 Callback
Mago Debugger
What is debugging? What is a debugger? What is Mago? Execution Agent Expression Evaluator Symbol Reader Debug EngineD and the debugger
D and the debugger
Rewrite in D eventually Only EE and parts of DE know about
D EE Test input generated by D
program Expression and expected value Uses compile-time reflection
GenUnaryTest.dvoid main( string[] args ) { writeln( "<test>" ); if ( set == 1 ) UnaryList!(byte, ubyte, short, ushort, int, uint,
long, ulong).Operation( op ); else if ( set == 2 ) UnaryList!(float, double, real, ifloat, idouble,
ireal, cfloat, cdouble, creal).Operation( op ); writeln( "</test>" );}
template Unary(T) { void Unary( void function( T ) func ) { foreach ( t; Vals!T.vals ) { Id++; writefln( " <verify id=\"%s_%d\">", Prefix, Id
); func( t ); writefln( " </verify>" ); } }}
template UnaryList(T...) { void Operation( Op op ) { foreach ( t; T ) { switch ( op ) { case Op.Negate: Unary!(t)(&UnOp!
(t).Negate); break; case Op.BitNot: static if (__traits( compiles, Unary!(t)
( &UnOp!(t).BitNot ))) Unary!(t)( &UnOp!(t).BitNot ); break; } } }}
template Vals(T) { static T[] vals; static this() { static if ( !__traits( isFloating, T ) ) // Add values like T.max, cast(T) -1, cast(T) 0 else // Add values like T.nan, -T.infinity, cast(T) 0 }}
GenUnaryTest.dtemplate UnOp(T) { static if ( __traits( isIntegral, T ) ) void BitNot( T t ) { writeln( " <bitnot>" ); CastTerm( t ); writeln( " </bitnot>" ); writeln( " <typedvalue>" ); PrintType!(typeof( ~t ))(); PrintTerm( ~t ); writeln( " </typedvalue>" ); } void PrintType(X)() { static if ( __traits( isArithmetic, X ) ) writefln( " <basictype name=\"%s\"/>",
typeid( X ) ); else writefln( " <reftype name=\"%s\"/>",
typeid( X ) ); } void CastTerm(X)( X x ) { writeln( " <cast>" ); PrintType!X(); PrintTerm( x ); writeln( " </cast>" ); } void PrintTerm(X)( X x ) { static if ( is(X==creal) || is(X==cdouble) ||
is(X==cfloat) ) { writeln( " <group>" ); writeln( " <add>" ); writefln( " <realvalue value=\"%a\"/>",
x.re ); writefln( " <realvalue value=\"%ai\"/>",
x.im ); writeln( " </add>" ); writeln( " </group>" ); } else if ( is( X == ireal ) || is( X == idouble ) || is(
X == ifloat ) ) { writefln( " <realvalue value=\"%ai\"/>",
x ); } else if ( __traits( isFloating, X ) ) { writefln( " <realvalue value=\"%a\"/>",
x ); } else if ( is( X == ulong ) || (is( X == long ) &&
(x < 0)) ) { writefln( " <intvalue value=\"%dUL\"/>",
x ); } else writefln( " <intvalue value=\"%d\"/>", x ); }}
Implementation Challenges Interface for sharing between
modules Take DMD front end or mimic it 80-bit floating point in expression
eval Disassembling variable length
instructions
Links Mago:
http://dsource.org/projects/mago_debugger D: http://digitalmars.com/d/ Visual D IDE by Rainer Schuetze:
http://dsource.org/projects/visuald Visual Studio Debugger Extensibility:
http://msdn.microsoft.com/en-us/library/bb161718.aspx
x86 and x64 Manuals: http://www.intel.com/products/processor/manuals/
DWARF Format: http://www.dwarfstd.org