© 2001 Barton P. MillerDecember 2001DynInst Security Playing Inside the Blackbox: Using Dynamic Instrumentation to Create Security Holes Barton P. Miller

DynInst Security© 2001 Barton P. Miller December 2001

Playing Inside the Blackbox:Using Dynamic Instrumentation to Create

Security Holes

Barton P. [email protected]

Computer Sciences DepartmentUniversity of Wisconsin

Madison, Wisconsin 53705USA

– 2 – DynInst Security© 2001 Barton P. Miller

Overview

1. How to easily do dangerous and malicious things to a running program.

2. How to detect when someone does something evil to your program.


A New View

Running programs are objects to be easily manipulated. Kinds of manipulations might include:

Instrumentation

Optimization

Control


The Vehicle:The DynInst API

A machine-independent library for machine level code patching.

Eases the task of building new tools.

Provides the basic abstractions to patch code on-the-fly


Dynamic InstrumentationDoes not require recompiling or relinking

• Saves time: compile and link times are significant in real systems.

• Can instrument without the source code (e.g., proprietary libraries).

• Can instrument without linking (relinking is not always possible.

Instrument optimized code.


Dynamic Instrumentation (con’d)Only instrument what you need, when you

need• No hidden cost of latent instrumentation.• Enables “one pass” tools.

Can instrument running programs (such as Web or database servers)• Production systems.• Embedded systems.• Systems with complex start-up procedures.


The Basic MechanismApplicationProgramApplicationProgram

Function fooFunction foo

TrampolineTrampoline

Pre-InstrumentationPre-Instrumentation

RelocatedRelocatedInstructionInstruction

Post-InstrumentationPost-Instrumentation


The DynInst Interface

Machine independent representationObject-based interface to build Abstract

Syntax Trees (AST’s)Write-once, instrument-many (portable)Hides most of the complexity in the API

• Process Hijacker: only 700 lines of user code!

• MPI tracer: 250 lines


Basic DynInst OperationsProcess control:

• Attach/create process• Monitor process status changes• Callbacks for fork/exec/exit

Image (executable program) routines:• Find procedures/modules/variables• Call graph (parent/child) queries• Intra-procedural control-flow graph


Basic DynInst OperationsInferior (application processor) operations:

• Malloc/free– Allocate heap space in application process

• Inferior RPC– Asynchronously execute a function in the

application.

• Load module– Cause a new .so/.dll to be loaded into the

application.


Basic DynInst OperationsInferior operations (continued):

• Remove Function Call– Disable an existing function call in the

application

• Replace Function Call– Redirect a function call to a new function

• Replace Function– Redirect all calls (current and future) to a

function to a new function.


Basic DynInst OperationsBuilding AST code sequences:

• Control structures: if and goto• Arithmetic and Boolean expressions• Get PID/TID operations• Read/write registers and global variables• Read/write parameters and return value• Function call


Security Applications of DynInst

Lots of tool applications of Dyninst by lots of

groups. Here are two security-oriented ones:

License server bypassing

Condor security attacks


License Server Attack: The Bypass

Program License Data

Network

License

Server

Normal: licensed program runs after communicates with license server.

Program License Data

Network

License

Server

Undesired: licensed program refuses to run if license server does not respond.


Example: Adobe FrameMaker

Two-step license verification:• retrieve license data from server [once]• check license data for correctness [often]

In practice:• allow FM to time-out waiting for server• allow FM to attempt to go into “demo” mode• switch FM back to normal mode• insure that future license checks always

succeed


StrategiesComplete reverse engineering:

• not an option– legal problems– complexity (FrameMaker is a 7 MB binary!)

Focus on certain characteristics:• I/O (network sockets) traffic• execution trace


ToolsHigh-level language translators:

• Dyner: interactive, interpreted C subset• Jdyninst: Java to DynInst compiler

Bypasser: an interactive call graph browser• Search and walk application call graph• Resolves function pointers at runtime• Call follow caller or callee paths• Can generate call trace


UseDetermining where to apply changes:

• get trace for a successful run• get trace for a (forced-)failure run• compare to find differences• repeat as needed


DetailsFM calls NlOpenlicenses on start up

• Contacts license server and caches credential if successful

At end of main, FM calls NluiCheckLicense• If credential is not present, call ChangeProductToDemo (cannot save files)

Frequently, during operation, FM will check for cached credentials.




• Allow this to fail.At end of main, FM calls NluiCheckLicense

• If credential is not present, call ChangeProductToDemo (cannot save files)

Frequently, during operation, FM will check for cached credentials.






• Delete the call to ChangeProductToDemo.Frequently, during operation, FM will check

for cached credentials.






• Delete the call to ChangeProductToDemo.Frequently, during operation, FM will check

for cached credentials.• Change this call to always return “true”.


Condor Attack: Lurking Jobs

Condor schedules jobs on idle workstationsIn a normal mode, jobs run as a common,

low-privilege user ID: “nobody”.This common user ID provides an

opportunity for an evil lurking process to ambush subsequent jobs (from other users):


Condor Job Structure

Submitting HostSubmitting Host

Shadow Process

Execution HostExecution Host

User Jobsystem calls




Shadow Process


EvilUser Job

system calls

LurkerProcess

forkfork



Submitting HostSubmitting Host Execution HostExecution Host

LurkerProcess




Shadow Process


InnocentUser Job

system calls

LurkerProcess




Shadow Process


InnocentUser Job

system calls

LurkerProcess

attach




Shadow Process


InnocentUser Job

system calls

Control remotesystem calls

LurkerProcess

rm -rf *



Can We Trust a Remote Job?The threats:

1. Cause the job to make improper remote system calls.

2. Cause the job to calculate an incorrect answer.

3. Steal data from the remote job.

Threat protection strategies:• File sand-boxing (#1)• System call sand-boxing (#1)• Obscure and encode binary (#1)• Replicate remote job (#2)


Sand-Boxing

Shadow process selectively rejects system calls:• Restrict access to specific files or directories• Disallow certain system calls• Disallow certain system call parameter values


Shadow Process


User Jobsystem calls


Obscuring the Executable

User Job

ModifiedUser Job

Checking Shadow

Modifier/Obscurer



Goal:Even if an intruder can see, examine, and fully

control the remote job, no harm can come to the local machine.



Modify the executable:• Replace each system call site with call to a

unique random function nameThis makes understanding the executable harder and

makes checking the system call stream easier.

• Delete any constant parametersValues like file names or descriptor numbers are hidden.

• Pad parameter other lists so all calls have equal number

• Permute the order of parameters• Add dummy system calls throughout code



Generate a Checking Shadow:• Build a FSM that reflects the inter-procedural

control-flow of the job.Note that this FSM has increased accuracy, since each

call site has a unique name

• Map random function names back to their actual system calls.

• Insert parameter values and present in correct order.

• Ignore dummy calls.



Some notes of interest:• All hiding and replacement is done before the

job is submitted to Condor.No knowledge of the encryption or keys is contained in

the remote job.

• All replacing is done in the Checking Shadow, which only resides on your local machine.

• FSM accuracy and computational complexity is simplified by the function renaming.

Our early results look good. A tech report will be available by the end of November.


How to Get a Copy of DynInst:Release 2.3 (release 3.0 imminent)

• Free for research use.• Runs on Solaris (SPARC & x86), Windows

NT, AIX/SP2, Linux (x86), Irix (MIPS),Tru64 Unix (Alpha).

http://www.paradyn.org

http://www.dyninst.org

[email protected]

Documents

© 2001 Barton P. MillerDecember 2001DynInst Security Playing Inside the Blackbox: Using Dynamic Instrumentation to Create Security Holes Barton P. Miller