CS717 Detection of Control Flow Errors Survey of Hardware and Software Techniques Greg Bronevetsky

CS717

Detection of Control Flow ErrorsSurvey of Hardware and Software Techniques

Greg Bronevetsky

CS717

Scope of Lecture

• No general techniques for detecting random faults

• Approach: focus on faults that show up as control flow errors– Control flow well defined problem– Legal paths through program reasonably

approximated via simple models

• If control flow errors made detectable, may detect majority of faults

• Will cover several papers in this field

CS717

Classes of Solutions

• Watchdog Processors– Separate processor watches instruction stream– Blows whistle on erroneous control flow– Pro:

• Small hit on application performance

– Con:• Needs new hardware

– Extra processor

» Though might be doable via Thread-Level Speculation

– Possibly, more bits in RAM

CS717

Classes of Solutions

• Software-only solutions– Extra instructions inserted into program– Check that jumps are correct– Pro:

• Works on existing hardware

– Con:• Potential for significant overhead• Inserted instructions can also fail

CS717

Outline

• Watchdog Processor– General overview– Concurrent Process Monitoring with No Reference

Signatures (1994)– SEIS (1995)

• Software– BSSC/ECI (1992)– CCA/ECCA (1999)– PECOS (2003)

• Control-flow Checking for Security

CS717

Watchdog Processors

• Watchdog processor acts as coprocessor• Can see all instructions loaded by CPU

– If watchdog above cache, CPU involvement required

• Signals error when sees invalid control flow

CPU

Watchdog

RAM

CS717

Block Signatures

• Each branch-free block gets a signature• Signature computed from block’s instructions• Signatures vary with scheme

– Ex: sum of the instructions in block

ld r3[0xff] r1add r1, r5, r2……mult r2, r2, r3

SB

jmp label

Block B, with signature SB

Insert SB before end of block

Watchdog sees all instructions, recomputes SB

At block end, watchdog compares computed signature

with embedded signatureError raised if not equal

Compile-time Runtime

CS717

Illegal Jump

• If illegal jump happens– watchdog will have partially

computed signature from prior block

– jump to wrong block will lead watchdog to complete signature with wrong instructions

– when watchdog sees next signature in code, will not match computed signature

Legal control flows

Illegal jump

Computed SBEmbedded SB

C

D

A

B

CS717

Error Coverage

• What if partial signature at point 1 equals partial signature at point 2?

• Then at end of block D Computed SB = Embedded SB

• Illegal jump not detected

• Problem if many blocks start out the same

Computed SB=Embedded SB

Point 1

Point 2

C

D

A

B

CS717

Error Detection Latency

• Latency - number of clock cycles before error is detected

• Blocks are typically small– 5-10 instructions

• Thus, time to fault detection fairly low• Requires many signatures to be embedded

– Memory overhead– Processor stalls while signature fetched

• Some schemes use one signature for >1 blocks– Worse detection latency

CS717

Limitations

• Approaches typically require knowledge of program control flow– Many cannot deal with programs that generate

jump destinations at runtime

• Only deal with errors manifested as control-flow errors– True for many errors– If error is originally a data error, long detection

latency

CS717

Outline





CS717

Upadbyaya & Ramamurthy

• Generalization of watchdog signatures approach

• Requires no signatures to be embedded into program code

• S.J. Upadbyaya and B. Ramamurthy, “Concurrent Process Monitoring with No Reference Signatures”, IEEE Transactions on Computers, 1994.

CS717

Outline of Scheme

• SIG: signature accumulated by watchdog• INST: current instruction• F: some signature generating function

– Ex: sum, XOR, etc.

• Initialize SIG to some value (such as all 0’s)• For every instruction in block,

SIG = F(SIG, INST)

CS717

Outline of Scheme

• When SIG satisfies a certain property, stop and mark instruction as check-point– Can use parity bit to mark

• At this point watchdog will notice check-point mark and will verify that its SIG also satisfies property

• Paper favorite property: m-out-of-n codeword– m 1’s out of n bits– Easy to reason about error coverage

CS717

An Single-Block Example

ld r3[0xff] r1

add r1, r5, r2……… mult r2, r2, r6

…

…

… st r6, r3[0xb2]

shl r1, r3[0xb2], 1

SIG = 0SIG = F(SIG, INST)

SIG = F(SIG, INST)

…

SIG = F(SIG, INST)m-out-of-n codeword!

mark instruction as

check-point

mult r2, r2, r6

WatchdogCompiler

SIG = 0SIG = F(SIG, INST)

SIG = F(SIG, INST)

…

SIG = F(SIG, INST)Verify that SIG is now an

m-out-of-n codeword

SIG = F(SIG, INST)

SIG = F(SIG, INST)...

SIG = F(SIG, INST)

SIG = F(SIG, INST)...

CS717

Branch Example

• Two blocks with different final signatures

• Join at block C• SIG will become legal

codeword at different points depending on whether we came from block A or B

• Must merge SIG1 and SIG2 into one

SIG1 SIG2

A B

C

CS717

Branch Example

• Idea: make sure that when we get to C, SIG is valid codeword

• But must still ensure that we correctly got to C

SIG1 SIG2

A B

C

CS717

At Compile-time

• Let TARGET = first instruction in block C• Before end of blocks A&B

SIG=F(SIG1, TARGET)EXIT_BYTE=value s.t. F(SIG, EXIT_BYTE) is a

valid codeword

TARGET

… code …

jmp label

EXIT_BYTE

• TARGET marked as a check-point

TARGET

SIG2= …SIG=F(SIG2, TARGET)

Insert EXIT_BYTE

… code …

jmp label

EXIT_BYTE

SIG1= …SIG=F(SIG1, TARGET)

Insert EXIT_BYTE

CS717

At Runtime

• At branch– Load Target instruction– Compute SIG=F(SIG2, TARGET)– Compute SIG=F(SIG, EXIT_BYTE)– SIG should now be a valid codeword

• TARGET is marked as check-point– Watchdog verifies that SIG is valid codeword

TARGET

… code …

jmp label

EXIT_BYTE

SIG2= …SIG=F(SIG2, TARGET)SIG=F(SIG2, EXIT_BYTE)

CS717

Summary of Scheme

• Scheme checks whether accumulated signature obeys given property– vs. ensuring that it matches a recorded signature

• Thus, don’t store signatures for each block– Savings in memory and fewer noops

• Need to store EXIT_BYTE– Paper suggests several mechanisms that don’t

require extra noop

• Can be generalized to any signature function and any property– Theoretical analysis given for m-out-of-n property

CS717

Performance Results

• They give some indications of what detection latency would be

• No information on – error coverage– performance overhead

CS717

Outline





CS717

Traditional Watchdog Techniques

• Monitor every single instruction• Not possible in machines with caches

(unless watchdog is part of processor)– External watchdog can’t see instructions fetches

from Instruction cache (I-cache)– With low I-cache miss rate, most control flow

invisible to watchdog

• In multi-tasking systems task switches will confuse watchdog

CS717

Assigned Signatures to the Rescue

• An Assigned Signature method– At various points in code, send unique signature to

watchdog– Watchdog knows legal order of signatures

• Pro:– Communication with watchdog explicit, will bypass

cache

• Con:– Since actual code invisible to watchdog, additional

required watchdog intelligence can be expensive

CS717

SEIS

• SEIS = Signature Encoded Instruction Stream• An Assigned Signature method• Aims to offer efficiency• Targets multiprocessing systems

• Documented in:I. Majzik, A. Pataricza, W. Hohl, J. Hönig and V. Sieh, “A High Speed Watchdog Processor for Multitasking Systems”, 1994.

CS717

SEIS Basic Idea

• Create control-flow graph of program– Nodes are jump-free blocks of code– Edges are possible control flows

• Assign unique label to each node s.t.– Labels have constant length– Labels of adjacent nodes have easy to detect

relationships• ex: label of each node contains IDs of valid successors

CS717

Label assignment

• Find Eulerian circuit through control-flow graph– Contains all edges exactly once– Touches nodes multiple times

• Guaranteed to exist if in all nodes in-degree = out-degree– Add phantom edges to graph to assure this

CS717

Label Assignment

• Walk circuit, assigning numbers to nodes in increasing order– When walking over phantom edge, skip a number

• Node’s label = concatenation of all assigned numbers (sublabels)– # sublabels per node = in-degree = out-degree

• If two nodes neighbors in control-flow graph, node A followed by node B– Node A label <a1, … ak>, node B label <b1, … bl>

– Then ai, bj s.t. bj = ai + 1

CS717

Example

CS717

Example

Initial Graph: in-degree out-degree

CS717

Example

Initial Graph: in-degree out-degreeAdd Phantom Edges

CS717

Example

Find Eulerian Circuit

CS717

Example

Assign Sublabels

0

1

25

,4

,6

7

11

,12

,10

,8

CS717

Example

Note: If node A succeeds node B, they’re off by 1 along some sublabel

0,6,12

1,4

2,85

7,10

11

CS717

At runtime

• In each block insert instruction to send block label to watchdog

• Watchdog watches incoming labels, stores last label seen

• If new label is neighbor of stored label along some sub-label, all OK– i.e. new label has some sub-label that is +1

relative to some sublabel of stored label

• If new label NOT neighbor of stored label, control-flow error!

CS717

Unbounded Signatures

• Method leads to efficient watchdog checks• Only problem:

– Number of successor/predecessor nodes unbounded

– Thus, label size unbounded

• Must cut out some labels without sacrificing correctness

CS717

Removing Sub-labels

• k predecessors k sublabels

• Pick one predecessor sublabel: ai

…,a1,… …,ak,…. . . . . . .

a1+1,…,ak+1

CS717

Removing Sub-labels

• k predecessors k sublabels

• Pick one predecessor sublabel: ai

• Have all predecessors use ai

• Node now needs one sublabel, not k

…,ai,… …,ai,…. . . . . . .

ai+1

CS717

Sub-label Removal Problem

• Problem: ajai but pre-predecessor still has aj-1

• Watchdog can’t tell if control-flow legal

ai+1

…,ai,…

…,aj-1,…

legal control flow?

…,aj,…

…,aj-1,…

a1+1,…,ak+1

CS717

Sub-label Removal Problem

• Paper rather cryptic on this issue• Possible solutions:

– More sublabels:• In predecessors replace single sublabel aj with sublabels

<ai, aj>• In node itself, replace, <a1, … , ak> with just ai

• Unclear if process will converge to specific label size bound

– Propagate replacement:• If predecessor node gets aj replaced with ai, in pre-

predecessor replace aj-1 with ai-1 and so on recursively• Many sublabels in program will be equal: reduced error

detection coverage

CS717

Constant Label Size

• Want label size constant, not just bounded• If label too large, use above procedure• If label too small, repeat a sublabel multiple

times– Has no effect on watchdog’s accuracy

• Paper’s target label size = 3 sublabels

CS717

Checking Procedure Calls

• Procedures have enhanced checking• At procedure start and end insert special

labels with unique procedure IDs• Watchdog maintains procedure stack

– Mirrors call stack– Ensures correct procedure returns

• Each block’s label contains ID of parent procedure– Allows for better&earlier detection of erroneous

inter-procedure jumps

CS717

Multi-Tasking Support

• Watchdog maintains several sets of last-label registers and procedure stacks– One per task– Switch between them during task switches– If more tasks than watchdog memory, caching

scheme can probably be used

• Program labels also contain task ID– Set at load time– Extra check for task scheduler and OS protection

mechanisms

CS717

Experimental Evaluation

• Program size overhead upto 30%• Execution time overhead can exceed 100%

– Particularly for programs with tight loops• one signature send for only few instructions in loop body

• Fault detection coverage: 50%• Authors implement several techniques to

reduce number of checks– Overhead brought down to 10%-15%– Fault detection coverage drops to 20%

CS717

Outline





CS717

Checking Control Flow in Software

• Modify program to contain instructions to check control flow

• Requires no additional hardware

• New instructions also vulnerable to faults

• Notable performance penalty– blocks are small: many additional instructions

CS717

BSSC

• BSSC = Block Signature Self Checking

• G. Miremadi, J. Karlsson, U. Gunneflo, J. Torin, “Two software techniques for on-line error detection”, FTCS 1992.

CS717

The Idea

• Assign signatures to block as you wish• Upon block entry, record signature• Upon block exit, check signature


jmp label


Compile-time Runtime

entry_proc() saves SB in static varchanges return ptr to next instruction

exit_proc() compares given SB to stored SB. If : YellsIf = : changes return ptr to next instruction

Insert exit_proc(SB) after block end

Insert entry_proc(SB) before block end

call entry_procSB

call exit_procSB

CS717

Optimization

• If SB= block’s starting address, can eliminate one signature


jmp label


Compile-time

Insert exit_proc(SB) after block end

Insert entry_proc() before block end

call entry_proc

call exit_procSB

• Address of instruction following entry_proc() call automatically pushed on stack before call

• Code planted at specific virtual address

CS717

Limitations of Technique

• BSSC ensures program leaves blocks it enters

call entry_proc(SA)

… block code …

call exit_proc(SA)

Legal Control Flow

Detected Bad Jumps

call entry_proc(SB)

… block code …

call exit_proc(SB)

call entry_proc(SC)

… block code …

call exit_proc(SC)

CS717

Limitations of Technique

• BSSC ensures program leaves blocks it enters• Cannot not detect:

– Erroneous jumps within block– Bad jumps to valid jump destinations

call entry_proc(SB)

… block code …

call exit_proc(SB)

InvisibleBad Jump

call entry_proc(SB)

… block code …

call exit_proc(SB)

call entry_proc(SB)

… block code …

call exit_proc(SB)

Legal Control Flow

call entry_proc(SB)

… block code …

call exit_proc(SB)

InvisibleBad Jump

CS717

ECI

• ECI = Error Capturing Instructions• G. Miremadi, J. Karlsson, U. Gunneflo, J.

Torin, “Two software techniques for on-line error detection”, FTCS 1992.

• Idea: fill unused memory with instructions that cause exception

• Bad jump to unused memory exception thrown

CS717

BSSC/ECI Results

BSSC + Some ECI All ECIOverhead 77%-128% ~6%Error Coverage 50%-56% 39%-50%

CS717

Outline





CS717

CCA/ECCA

• CCA = Control Flow Checking Using Assertions

• ECCA = Enhanced CCA

• Z. Alkhalifa, V.S.S. Nair, N. Krishnamurthy, J.A. Abraham. “Design and Evaluation of System-Level Checks for On-Line Control Flow Error Detection”, IEEE Transactions on Parallel and Distributed Systems, 1999.

CS717

CCA

• CCA a lot like BSSC but also checks branches for validity

call entry_proc(SB)

… block code …

call exit_proc(SB)

call entry_proc(SB)

… block code …

call exit_proc(SB)

Legal Control Flow

call entry_proc(SB)

… block code …

call exit_proc(SB)

Bad JumpsVisible with CCA

CS717

CCA Jump Checking

• Each basic block gets unique ID• Recorded on block entry, checked on block

exit (like BSSC)


jmp label


Record SB

Verify SB

CS717

CCA Jump Checking

• 2-element queue used to ensure valid flows are followed


jmp label


Dequeue signature, check its = to current block

Enqueue signature of following block

Enqueue Sig of next blockRecord SB

Verify SB

Dequeue Sig

CS717

Queue Example

BlockID=BFlowID=2

BlockID=CFlowID=2

Queue

1

BlockID=AFlowID=1

BlockID=DFlowID=3

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

2

1

BlockID=DFlowID=3

BlockID=BFlowID=2

BlockID=CFlowID=2

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

Dequeue & Check

2

1

BlockID=BFlowID=2

BlockID=CFlowID=2

BlockID=DFlowID=3

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

Dequeue & Check

3

BlockID=DFlowID=3

BlockID=BFlowID=2

BlockID=CFlowID=2

Enqueue 3

2

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

Dequeue & Check

3

BlockID=DFlowID=3

BlockID=BFlowID=2

BlockID=CFlowID=2

Enqueue 3

Dequeue & Check2

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

Dequeue & Check

?

BlockID=DFlowID=3

BlockID=BFlowID=2

BlockID=CFlowID=2

Enqueue 3

Dequeue & Check3

Enqueue ?

CS717

Queue Example

BlockID=AFlowID=1

Queue

Enqueue 2

Dequeue & Check

?

BlockID=DFlowID=3

BlockID=BFlowID=2

BlockID=CFlowID=2

Enqueue 3

Dequeue & Check3

Enqueue ?

Dequeue & Check

CS717

CCA Summary

• CCA detects bad jumps – from/to insides of blocks– from end of one block to start of wrong block

• Coverage still not complete:

– Inserted code includes unchecked branches

• Major Issue: High Overhead

Enqueue SigRecord SB

… block code …

Verify SB

Dequeue & Check

InvisibleBad Jump

CS717

ECCA

• Same checks as CCA but via arithmetic• Idea:

– Each block gets unique prime ID > 2– Global variable id contains current signature– Before end of block insert:id=(nextBlockIDs + !!(id - lastBlockID))• nextBlockIDs = product of IDs of succeeding blocks• lastBlockID = ID of block we just left

– Before entering block insert:id=curBlock/(!(id%curBlock)*(id%2))• curBlock = ID of block we’re about to enter

• Claim, bad control flow will cause divide by 0

CS717

How ECCA Works

Invariant: Inside a block, id=ID of current blockid = 3Block ID = 3

id=(nextBlockIDs + !!(id - lastBlockID))

(id – lastBlockID) =0 if id = lastBlockIDNot 0 if id lastBlockID

!(id – lastBlockID) =1 if id = lastBlockID0 if id lastBlockID

!!(id – lastBlockID) =0 if id = lastBlockID1 if id lastBlockID

• If program left same block it entered, at end id = nextBlockIDs = product of ID’s of valid next blocks

• Otherwiseid = nextBlockIDs + 1 (which is even)

CS717

How ECCA Works

id = 3Block ID = 3

id=(nextBlockIDs + !!(id - lastBlockID))

id=curBlock/(!(id%curBlock)*(id%2))

Block ID = 5

Block ID = 7

Block ID = 11

Block ID = 13

id%curBlock =0 if id is multiple of curBlockNot 0 otherwise

!(id%curBlock) =1 if id is multiple of curBlock0 otherwise

id%2 =0 if id is even (error exiting last block)1 if id is odd

CS717

How ECCA Works

id=curBlock/(!(id%curBlock)*(id%2))

Block ID = 13

id%curBlock =0 if id is multiple of curBlockNot 0 otherwise

!(id%curBlock) =1 if id is multiple of curBlock0 otherwise

id%2 =1 if id is odd0 if id is even (error exiting last block)

curBlock/(!(id%curBlock)*(id%2)) =curBlock if all is welldivide by 0 exception if came from wrong block or erroneous jump into last block

Block ID = 5

Block ID = 7

Block ID = 11

CS717

ECCA Summary

• Same Checking as CCA but arithmetic• Introduces no new jumps into code• Experiments:

– Error coverages improves notably• from ~92% to ~99% or from ~45% to ~92%• predict <5% overhead if we coalesced 20 branch-free

intervals into block and had 500 blocks– high error detection latency

CS717

Outline





CS717

PECOS

• Checks control flow errors in software• Doesn’t need to know valid control flows at

compile time (!!)– Magic: they don’t really check bad branches

• S. Bagchi, Z. Kalbarczyk, R. Iyer, Y. Levendel. “Design and Evaluation of Preemptive Control Signature(PECOS) Checking”. Submission to IEEE Transactions on Computers, 2004.

CS717

PECOS Details

• PECOS checks for memory errors, not control-flow errors

• Memory error becomes control-flow error when faulty address used for branch

• Idea: make sure that data used for branches hasn’t changed since last write

• Will detect such errors BEFORE they manifest themselves– Error avoidance better than detection

CS717

Approach

• Right before branch, look at– Valid target address options: {X1, X2, …}

– Runtime target address: Xout

• In memory, register or instruction body

• Compute:

• If Xout not valid: divide by 0 exception

]))([(! 21 XXXX

X

outout

out

CS717

Branches with constant operands

• Branch has single valid destination: X1

jmp Xout

• Before branch load up X1

• Ensure that branch instruction’s Xout valid– via above calculation

• If instruction got corrupted: divide by 0 exception

regjmp

XX

X

out

out

)(! 1

CS717

Branches with runtime targets

• Ex: jmp register1• Find all reaching definitions of register1

• At def site, save assigned value Xout

• At branch, check value for corruption via above formula– Function returns probably done this way

regjmp

valregXreg

reg

registerupbackregtmp

valreg

)])([(!

...

//

1

1

CS717

PECOS Experiments

• Evaluation code: DHCP server– largest I’ve seen in papers I read

• Error coverage improvement 95% 98.5%• Notably fewer process aborts/hangs

– 90% 52%– Allows more graceful response to errors

• Overheads tolerable: 15%-30%

CS717

PECOS Summary

• PECOS is a memory checker– More efficient than protecting all memory– Does not protect against instruction execution

errors• Paper argues these are rare since hamming distance

between branch&non-branch opcodes is large

• Detects memory errors before they cause crash / corrupt state

CS717

Outline





CS717

From Fault Tolerance to Security

• Above solutions have dealt with Random Fault Adversary

• Similar techniques can apply to Human Adversary• Many attacks inject code into target application and

jump to it• Not very different from what random faults do

• Paper: Milena Milenkovic, Aleksandar Milenkovic and Emil Jovanov, “Using Instruction Block Signatures to Counter Code Injection Attacks”, WASSA 2004.

CS717

Watchdog Processor for Security

• Target Systems:– Computers with caches– Computer assumed to be non-faulty– Faults come from application

• Thus, watchdog watches for errors above cache, not below

• Assumption: if attacker forces program to execute new code, program will never return to original control flow– True for attacks of today but can be circumvented

CS717

Basic Picture

• Compute signature for each code block

• On instruction cache fetch accumulate instructions into runtime signature

• On block end, check that runtime signature matches block’s compile-time signature

CPU

Cache

Checker

Instruction Stream

CS717

Checking Signatures

• Multiple ways to store/check signatures

• SIGT – uses table inside watchdog• SIGE – embeds signatures in program code• SIGC – assigns signatures to i-cache lines

• Signatures encrypted before execution– Harder for attacker to spoof them

CS717

SIGT

• Watchdog contains table of legal block signatures

• Table populated like cache– Swap in signatures as necessary– May be initialized with popular signatures at

program start

• When code block ends, compare its signature to table contents– May need to swap in appropriate signature

CS717

SIGE

• Signatures embedded in code before each block

• At block start watchdog loads signature

• At block end, runtime signature compared to signature just loaded

CS717

SIGC

• Signatures assigned to lines in the i-cache• Embedded in program code• When i-cache line fetched, signature also

fetched• Unlike SIGE, signatures not stored in cache

– Thus, lower impact on cache miss rate

• No need for compiler

CS717

Experimental Evaluation

• Codes: SPEC CPU2000• SIGT: small 2-way caches lead to low

signature table miss rates• SIGT/SIGE: overall i-cache miss rate doesn’t

increase that much– But in one code went up by factor of 12

• SIGC: not evaluated but suggested as way to keep i-cache miss rate down

CS717

Outline





CS717

Control-flow checking

• Fairly general-purpose technique

• Limited error detection coverage traded off for lower overhead than replication(though overhead not always low)

• Compile-time techniques– Primitive– The most advanced in random fault detection field

CS717

Uncovered Areas

• No techniques for placing watchdog inside processor itself(that I know of)

• Different techniques evaluated using very different codes&fault injection methodologies– Need to be compared on common ground

• Limited compiler intelligence– Control flow reduced to context-free grammars or

graph reachability• In reality Turing-complete

– Existing techniques don’t use program’s own dependencies on correct control flow

CS717

Using Control-flow Dependencies

• Current techniques insert new code/signatures into program

• But program’s own instructions depend on correct program flow

• Can we detect impact of bad control flows on program’s own instructions?– Will insert far fewer additional

instructions/signatures

CS717

Summary

• Field fairly mature but hardly finished

• Techniques catch a fair share of errors but leave plenty behind– Should be used along with other techniques

• Good example of giving decent solution to a limited version of detection problem

Documents

CS717 Detection of Control Flow Errors Survey of Hardware and Software Techniques Greg Bronevetsky