Typed Assembly Typed Assembly Languages and Security Languages and Security

AutomatonsAutomatonsBen WatsonBen Watson

The George Washington UniversityThe George Washington UniversityCS 297 Security and Programming LanguagesCS 297 Security and Programming Languages

June 2, 2005June 2, 2005

Problems with AssemblyProblems with Assembly

Assembly is completely data-agnostic: it Assembly is completely data-agnostic: it doesn’t care, doesn’t want to care what doesn’t care, doesn’t want to care what type of data you’re moving aroundtype of data you’re moving around

If C is the rope to hang yourself with, then If C is the rope to hang yourself with, then assembly is parking your car on the train assembly is parking your car on the train tracks. At rush hour. With a full tank of tracks. At rush hour. With a full tank of gas. And you can’t leave your car. Good gas. And you can’t leave your car. Good luck.luck.

TALx86TALx86

Typed Assembly Language for Intel x86 Typed Assembly Language for Intel x86 processorprocessorImplements subset of Intel IA32 instruction Implements subset of Intel IA32 instruction setsetDesigned to be “realistic”, as in compilable Designed to be “realistic”, as in compilable and usable on a real computer by real and usable on a real computer by real people.people.Allow compilation from multiple high-level Allow compilation from multiple high-level languageslanguages

TALx86TALx86

Designed to overcome specific limitations in Designed to overcome specific limitations in Java bytecodeJava bytecode JVML has semantic errors that could have been JVML has semantic errors that could have been

discovered with a formal modeldiscovered with a formal model Difficult to compile other high-level languages to Difficult to compile other high-level languages to

bytecodebytecode Difficult to extend Java itself due to bytecode Difficult to extend Java itself due to bytecode

limitations (impossible to correctly compile Scheme to limitations (impossible to correctly compile Scheme to bytecode, for example)bytecode, for example)

Bytecode interpretation is slow, thus JIT is often usedBytecode interpretation is slow, thus JIT is often used—but this is an afterthought, not fundamental—but this is an afterthought, not fundamental

Alternate solutionAlternate solution(because I like .Net)(because I like .Net)

Many of these problems that TALx86 was Many of these problems that TALx86 was designed to address were also addressed designed to address were also addressed in .Netin .Net Formal design models were usedFormal design models were used CLR designates minimal feature set for supported CLR designates minimal feature set for supported

languageslanguagesScheme compilations is possible, along with dozens of other Scheme compilations is possible, along with dozens of other languageslanguages

JITted code part of the design of runtime and JITted code part of the design of runtime and languagelanguage

MSIL, like bytecode, is a typed intermediate assembly MSIL, like bytecode, is a typed intermediate assembly languagelanguage

TALx86 FeaturesTALx86 Features

Most basic assembly featuresMost basic assembly features

Stack-allocationStack-allocation

Type-checkingType-checking

Arrays and unionsArrays and unions

Recursive types (i.e., linked lists)Recursive types (i.e., linked lists)

AnnotationsAnnotations

PopcornPopcorn

Standard C isn’t strongly typed and thus Standard C isn’t strongly typed and thus can’t be represented as TALx86can’t be represented as TALx86

A strongly-typed C-based languageA strongly-typed C-based language

Support for polymorphism, abstract types, Support for polymorphism, abstract types, tagged unions, and exceptionstagged unions, and exceptions

Won’t discuss too muchWon’t discuss too much

TALx86 Compilation ProcessTALx86 Compilation Process

Main.tal – assembly Main.tal – assembly listinglisting

Main_i.tali – import Main_i.tali – import interfacesinterfaces

Main_e.tale – export Main_e.tale – export interfacesinterfaces

TALx86 AnnotationsTALx86 Annotations

Import/export interfaces (for type-checking Import/export interfaces (for type-checking separate object files)separate object files)

Type constructors (how to declare new types)Type constructors (how to declare new types) Preconditions on code labels (register must Preconditions on code labels (register must

have type X before code entered)have type X before code entered) Types for static dataTypes for static data Type coercions (converting one type to Type coercions (converting one type to

another)another) Macros (type checker can treat entire section Macros (type checker can treat entire section

as single action)as single action)

TALx86: Register PreconditionsTALx86: Register Preconditions

int i = n+1;int i = n+1;

int s = 0;int s = 0;

While (--i > 0)While (--i > 0)

s+=i;s+=i;

mov eax, ecxmov eax, ecx ;i=n;i=n

inc eaxinc eax ;++i;++i

mov ebx, 0mov ebx, 0 ;s=0;s=0

jmp testjmp test

body: {eax: B4, ebx: B4}body: {eax: B4, ebx: B4}

add ebx, eaxadd ebx, eax ;s+=i;s+=i

test: {eax: B4, ebx: B4}test: {eax: B4, ebx: B4}

dec eaxdec eax ;--i;--i

cmp eax, 0cmp eax, 0 ;i>0;i>0

jg bodyjg body• B4: 4-byte integer

TALx86: Supporting C/Win32 TALx86: Supporting C/Win32 calling conventionscalling conventions

Predicate to describe state of stackPredicate to describe state of stack {esp: sptr {eax: B4}::B4::se}{esp: sptr {eax: B4}::B4::se} The stack must contain a pointer to a 4-byte The stack must contain a pointer to a 4-byte

int, an int (function argument), then nothing int, an int (function argument), then nothing elseelse

Can be generalized to any stack “shape”Can be generalized to any stack “shape” :Ts {esp: sptr {eax: B4, esp: sptr :Ts {esp: sptr {eax: B4, esp: sptr

B4::B4::}::B4::}::B4::}}

TALx86TALx86

A type verifier checks the validity of each A type verifier checks the validity of each instruction in each label’s blockinstruction in each label’s block The type checker is programmed for the The type checker is programmed for the

semantics of TALx86/IA32 instructionssemantics of TALx86/IA32 instructions

Additional rules forAdditional rules for Memory allocationMemory allocation

But not deallocation! (hence the use of a garbage But not deallocation! (hence the use of a garbage collector)collector)

ArraysArrays Lists, structsLists, structs

TALx86: OptimizationsTALx86: Optimizations

Abbreviations (to take less space in Abbreviations (to take less space in source)source)

Remove repetitions (i.e., a stack and its Remove repetitions (i.e., a stack and its return address have the same type)return address have the same type)

Forward branch targets need no Forward branch targets need no preconditionprecondition

TALx86: A foundation for securityTALx86: A foundation for security

TALx86 gives you assurance when you TALx86 gives you assurance when you compile your code that type safety is compile your code that type safety is enforcedenforced

It does not add security It does not add security per seper se

For that, let’s move on to…For that, let’s move on to…

Type Systems for Expressive Type Systems for Expressive Security PoliciesSecurity Policies

Assumes a strongly typed language (such Assumes a strongly typed language (such as TALx86)as TALx86)

Uses security automataUses security automata Can always be enforced by runtime checksCan always be enforced by runtime checks Can rewrite programs to obey policy Can rewrite programs to obey policy

Security AutomatonSecurity Automaton

start has_read

read(f)

read(f)send()

Enforcing Security AutomatonsEnforcing Security Automatons

Code InstrumentationCode Instrumentation Auxiliary code (usually for monitoring Auxiliary code (usually for monitoring

purposes)purposes) C/C++ usually has to be recompiledC/C++ usually has to be recompiled .Net and Java don’t always (runtime .Net and Java don’t always (runtime

environment + reflection)environment + reflection)

Enforcing Security AutomatonsEnforcing Security Automatons

Formal:Formal:

Let Let nextnext = = sendsend((currentcurrent) )

If If nextnext = = badbad then halt then halt

else else sendsend()()

Enforcing Security AutomatonsEnforcing Security Automatons

Example:Example:

Before:Before:

……

Send();Send();

……

Enforcing Security AutomatonsEnforcing Security Automatons

After:After:……State nextState = State nextState = GetNextState(currentState);GetNextState(currentState);

ifif (nextState == badState) { (nextState == badState) {throwthrow newnew SecurityException(); SecurityException();

}}Send();Send();……

Security Instrumentation SystemsSecurity Instrumentation Systems

SASISASI Security Automata SFI Implementation Security Automata SFI Implementation

Software Fault IsolationSoftware Fault Isolation An implementation of security automatons An implementation of security automatons

and instrumentationand instrumentation Slows down native code, but only slightlySlows down native code, but only slightly Reimplemented Java security manager—at Reimplemented Java security manager—at

least as efficientleast as efficient

Enforcing Security AutomatonsEnforcing Security Automatons

But what if someone hacks the state-But what if someone hacks the state-checking code?checking code? Prove that the state-checking code is correctProve that the state-checking code is correct Augment the type system to include value Augment the type system to include value

states, similar to TALx86states, similar to TALx86 Associate predicates with each enforceable Associate predicates with each enforceable

statementstatementThese are decidable at compile-timeThese are decidable at compile-time

Enforcing Security AutomatonsEnforcing Security Automatons

OptimizationsOptimizations Some predicates are always true, given a Some predicates are always true, given a

certain state – the check can be removedcertain state – the check can be removed Perform control-flow analysis that propagates Perform control-flow analysis that propagates

proven predicates throughout programproven predicates throughout programPossibly proving further predicates that can be Possibly proving further predicates that can be removedremoved

Benefits of Previous TechniquesBenefits of Previous Techniques

Prevents things like buffer overflow attacksPrevents things like buffer overflow attacks

More confidence in machine-level codeMore confidence in machine-level code

Stronger high-level Stronger high-level low-level mapping low-level mapping

ReferencesReferences

MorrisettMorrisett, et al. , et al. TALx86: A realistic typed TALx86: A realistic typed assembly languageassembly language, ACM SIGPLAN Workshop , ACM SIGPLAN Workshop on Compiler Support for System Software, on Compiler Support for System Software, pages 25-35, Atlanta, GA, USA, May 1999pages 25-35, Atlanta, GA, USA, May 1999

Walker, David. Walker, David. A type system for expressive A type system for expressive security propertiessecurity properties. Twenty-Seventh ACM . Twenty-Seventh ACM SIGPLAN-SIGACT Symposium on Principles of SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 254-267, Programming Languages, pages 254-267, Boston, MA, USA, January 2000Boston, MA, USA, January 2000