TxBox : Building Secure, Efficient Sandboxes with System Transactions

TXBOX: BUILDING SECURE, EFFICIENT SANDBOXES WITH SYSTEM TRANSACTIONS

Suman Jana The University of Texas at AustinDonald E. Porter Stony Brook UniversityVitaly Shmatikov The University of Texas at AustinIEEE Security & Privacy 2011

Outline

Introduction Building a Better Sandbox Related Work System Transactions Design and Implementation Evaluation Limitations Conclusions

Introduction

ACID: atomicity, consistency, isolation, and durability [wiki]

TxBox: Speculative execution of untrusted

application Uncircumventable enforcement of

system-call policies Automatic recovery from the effects of

malicious execution

Introduction

TxBox components: Grouping Rollback Access summary Access check

Based on TxOS [link] Linux with support for system

transactions Prevent race conditions

Building a Better Sandbox

TOCTTOU example

Softlink

Public file

Private file

1. Link to public file

2. Change to private file

System-call monitor must know all possible system-call sequences that lead to a violation and the monitor

must check every call !

Difficult and Slow !

Building a Better Sandbox

TxBox is immune to TOCTTOU attacks

Sandboxed processes run inside separate transactions

Whitelist / blacklist access-control policies

Building a Better Sandbox

Drawback of conventional sandbox Violation can be detected only after

multiple system calls, the damage may already have been done

TxBox: Transaction is aborted, and the system

automatically reverts

Building a Better Sandbox

Parallelism (Multi-core processors) Conventional sandbox or anti-virus:

Sandboxed program makes a system call, its execution must be paused

Split-personality malware TxBox:

Sandboxed program run with close-to-native performance while performing checks in parallel

Only a single copy is executed

Related Work

Speck, 2008 Multi-core processor Concurrently perform checks on an

instrumented copy on another core Cannot detect “split-personality”

malware Use Pin [link] Pin can be detected by malicious

program Suffer TOCTTOU attacks

Related Work

Related Work

Software Transactional Memory (STM) [wiki] Lock-based synchronization

STM-based systems Enforce application-specific security

policies Cannot span system calls

Related Work

Capsicum, 2010 Capability-based sandboxing system Help benign application to increase their

trustworthiness Native Client, 2009

Require the code to be recompiled to a restricted subset of the x86 ISA and also confines it using segmentation

System Transactions

ACID properties for sequences of updates to system resources

If the transaction aborted before it finish, all intermediate updates are rolled back

All accesses to the system within a transaction are kept isolated and invisible to the rest of the system until the transaction commits

System Transactions

TxOS: sys_xbegin – start transaction sys_xend – end transaction Maintain a transaction workset

Kernel objects (inodes, etc.) Each entry in workset contains a pointer to

the stable object, a pointer to the shadow copy

information: read-only / read-write Methods: commit / abort / lock / unlock /

release

Design and Implementation

Design and Implementation

TxBox policy BLACKLIST

Forbidden behavior WHITELIST

Required behavior DENY

Lists of system calls should be denied

Design and Implementation

BLACKLIST and WHITELIST primitives Regular expressions

System-call names and arguments, or objects Objects

(I) inodes Inode number Mode: read or write

(S) sockets Type: INET or UNIX Attributes: IP address (for INET socket) or names

(for UNIX datagram sockets)

Design and Implementation

Example (e)* means “match expression e any

number of times” Any deviation from a model-based

whitelist policy means that the sandboxed program is no longer executing the original code (code injection attacks)

Design and Implementation

Design and Implementation

DENY primitives DENY primitives is violated

Block calls and continue execution Run a sandboxed program without

letting it perform certain operations

Design and Implementation

Policy enforcement Character device driver

Send policy and pid to monitor Only communicate with root

Design and Implementation

System-call interposer TxBox call wrappers Symbolic link issue rename calls

Enforcer Force sandboxed

process to run in transaction mode

Design and Implementation

Policy decision engine Invoked when

sandboxed process attempts to perform a critical call

Design and Implementation

Implementation Based on Dazuko [wiki], by Avira

File Access Control Can also be based on LSM

xbegin -> beginTransaction kernel function beginTransaction checks if the current call

is xbegin Forced_transaction flag

Design and Implementation

Handling external I/O by sandboxed process Bidirectional communication cannot be

buffered until the end of a transaction Conventional system-call monitors

Call-by-call TxBox

If DENY primitive exists return error

Check WHITELIST and BLACKLIST Inspects the trace of the process Terminate and roll back

Evaluation

Performance Test TxBox with Linux 2.6.22.6 Linux with Dazuko module Quad-core Intel X5355 2.66GHz 4GB RAM

Evaluation

Micro-benchmarks Policy

BLACKLIST WREGEX *I:1234*

Worst system call for TxBox

Evaluation

Application benchmarks

Evaluation

Application benchmarks PostMark: a file system benchmark

Due to the fact that the transaction commit groups all writes and presents them to the I/O scheduler all at once, thus improving disk arm scheduling

Evaluation

Scalability Opens 100 existing files Intel Core Due 2.00 GHz with 2GB RAM

Evaluation

Original TxBoxfd =

open(“foo”, ..)

read(fd, ..)sockfd =

socket( .. )sendto =

(sockfd, .. )close(sockfd)close(fd)…

TX BEGINfd = open(“foo”, ..)read(fd, ..)sockfd = socket( .. )CHECK TX POLICY VIOLATIONNONTRANSACTIONAL sendto(sockfd, ..)close(sockfd)close(fd)…CHECK TX POLICY VIOLATIONTX END

Evaluation

Non-transactional system call sendto / reccvfrom / connect / send /

recv / ioctl / read (from a socket) / write (to a socket)

IO-intensive applications wget System time increases by 30%-40%

Evaluation

Overhead for trusted applications A single non-transactional system call

44% overhead Compile a non-transactional Linux kernel

Less than 2% overhead

Evaluation

Creating files in a protected directory make install for vim BLACKLIST WREGEX *I:164564* 164564 -> /usr/local/bin/ Correctly roll back all effects on the

system

Evaluation

Malicious MIME handler in a browser Tarhandler with lynx browser for MIME

type ‘application/x-tar’ Tarhandler reads file from /home/secret

and writes into /tmp/foo BLACKLIST WREGEX *I:183145* 183145 -> /home/secret

Evaluation

Multimedia converter ffmpeg: a popular codec

Create output files in the /home/user1/ BLACKLIST WREGEX *I:181064*

181064 -> /home/user1

Evaluation

Javascript engine Google V8 benchmark SpiderMonkey Javascript engine

Less than 5%

Evaluation

On-access anti-virus scanning and parallelization of security checks ClamAV

Small file -> the overhead of transactional mechanism dominates

Evaluation

With 4 threads, TxBox is 2.3 times better than Linux+ Dazuko

Limitations

Kernel-based security monitor is vulnerable to kernel atttacks

Transactional semantics may change the behavior of sandboxed processes Transactional conflict Auto-retry

Transactional state is not shared Some anti-virus may need to observe

the execution of the program

Limitations

Colluding malware may evade security policies No sandboxing mechanism can reliably

prevent this Processes tha generate very large

worksets are killed DoS Attack

Conclusions

Use system transactions Cannot be circumvented by

TOCTTOU Performance improvement (Multi-

core)