Secure Tera-scale Data Crunching with a Small TCBbvavala/pub/dsn2017slides.pdf · -If Supervisor is...

Secure Tera-scale Data Crunching

with a Small TCBBruno Vavala Nuno Neves Peter Steenkiste

UL / CMU UL CMU

47th IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’17)

delivering security guarantees for generic and large-scale data processing on untrusted hosts

delivering security guarantees for large-scale data processing

on untrusted hosts with a small TCB

Goal

2

3

delivering security guarantees for large-scale data processing

on untrusted hosts with a small TCBtrusted HW based data

integrity

4

delivering security guarantees for large-scale data processing

on untrusted hosts with a small TCB

1 TB

5

delivering security guarantees for large-scale data processing

on untrusted hosts with a small TCB

small code

small interface

No HW devices

Some use cases

6

public cloud service provider

Some use cases

computational genomics

7

public cloud service provider

0.3TB per genome

…more generally…

Model trusted hardware module

SV

P

9

Model

1. provide state authentication data

2. outsource large state

SV

P

10

Model3. send request

SV

P

1. provide state authentication data

2. outsource large state

11

Model3. send request

4. execute command

SV

P

1. provide state authentication data

2. outsource large state

12

Model3. send request

5. receive authenticated reply

SV

P

1. provide state authentication data

2. outsource large state

4. execute command

13

Outline

• Goal • Previous Work • Our solution: key ideas and

overview • Evaluation

Outline

• Goal • Previous Work • Our solution: key ideas and

overview • Evalution

Haven (OSDI’14)

picoprocess

host OS

enclave

service

libOS

interface

— designed for Intel SGX — large TCB (due to libOS) — 10s of new interface calls + works with unmodified applications

VHD

16

VC3 (IEEE S&P’15)

process

host OS

enclave

map & reduce functions

job execution protocol

narrow interface

— designed for Intel SGX — specific for Hadoop + small TCB + data confidentiality + can run unmodified Hadoop applications

17

small TCB

Large State

Interface calls

App Specific

Trusted Computing

arch.

Haven (OSDI’14) No Yes tens No SGX

VC3 (S&P’15) Yes Yes

MapReduce workloads

R,W Yes SGX

XMHF-TrustVisor

(S&P’13,’10)Yes No none

(but Minibox has tens)

No TPM / TXT

LaStGT Yes Yes zero! No TV&SGX

A Niche in the State of the Art

18

Outline

• Goal • Previous Work • Our solution: key ideas and

overview • Evalution

untrusted env.

trusted env.

20

Scenario: two execution environments

untrusted env.

trusted env.

app’s execution flow

21

the service code is running

untrusted env.

trusted env.

access data in block bi

is bi in memory?

app’s execution flow

22

the service code accesses data in memory

untrusted env.

trusted env.

access data in block bi

is bi in memory?

app’s execution flow

keep goingyes

23

when data is available, there are no interruptions

untrusted env.

trusted env.

access data in block bi

is bi in memory?

app’s execution flow

handle page fault

load data

keep goingyes

no

24

otherwise, the service is interrupted anddata memory pages are loaded

untrusted env.

trusted env.

access data in block bi

is bi in memory?

app’s execution flow

handle page fault

load data

validate data

keep goingyes

no

25

data is validated inside trusted environment,independently from service execution

untrusted env.

trusted env.

access data in block bi

is bi in memory?

app’s execution flow

handle page fault

load data

validate data

keep goingyes

no

resume

26

service is resumed andonly if data is valid, service can make progress

…in practice…

trusted

untrustedUntrusted address

space

HardwareSGX/TPM

Architecture

other untrusted services

Trusted address space

state handler

service code

SMM(State map manager)

OSSupervisor

28

untrusted

untrustedUntrusted address

space

HardwareSGX/TPM

Architecture

other untrusted services

Trusted address space

state handler

service code

SMM(State map manager)

OSSupervisor

29

on TrustVisor, Supervisor is trusted on SGX, Supervisor is untrusted

LaStGT in 5 steps

•Offline data protection at the source •State registration •Data processing •Lazy loading from memory & disk •Execution verification

4

•Offline data protection at the source

•State registration •Data processing •Lazy loading from memory & disk •Verification

1

0 1 0 0 0 1 1 1 1 0 1 0 0 0

Data protectionHierarchical

- Incremental as data is createdMade for:

- Incremental validation as data is loaded

- Fast verification - Single hash tree is

unsuitable210

220

230

240

220 225 230 235 240 245 250

Tree

siz

e (b

ytes

)

State size (bytes)

bytes/block 210

215

220

225

230

0 1 0 0 0 1 1 1 1 0 1 0 0 0

32

State Hierarchy

files

blocks

chunk chunkchunks

masterchunkmaster chunks

directorydirectory

state rootroot

33

State Hierarchy

34

state root

directory

masterchunk

chunk chunkmasterchunk

chunk chunk

- components are loaded separately

- unneeded components not loaded in memory

- state root (1 hash) allows state validation

•Offline data protection at the source •State registration •Data processing •Lazy loading from memory & disk •Verification

2

Untrusted address space

Trusted address space

SMM(State map manager)

OSSupervisor

state handler

service code

When the trusted execution environment is created, only the code is available inside

36

Untrusted address space

Trusted address space

SMM(State map manager)

OSSupervisor

state handler

service code

grab root from disk

37

Untrusted address space

Trusted address space

SMM(State map manager)

OSSupervisor

state handler

service code

grab root from diskregister state

- registration is the first execution

- state handler installs root

- root is trusted

38

Untrusted address space

Trusted address space

SMM(State map manager)

OSSupervisor

state handler

service code

state root

39

- state root is available before service code runs

•Offline data protection at the source •State registration •Data processing •Lazy loading from memory & disk •Verification

3

- service code has view of entire state - state not readily available: inefficient loading it upfront

pages NOT availablepages available

OSSupervisor

Untrusted address space

Trusted address space

SMM

41

state handler

service code

state root data

page hit on access

- Service code execution begins - Service accesses data in memory - Data retrieval is fast if data is already available

OSSupervisor

Untrusted address space

Trusted address space

SMM

42

state handler

service code

state root data

page miss on access

OSSupervisor

- Service code may access data on missing pages

Untrusted address space

Trusted address space

SMM

43

state handler

service code

state root data

- A page fault is triggered - Execution is interrupted,

seamlessly waiting to continue

OSSupervisor

page fault!

Untrusted address space

Trusted address space

SMM

44

state handler

service code

state root data

•Offline data protection at the source •State registration •Data processing •Lazy loading from memory & disk •Verification

4

Untrusted address space

Trusted address space

SMM

OSSupervisor

page fault!

46

state handler

service code

state root data

Untrusted address space

Trusted address space

OSSupervisor

page fault!

page address

grab state component from disk- Let SMM handle missing data

- SMM loads data from disk

SMM

47

state handler

service code

state root data

Untrusted address space

Trusted address space

OSSupervisor

page fault!

SMM data

validate data

- in TrustVisor, validate in place - in SGX, copy, validate, copy

48

state handler

service code

state root data

Untrusted address space

Trusted address space

OSSupervisor

page fault!

SMM data

data is valid

- If Supervisor is trusted,invalid data => no resume (e.g.: TrustVisor)

- If Supervisor is untrustedinvalid data => no accept,so no access (e.g.: SGX)

49

state handler

service code

state root data

Untrusted address space

Trusted address space

OSSupervisor

SMM data

page hit on access

resume

fault solved, data accessible on resume, continue…

50

state handler

service code

state root data

•Offline data protection at the source •State registration •Data processing •Lazy loading from memory & disk •Execution verification

5- HW-based attestation of code identity,

including input request, state root, output reply, nonce

- Client checks validity of attestationand intended identities/hashes

Outline

• Goal • Previous Work • Our solution: key ideas and

overview • Implemention(s) • Evaluation

TCB size

VC3 HavenLaStGT

hypervisor library SQLite (example)

KSLoC (lines of

code x 1000)9.2 O(103) 17 7.7 92.6

library is small compared to real service

SGX-basedTPM/TXT

based

53

load&hash data upfront

LaStGT entry/exit & block validation

ComparisonXMHF-TrustVisor vs. LaStGT

024681012

0MB

128MB

256MB

384MB

512MB

seconds

TrustVisorLaSt-GT

LaStGT is Incremental, Faster & Scalable

LaStGT chunk loading

54

SQLite on LaStGT

0.0

0.2

0.4

128MB

256MB

512MB

1GB

2GB

... 0.25TB

3.2

3.4

3.6seconds

 0

 5

10

15

20

1MB

2MB

4MB

8MB

16MB

32MB

64MB

128MB

256MB

512MB

1GB

2GB

... 0.25TB

seconds

- First large-scale experiment on hypervisor

- Data I/O can be optimized through state hierarchy

- SGX expected to improve substantially

55

Conclusions• Security for large-scale data processing can

be guaranteed with a small TCB

• Virtual memory-based data handling => zero interface

• No change to source code => easy integration

• One design can fit diverse HW & SW

Secure Tera-scale Data Crunching with a Small TCBBruno Vavala1,2, Nuno Neves1, Peter Steenkiste2

1LaSIGE, Faculdade de Ciencias, Universidade de Lisboa, Portugal 2CSD, Carnegie Mellon University, U.S.

Abstract—Outsourcing services to third-party providers comeswith a high security cost—to fully trust the providers. Us-ing trusted hardware can help, but current trusted executionenvironments do not adequately support services that processvery large scale datasets. We present LASTGT, a system thatbridges this gap by supporting the execution of self-containedservices over a large state, with a small and generic trustedcomputing base (TCB). LASTGT uses widely deployed trustedhardware to guarantee integrity and verifiability of the executionon a remote platform, and it securely supplies data to theservice through simple techniques based on virtual memory. Asa result, LASTGT is general and applicable to many scenariossuch as computational genomics and databases, as we showin our experimental evaluation based on an implementation ofLASTGT on a secure hypervisor. We also describe a possibleimplementation on Intel SGX.

1. INTRODUCTIONOutsourced applications such as cloud services (databases,

storage, etc.) are widely deployed but strong security guaran-tees are taken for granted. The de facto security model assumesthat the service provider is fully trusted. In the real world, how-ever, one third of the top threats listed by the Cloud SecurityAlliance [33] concern an attacker tampering with the integrityof computation or data, namely: (i) service hijacking [38], (ii)malicious insiders [1], (iii) system vulnerabilities [39], and (iv)shared technology issues [2]. This can raise suspicions on thetrustworthiness of the results produced by a service.

The above threats stem from at least three issues:• the lack of strong execution isolation, whereby a sub-

verted OS, or hypervisor, or service application, can makethreats affect other running software.

• a large TCB, which makes systems hard to verify; also,when it includes the OS—containing millions of lines ofcode [35]—a bug in the kernel [34] endangers security ofall the applications and data.

• a complex OS interface—hundreds of system calls—which is difficult to secure [30] and whose maliciousalteration can subvert an application [3].

Unfortunately, service owners and end-users have little or nomeans to distinguish between correct and compromised servicecode or input data by just looking at the results received fromthe cloud.

Trusted Computing (TC) technology is making progresstowards allowing clients to verify results. The technology (e.g.,Trusted Platform Modules (TPMs) [4] and Intel SGX [29]) isavailable in commodity platforms, and it is tied to a hardwareroot of trust certified by the manufacturer. This can be usedby a service provider to isolate the service execution and toattest the identity of the executed code for remote verification.

Software support for such trusted hardware however is not(or just partially) suitable for many applications that processa huge amount of data (e.g., clinical decision support [5],predictive risk assessment for diseases [6], malware detection[7], workloads for sensitive financial records outsourced onpublic clouds [8], and genome analytics [9]). Previous systems

support the execution of either small pieces of code and data[10], or large code bases [11], or specific software like databaseengines [12] or MapReduce applications [13]. Recent work[14] has shown how to support unmodified services. However,since ”the interface between modern applications and operatingsystems is so complex” [30], it relies on a considerable TCBthat includes a library OS. In addition, the above systems arespecific for TPMs [10], [15], secure coprocessors [12], or IntelSGX [13]. Hence, porting them to alternative architectures(e.g., the upcoming AMD Secure Memory Encryption andSecure Encrypted Virtualization [36], [37]) requires significanteffort. Clearly, it is desirable to design a generic system “notrelying on idiosyncratic features of the hardware” [16].

We present LASTGT, a system that can handle a LArgeSTate on a Generic Trusted component with a small TCB.LASTGT supports a wide range of applications and hardwarebecause its design only relies on commonly available hardwarefeatures—mainly paged virtual memory. LASTGT uses mem-ory maps that allow the application to manage the placementof data in memory, and authenticated data structures forefficiently validating the data before it is processed. As mostof the LASTGT’s mechanisms (e.g., data validation and mem-ory management) are implemented at the application level,they can be optimized for different application requirements.LASTGT ultimately delivers the following guarantee: if theclient can verify the results attested by the trusted componenton the service provider platform, then the client request wasprocessed by the intended code on the intended input state, sothe received response can be trusted.

We provide the following contributions.• We describe LASTGT’s design, and show how it can

protect large-scale data in memory efficiently and how itenables a client to verify the correctness of service code,data and results.

• We detail how LASTGT has been implemented on XMHF-TrustVisor [10] using a commodity platform equippedwith a TPM. Also, we discuss a possible implementationusing the Intel SGX instruction set. In addition, we high-light important differences between the two architecturesand how LASTGT deals with them.

• We evaluate our XMHF-TrustVisor-based implementationfor datasets up to one terabyte. We show that LASTGT hasa small TCB compared to state-of-the-art prototypes, andgood performance. We also discuss expected improve-ments with an SGX-based implementation.

2. RELATED WORKWe describe related work on trusted execution, trusted

execution targeting large-scale data, and other solutions forensuring the integrity of computation on large data.Trusted Execution Environments. TrustVisor [10], Mini-box [15] and Haven [14] all support secure execution. Thefirst two focus on keeping the TCB small by removing theOS from the trust boundaries, thus supporting self-containedapplications (i.e., with statically linked libraries and no OS

ad maiora.

IEEE/IFIP DSN’17 Bruno Vavala, UL / CMU, bvavala@cs.cmu.edu

(blank)

58

Steve Wozniak

“Never trust a computer you can’t throw out the window.”

59

(excerpt from) Intel’s Legal Desclaimer

“No computer system can be absolutely secure.”

60