Towards Trusted Cloud Computing

Nuno Santos Krishna P. Gummadi Rodrigo Rodrigues

MPI-SWS

Abstract

Cloud computing infrastructures enable companies to cutcosts by outsourcing computations on-demand. How-ever, clients of cloud computing services currently haveno means of verifying the confidentiality and integrity oftheir data and computation.

To address this problem we propose the design of atrusted cloud computing platform (TCCP). TCCP en-ables Infrastructure as a Service (IaaS) providers suchas Amazon EC2 to provide a closed box execution envi-ronment that guarantees confidential execution of guestvirtual machines. Moreover, it allows users to attest tothe IaaS provider and determine whether or not the ser-vice is secure before they launch their virtual machines.

1 Introduction

Companies can greatly reduce IT costs by offloadingdata and computation to cloud computing services. Still,many companies are reluctant to do so, mostly due tooutstanding security concerns. A recent study [2] sur-veyed more than 500 chief executives and IT managersin 17 countries, and found that despite the potentialbenefits, executives “trust existing internal systems overcloud-based systems due to fear about security threatsand loss of control of data and systems”. One of themost serious concerns is the possibility of confidential-ity violations. Either maliciously or accidentally, cloudprovider’s employees can tamper with or leak a com-pany’s data. Such actions can severely damage the repu-tation or finances of a company.

In order to prevent confidentiality violations, cloudservices’ customers might resort to encryption. Whileencryption is effective in securing data before it is storedat the provider, it cannot be applied in services wheredata is to be computed, since the unencrypted data mustreside in the memory of the host running the computa-tion. In Infrastructure as a Service (IaaS) cloud services

such as Amazon’s EC2, the provider hosts virtual ma-chines (VMs) on behalf of its customers, who can doarbitrary computations. In these systems, anyone withprivileged access to the host can read or manipulate acustomer’s data. Consequently, customers cannot protecttheir VMs on their own.

Cloud service providers are making a substantial effortto secure their systems, in order to minimize the threatof insider attacks, and reinforce the confidence of cus-tomers. For example, they protect and restrict accessto the hardware facilities, adopt stringent accountabil-ity and auditing procedures, and minimize the numberof staff who have access to critical components of theinfrastructure [8]. Nevertheless, insiders that administerthe software systems at the provider backend ultimatelystill possess the technical means to access customers’VMs. Thus, there is a clear need for a technical solu-tion that guarantees the confidentiality and integrity ofcomputation, in a way that is verifiable by the customersof the service.

Traditional trusted computing platforms like Terra [4]take a compelling approach to this problem. For ex-ample, Terra is able to prevent the owner of a physi-cal host from inspecting and interfering with a compu-tation. Terra also provides a remote attestation capabilitythat enables a remote party to determine upfront whetherthe host can securely run the computation. This mecha-nism reliably detects whether or not the host is runninga platform implementation that the remote party trusts.These platforms can effectively secure a VM running ina single host. However, many providers run data cen-ters comprising several hundreds of machines, and a cus-tomer’s VM can be dynamically scheduled to run on anyone of them. This complexity and the opaqueness of theprovider backend creates vulnerabilities that traditionaltrusted platforms cannot address.

This paper proposes atrusted cloud computing plat-form (TCCP) for ensuring the confidentiality and in-tegrity of computations that are outsourced to IaaS ser-

C M N 1 N 2N 3 N 4U s e r P u b l i cN e t w o r kI a a S P e r i m e t e rC l u s t e r S y s a d m i n

Figure 1: Simplified architecture of Eucalyptus.

vices. The TCCP provides the abstraction of a closed boxexecution environment for a customer’s VM, guarantee-ing that no cloud provider’s privileged administrator caninspect or tamper with its content. Moreover, before re-questing the service to launch a VM, the TCCP allows acustomer to reliably and remotely determine whether theservice backend is running a trusted TCCP implementa-tion. This capability extends the notion of attestation tothe entire service, and thus allows a customer to verify ifits computation will run securely.

In this paper we show how to leverage the advancesof trusted computing technologies to design the TCCP.Section 2 introduces these technologies and describes thearchitecture of an IaaS service. Section 3 presents ourdesign of TCCP. Although we do not yet have a work-ing prototype of TCCP, the design is sufficiently detailedthat we are confident that a solution to the problem underdiscussion is possible.

2 Background

2.1 Infrastructure as a Service

Today, myriads of cloud providers offer services at vari-ous layers of the software stack. At lower layers, Infras-tructure as a Service (IaaS) providers such as Amazon,Flexiscale, and GoGrid allow their customers to haveaccess to entire virtual machines (VMs) hosted by theprovider. A customer, and user of the system, is respon-sible for providing the entire software stack running in-side a VM. At higher layers, Software as a Service (SaaS)systems such as Google Apps offer complete online ap-plications than can be directly executed by their users.

The difficulty in guaranteeing the confidentiality ofcomputations increases for services sitting on higher lay-ers of the software stack, because services themselvesprovide and run the software that directly manipulatescustomer’s data (e.g., Google Docs). In this paper wefocus on the lower layer IaaS cloud providers where se-curing a customer’s VM is more manageable.

While very little detail is known about the internal or-ganization of commercial IaaS services, we describe (andbase our proposal on) Eucalyptus [6], an open sourceIaaS platform that offers an interface similar to EC2. Fig-

ure 1 presents a very simplified architecture of Eucalyp-tus. This system manages one or more clusters whosenodes run a virtual machine monitor (typically Xen) tohost customers’ VMs. Eucalyptus comprehends a set ofcomponents to manage the clusters. For simplicity, ourdescription aggregates all these components in a singlecloud manager (CM) that handles a single cluster; werefer the reader to [6] for more details.

From the perspective of users, Eucalyptus provides aweb service interface to launch, manage, and terminateVMs. A VM is launched from a virtual machine image(VMI) loaded from the CM. Once a VM is launched,users can log in to it using normal tools such as ssh.Aside from the interface to every user, the CM exportsservices that can be used to perform administrative taskssuch as adding and removing VMIs or users. Xen sup-ports live migration, allowing a VM to shift its physicalhost while still running, in a way that is transparent to theuser. Migration can be useful for resource consolidationor load balancing within the cluster.

2.2 Attack model

A sysadmin of the cloud provider that has privileged con-trol over the backend can perpetrate many attacks in or-der to access the memory of a customer’s VM. With rootprivileges at each machine, the sysadmin can install orexecute all sorts of software to perform an attack. Forexample, if Xen is used at the backend, Xenaccess [7] al-lows a sysadmin to run a user level process in Dom0 thatdirectly accesses the content of a VM’s memory at runtime. Furthermore, with physical access to the machine,a sysadmin can perform more sophisticated attacks likecold boot attacks and even tamper with the hardware.

In current IaaS providers, we can reasonably considerthat no single person accumulates all these privileges.Moreover, providers already deploy stringent securitydevices, restricted access control policies, and surveil-lance mechanisms to protect the physical integrity of thehardware. Thus, we assume that, by enforcing a secu-rity perimeter, the provider itself can prevent attacks thatrequire physical access to the machines.

Nevertheless, sysadmins need privileged permissionsat the cluster’s machines in order to manage the softwarethey run. Since we do not precisely know the praxis ofcurrent IaaS providers, we assume in our attack modelthat sysadmins can login remotely to any machine withroot privileges, at any point in time. The only way asysadmin would be able to gain physical access to a noderunning a costumer’s VM is by diverting this VM to amachine under her control, located outside the IaaS’s se-curity perimeter. Therefore, the TCCP must be able to1) confine the VM execution inside the perimeter, and 2)guarantee that at any point a sysadmin with root privi-

leges remotely logged to a machine hosting a VM cannotaccess its memory.

2.3 Trusted Computing

The Trusted Computing Group (TCG) [10] proposed aset of hardware and software technologies to enable theconstruction of trusted platforms. In particular, the TCGproposed a standard for the design of thetrusted platformmodule (TPM) chip that is now bundled with commodityhardware. The TPM contains an endorsement private key(EK) that uniquely identifies the TPM (thus, the physi-cal host), and some cryptographic functions that cannotbe modified. The respective manufacturers sign the cor-responding public key to guarantee the correctness of thechip and validity of the key.

Trusted platforms [1, 4, 5, 9] leverage the features ofTPM chips to enableremote attestation. This mecha-nism works as follows. At boot time, the host computes ameasurement listML consisting of a sequence of hashesof the software involved in the boot sequence, namelythe BIOS, the bootloader, and the software implementingthe platform. TheML is securely stored inside the host’sTPM. To attest to the platform, a remote party challengesthe platform running at the host with a noncenU . Theplatform asks the local TPM to create a message contain-ing both theML and thenU , encrypted with the TPM’sprivate EK. The host sends the message back to theremote party who can decrypt it using theEK ’s corre-sponding public key, thereby authenticating the host. Bychecking that the nonces match and theML correspondsto a configuration it deems trusted, a remote party canreliably identify the platform on an untrusted host.

A trusted platform like Terra [4] implements a thinVMM that enforces aclosed box execution environment,meaning that a guest VM running on top cannot be in-spected or modified by a user with full privileges overthe host. The VMM guarantees its own integrity until themachine reboots. Thus, a remote party can attest to theplatform running at the host to verify that a trusted VMMimplementation is running, and thus make sure that hercomputation running in a guest VM is secure.

Given that a traditional trusted platform can secure thecomputation on a single host, a natural approach to se-cure an IaaS service would be to deploy the platform ateach node of the service’s backend (see Figure 1). How-ever, this approach is insufficient: a sysadmin can diverta customer’s VM to a node not running the platform, ei-ther when the VM is launched (by manipulating the CM),or during the VM execution (using migration). Conse-quently, the attestation mechanism of the platform doesnot guarantee that the measurement list obtained by theremote party corresponds to the actual configuration ofthe host where the VM has been running (or will be run-

C M N 1 N 2N 3 N 4U s e rI a a S P e r i m e t e rT C C P S y s a d m i nE T ET C

Figure 2: The components of the trusted cloud comput-ing platform include a set oftrusted nodes (N) and thetrusted coordinator (TC). The untrustedcloud manager(CM) makes a set of services available to users. The TCis maintained by anexternal trusted entity (ETE).

ning in the future). Therefore, the TCCP needs to providea remote attestation that guarantees the immutability ofthe platform’s security properties in the backend.

3 Trusted Cloud Computing Platform

We present thetrusted cloud computing platform (TCCP)that provides a closed box execution environment by ex-tending the concept of trusted platform to an entire IaaSbackend. The TCCP guarantees the confidentiality andthe integrity of a user’s VM, and allows a user to de-termine up front whether or not the IaaS enforces theseproperties. Next section gives an overview of TCCP, andSection 3.2 presents a detailed design.

3.1 Overview

TCCP enhances today’s IaaS backends to enable closedbox semantics without substantially changing the archi-tecture (Figure 2). The trusted computing base of theTCCP includes two components: atrusted virtual ma-chine monitor (TVMM), and atrusted coordinator (TC).

Each node of the backend runs a TVMM that hostscustomers’ VMs, and prevents privileged users from in-specting or modifying them. The TVMM protects itsown integrity over time, and complies with the TCCPprotocols. Nodes embed a certified TPM chip and mustgo through a secure boot process to install the TVMM.Due to space limitations we will not go into detail aboutthe design of the TVMM, and we refer the reader to [5]for an architecture that can be leveraged to build aTVMM that enforces local closed box protection againsta malicious sysadmin.

The TC manages the set of nodes that can run a cus-tomer’s VM securely. We call these nodestrusted nodes.To be trusted, a node must be located within the secu-rity perimeter, and run the TVMM. To meet these con-ditions, the TC maintains a record of the nodes locatedin the security perimeter, and attests to the node’s plat-form to verify that the node is running a trusted TVMM

N T C1 .2 .3 .4 . 1. nN

2. {MLTC , nN}EK

p

TC, n

TC

3. {{MLN , nTC}EK

p

N, TKP

N}

TKPT C

4. {accepted}TKP

N

Figure 3: Message exchange during node registration.

implementation. The TC can cope with the occurrenceof events such as adding or removing nodes from a clus-ter, or shutting down nodes temporarily for maintenanceor upgrades. A user can verify whether the IaaS servicesecures its computation by attesting to the TC.

To secure the VMs, each TVMM running at each nodecooperates with the TC in order to 1) confine the exe-cution of a VM to a trusted node, and to 2) protect theVM state against inspection or modification when it isin transit on the network. The critical moments that re-quire such protections are the operations tolaunch, andmigrate VMs. In order to secure these operations, theTCCP specifies several protocols (see Section 3.2). Dueto space constraints, we do not address other critical op-erations such assuspend/resume allowed by Xen.

We assume anexternal trusted entity (ETE) that hoststhe TC, and securely updates the information provided tothe TC about the set of nodes deployed within the IaaSperimeter, and the set of trusted configurations. Most im-portantly, sysadmins that manage the IaaS have no priv-ileges inside the ETE, and therefore cannot tamper withthe TC. We envision that the ETE should be maintainedby a third party with little or no incentive to collude withthe IaaS provider e.g., by independent companies analo-gous to today’s certificate authorities like VeriSign.

3.2 Detailed Design

In this section we detail the most relevant TCCP mech-anisms. We describe the protocols that manage the setof nodes of the platform that are trusted (Section 3.2.1),and the protocols that secure the operations involvingVM management, namely launching and migrating VMs(Section 3.2.2). In these protocols, we use the fol-lowing notation for cryptographic operations. The pair〈Kp, KP 〉 represents the private-public keys of an asym-metric cryptography keypair. Notation{y}Kx indicatesthat datay is encrypted with keyKx. We use a specificnotation for the following keys:EKx denote endorse-ment keys,TKx indicate trusted keys, andKx denotesession keys. Noncesnx, unique numbers generated byx, help detect message replays.

3.2.1 Node management

The TC dynamically manages the set of trusted nodesthat can host a VM by maintaining a directory contain-

C MN T C2 .3 . U1 .4 . F

1. {α, #α}KV M{nU , KV M}

TKPT C

2. {{{nU , KV M}TKP

T C

, nN}TK

p

N,

N}TKP

T C

3. {{nN , nU , KV M}TKP

N

}TK

p

T C

4. {nU , N}KV M

Figure 4: Message exchange during VM launch.

ing, for each node within the security perimeter, thepublic endorsement keyEKP

N identifying the node’sTPM, and the expected measurement listMLN . TheETE makes some properties of the TC securely avail-able to the public, namely theEKP

TC , theMLTC , andtheTKP

TC (identifying the TC). Both theMLN and theMLTC express the canonical configurations that a re-mote party is expected to observe when attesting to theplatform running on a node N or on the TC, respectively.

In order to be trusted, a node must register with the TCby complying to the protocol depicted on Figure 3. Insteps 1 and 2, N attests to the TC to avoid an imperson-ation of the TC by an attacker: N sends a challengenN

to the TC, and the TC replies with its bootstrap measure-mentsMLTC encrypted withEK

pTC to guarantee the

authenticity of the TC. If theMTC matches the expectedconfiguration, it means the TC is trusted. Reversely, theTC also attests to N by piggybacking a challengenTC inmessage 2, and checking whether the node is authentic,and is running the expected configuration (step 3). Thenode generates a keypair〈TK

pN , TKP

N〉, and sends itspublic key to the TC. If both peers mutually attest suc-cessfully, the TC addsTKP

N to its node database, andsends message 4 to confirm that the node is trusted. KeyTKN certifies that node N is trusted.

In the case that a trusted node reboots, the TCCP mustguarantee that the node’s configuration remains trusted,otherwise the node could compromise the security of theTCCP. To ensure this, the node only keepsTK

pN in mem-

ory causing the key to be lost once the machine reboots.The node is thus banned from the TCCP, since it will notbe able to decrypt messages encrypted with the previouskey, and must repeat the registration protocol.

3.2.2 Virtual machine management

We present the TCCP protocols to secure the VM launchand migration operations. When launching a VM, theTCCP needs to guarantee that 1) the VM is launched ona trusted node, and 2) the sysadmin is unable to inspector tamper with the initial VM state as it traverses the pathbetween the user and the node hosting the VM. The ini-tial VM stateα contains the VM image (VMI) (that canbe personalized and contain secret data) and the user’s

C MN s T C6 . N dV M 3 .7 . 4 . 5 .1 .2 .

1. {{Nd, ns1}TKp

N, Ns}TKP

T C

2. {{ns1, TKP

Nd}

TKPNs

}TK

p

T C

3. {{KS , ns2}TKp

Ns

, Ns}TKPNd

4. {{Ns, nd}TKp

Nd

, Nd}TKPT C

5. {{nd, TKP

Ns}

TKPNd

}TK

p

T C

6. {nd}KS

7. {V Mid,#V Mid}KS

Figure 5: Message exchange during VM migrate.

public key (used for ssh login)1. In practice, the user candecide to use a VMI provided by the IaaS.

To enforce these requirements, the parties involved inlaunching a VM follow the protocol depicted in Figure 4.The protocol is designed on the fact that, before launch-ing the VM, a user does not know which physical nodethe VM will be assigned, and, among the components ofthe service, only trusts the TC. First, the user generates asession keyKV M , and sends message 1 to the CM con-taining: α andα’s hash encrypted with the session key(to protect the confidentiality and integrity of the initialstate), andKV M encrypted withTKP

TC . Encrypting thesession key with the TC’s public key ensures that onlythe TC can authorize someone to accessα. The TC onlyauthorizes trusted nodes.

Upon receiving the request to launch a VM, the CMdesignates a node N from the cluster to host the VM, andforwards the request to N. Since the node needs to ac-cessα in order to boot the VM, it sends message 2 toTC which decryptsKV M on N’s behalf. This message isencrypted withTK

pN so that the TC can verify whether

N is trusted. If the corresponding public key is not foundin the TC’s trusted node database, the request is denied.This would have been the case had the CM diverted therequest to a node controlled by a malicious sysadmin.Otherwise, the node is reckoned to be trusted; the TCdecrypts the session key, and sends it to the node in mes-sage 3, such that only N can read the key. N is now ableto decryptα, and boot the VM. Finally, the node sendsmessage 4 to the user containing the identity of the noderunning the VM.

In live migration [3], the state of an executing VM istransfered between two nodes: a source Ns and a des-tination Nd. To secure this operation, both nodes mustbe trusted, and the VM state must remain confidentialand unmodified while it is in transit over the network.Figure 4 shows the sequence of messages involved in se-curing the migration of a VM. In steps 1 and 2, Ns asksTC to check whether Nd is trusted. In message 3, Ns ne-gotiates a session keyKS with Nd that will be used to

1In current IaaS services, the user public key is injected in the VMat launch time. A possible attack could be to inject more keysor othermalicious software.

secure the transfer of the VM state. Before accepting thekey, Nd first verifies that Ns is trusted (steps 4 and 5).If both nodes mutually authenticate successfully, Nd ac-knowledges the acceptance of the session key to theKS

(step 6), and, in message 7, Ns finally transfers the en-crypted and hashed VM state to the Nd, guaranteeing theconfidentiality and integrity of the VM.

4 Conclusions and Future Work

In this paper, we argue that concerns about the confiden-tiality and integrity of their data and computation are amajor deterrent for enterprises looking to embrace cloudcomputing. We present the design of atrusted cloudcomputing platform (TCCP) that enables IaaS servicessuch as Amazon EC2 to provide a closed box executionenvironment. TCCP guarantees confidential execution ofguest VMs, and allows users to attest to the IaaS providerand determine if the service is secure before they launchtheir VMs. We plan to implement a fully functional pro-totype based on our design and evaluate its performancein the near future.

References

[1] S. Berger, R. Caceres, K. A. Goldman, R. Perez, R. Sailer, andL. van Doorn. vTPM: virtualizing the trusted platform module.In Proc. of USENIX-SS’06, Berkeley, CA, USA, 2006.

[2] Survey: Cloud Computing ’No Hype’, But Fear ofSecurity and Control Slowing Adoption. http://www.circleid.com/posts/20090226 cloudcomputing hype security/.

[3] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach,I. Pratt, and A. Warfield. Live migration of virtual machines.In Proc. of NSDI’05, pages 273–286, Berkeley, CA, USA, 2005.USENIX Association.

[4] T. Garfinkel, B. Pfaff, J. Chow, M. Rosenblum, and D. Boneh.Terra: A Virtual Machine-Based Platform for Trusted Comput-ing. In Proc. of SOSP’03, 2003.

[5] D. G. Murray, G. Milos, and S. Hand. Improving Xen securitythrough disaggregation. InProc. of VEE’08, pages 151–160, NewYork, NY, USA, 2008.

[6] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman,L. Youseff, and D. Zagorodnov. Eucalyptus: A Technical Re-port on an Elastic Utility Computing Architecture Linking YourPrograms to Useful Systems. Technical Report 2008-10, UCSBComputer Science, 2008.

[7] B. D. Payne, M. Carbone, and W. Lee. Secure and Flexible Mon-itoring of Virtual Machines. InProc. of ACSAC’07, 2007.

[8] T. R. Peltier, J. Peltier, and J. Blackley.Information SecurityFundamentals. Auerbach Publications, Boston, MA, USA, 2003.

[9] R. Sailer, T. Jaeger, E. Valdez, R. Caceres, R. Perez, S. Berger,J. L. Griffin, and L. v. Doorn. Building a MAC-Based SecurityArchitecture for the Xen Open-Source Hypervisor. InProc. ofACSAC ’05, Washington, DC, USA, 2005.

[10] TCG. https://www.trustedcomputinggroup.org.