Consistency-Based Service Level Agreements

for Cloud StorageDouglas B. Terry, Vijayan Prabhakaran, Ramakrishna Kotla,

Mahesh Balakrishnan, Marcos K. Aguilera, Hussam Abu-Libdeh

Microsoft Research

2

“A foolish consistency is the hobgoblin of little minds”

-- Ralph Waldo Emerson (1841)

“… and of large clouds”-- Douglas Brian Terry (2013)

3

Today’s Cloud Storage Providers

• Replicate data widely• Offer choice of strong or eventual

consistencye.g. Amazon DynamoDB, Yahoo PNUTS, Google App Engine, Oracle NoSQL, Cassandra, … Microsoft Windows Azure

• Tradeoff consistency, availability and performance

4

Problem

• Developers must choose consistency • No single choice is best for all clients and situations

Client

ConsistencyU.S.

(secondary)England (primary)

India (secondary)

China (client only)

strong 147.5 1.2 435.5 307.23

eventual 1.1 1.0 1.1 160.2

Shopping cart: Want read in

under 300 ms.

roundtrip times in milliseconds

5

Pileus key features

•Replicated, partitioned key-value store•Choice of consistency•Consistency-based service level

agreements (SLAs)

a cap cloud

6

Pileus System Model

primary core

secondary nodes

Get(key, SLA)

Put(key, value)

Get(key, SLA)

syncreplication

lazyreplication

Get(key, SLA)

API

BeginSession (SLA)BeginTx (SLA)

Put (key, value)Get (key, SLA) returns value, consistency

EndTx ()EndSession ()

7

Read Consistency GuaranteesStrong Consistency Return value of latest Put.

Causal Consistency Return value of latest causally preceding Put.

Bounded Staleness (t) Return value that is stale by at most t seconds.

Read My Writes Return value of latest Put in client session or a later value.

Monotonic Reads Return same or later value as earlier Get in client session.

Eventual Consistency Return value of any Put.

[COPS 2011]

[TACT 2002]

[Bayou 1994]

8

Read LatenciesClient/Consistency U.S.

(secondary)England

(primary)India

(secondary)China

(client only)

strong 147.5 1.2 435.5 307.3

causal 146.3 1.0 431.6 306.4

bounded(30) 75.1 1.0 234.6 241.9

read-my-writes 13.0 1.1 18.4 166.8

monotonic 1.1 1.0 1.1 160.2

eventual 1.1 1.0 1.1 160.2

roundtrip times in milliseconds

consistency affects latency

client location affects latency

9

Consistency-based SLA

• Applications declare desired consistency/latency

strong 300 ms.consistency latency

read my writes 300 ms.

1.

2.

1.0utility

0.5

eventual 300 ms.3. 0.1

Shopping Cart:

10

SLA Enforcement: Client Monitoring

Node Primary? RTTs High Timestamp

A yes 210

B no 166

C no 203

For each tablet:

from configuration

service

measured on Gets, Puts, and pings

returned from Gets, Puts, and

pings

11

SLA Enforcement: Node Selection

1. For each subSLA and node, a. compute Platency

b. compute Pconsistency

c. compute Platency x Pconsistency x utility2. Select node with maximum expected utility3. Send Get operation to node4. Measure RTT and update records5. Return data and delivered consistency to caller

On Get (key, SLA):

12

Experimental SetupSystem configuration:

Primary: EnglandSecondaries: U.S., IndiaClients: U.S., England, India, China

Benchmark:• YCSB with 50/50 Gets/Puts• 500-op sessions

Node selection schemes:• Primary = get from primary• Random = get from random node• Closest = get from closest node• Pileus = get from node with highest

expected utility

Measurement: • Average utility for Get operations

U.S.England

China

India

149

287436

161

308

181

13

Experiment #1: SLA

Simplified shopping cart SLA:

consistency latency

read my writes 300 ms.1.

2.

utility

1.0

eventual 300 ms. 0.5

14

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

15

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

Primary selection works well when close to the primary,

but poorly when distant

16

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

Random selection rarely works well

17

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

100% Gets from England;

100% meet top subSLA

18

Aver

age

utilit

ype

r Get

Experiment #1: Delivered Utility

Client datacenter(secondary) (secondary)(primary) (client only)

91% from U.S.;9% from England;100% meets top

subSLA

14.5 ms. avg. latency

vs. 148 ms. for primary

19

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

99.6% from U.S.;0.4% from India; 96% meets top

subSLA

20

(secondary) (secondary)(primary) (client only)Client datacenter

Experiment #1: Delivered Utility

Pileus always delivers the most utility!

Aver

age

utilit

ype

r Get

21

Experiment #1: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter(secondary) (secondary)(primary) (client only)

9% fail to meet read-my-write

22

Experiment #2: SLA

Password checking SLA:

consistency latency

strong 150 ms.1.

2.

utility

1.0

eventual 150 ms. 0.5

3. strong 1000 ms. 0.25

23

Experiment #2: Delivered UtilityAv

erag

e uti

lity

per G

et

Client datacenter

(secondary) (secondary)(primary) (client only)

Conclusions: Main Contributions

Our Pileus system• provides a broad choice of consistency guarantees and

range of delivered read latency• allows declarative specification of desired consistency

and latency through consistency-based SLAs• selects nodes to maximize expected utility while

adapting to varying conditions