(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014

November 14, 2014 | Las Vegas, NV

Valentino Volonghi, CTO, AdRoll

Siva Raghupathy, Principal Solutions Architect, AWS

60 billion requests/day

We

Must

Stay

Up

1% downtime

=

>$1M

No

Infinitely

Deep

Pockets

100ms MAX Latency

Paris-New York: ~6000km

Speed of Light in fiber: 200,000 km/s

RTT latency without hops and copper:

60ms

Paris-New York: ~6000km

Speed of Light in fiber: 200,000 km/s

RTT latency without hops and copper:

60ms6000

km

60 ms

c-RTT

a

a

a

r

• Memcache

aOpen source

aMature

aBlazingly fast

rNo strong guarantees

• Redis

aOpen source

rStorage scale

rNot really distributed

rOperationally intense.

• Hbase (we still use this)

aOpen source

aMaturing quickly

aGreat scale

rReally hard to operate

http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html

Cassandra DynamoDB Delta

10/90 mix, $/month $287,064 $131,040 219%

50/50 mix, $/month $287,064 $280,800 ~0%

10/90, 3-yr reserved $27,075.6

($904k upfront)

$15,736

($504k upfront)

180%

• 10 people Cassandra ops team: $150k/month (fully loaded)

• 0 DynamoDB ops team: $0

Instance

Nginx Erlang Bidder Black Box

Supervisor

ASG

Amazon

Kinesis

Amazon S3

DynamoDB

Elastic Load

Balancing

Instance

Nginx Erlang Bidder Black Box

Supervisor

ASG

Elastic Load

Balancing

Amazon S3

Amazon

Kinesis

DynamoDB

DynamoDB

DynamoDB

HashKey Value

Key {‘segments’: set([‘123’, ‘345’]),

‘timestamp’: ‘1234556’,

‘leads’: ‘to-write-contention’}

HashKey RangeKey Value (just a single

timestamp)

Key Segment 1234554343254

Key Segment1 1231231433235

ArrayList

List WriteRequest ArrayList WriteRequest

Key

AtomicInteger AtomicInteger

Set WriteRequest CollectionsHashSet WriteRequest

String

AmazonDynamoDB

Runnable Runnable run

List WriteRequest

BatchWriteItemResult BatchWriteItemRequest

ImmutableMap List WriteRequest

Map

Not for me!

Data Collection

Data Storage

Global

Distribution

Bid Storage

Bidding

Data Collection

Data Storage

Global

Distribution

Bid Storage

Bidding

Data Collection = Batch Layer Bidding = Speed Layer

Data Collection

Data Storage

Global

Distribution

Bid Storage

Bidding

US East region

Availability Zone Availability Zone

Elastic Load

Balancing

instances instancesAuto Scaling group

Amazon S3Amazon

Kinesis

US East region

Availability Zone Availability Zone

Elastic Load

Balancing

instances instancesAuto Scaling group

Amazon S3Amazon

KinesisApache

StormDynamoDB

US West region

EU West region

DynamoDB

DynamoDB

BiddingData Collection

US East region

Availability Zone Availability Zone

Elastic Load Balancing

instances instances

Auto Scaling group

Amazon

S3

Amazon

KinesisApache

StormDynamoDB

Availability Zone Availability Zone

Auto Scaling group

Elastic Load Balancing

Data Collection

• Amazon EC2, Elastic Load Balancing, Auto Scaling

Store

• Amazon S3 + Amazon Kinesis

Global Distribution

• Apache Storm on Amazon EC2

Bid Store• DynamoDB

Bidding

• Amazon EC2, Elastic Load Balancing, Auto Scaling

Data Collection

Bidding

Ad Network 2Ad Network 1

Auto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling Group Auto Scaling GroupAuto Scaling Group

Auto Scaling GroupAuto Scaling Group Auto Scaling Group

Apache Storm

v2 V3 V3v1 v2 V3 V3v1

V2 V3 V3V1

Auto Scaling Group

V3 V4

Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing

DynamoDB

Write

Read Read Read ReadRead Read

WriteWrites

WriteWrite

Read

V3`

Elastic Load Balancing

Elastic Load Balancing

Elastic Load Balancing

Elastic Load Balancing

Elastic Load Balancing

Elastic Load Balancing

DynamoDB

Data Collection

Bidding

DynamoDB

Write

Read

Read

Write

Write

WriteAmazon S3

Amazon Kinesis

http://bit.ly/awsevals