© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scaling on AWS for the

First 10 Million UsersMohan Vedula, Enterprise Solutions Architecture

Brett Francis, Enterprise Solutions Architecture

March 26, 2014

Scaling on AWS for the First 10 Million Users

• US: – Brett Francis – brettf@amazon.com

– Mohan Vedula – mohanv@amazon.com

• YOU: Here to learn more about scaling

infrastructure on AWS

• TODAY: about best practices and things to think

about when building for large scale

Scaling to 10M Users: A Story in Four Parts

• Intro and Initial Steps

• Building Blocks

• Tools and Monitoring

• 10M Users and Beyond

So how do we scale?

a lot of things to read

not where we want to start

a lot of things to read

us today

Auto Scaling is a tool and a

destination. It’s not the single

thing that fixes everything.

What do we

need first?

Some basics…

RegionsUS-WEST (Oregon)

EU-WEST (Ireland)

ASIA PAC (Tokyo)

US-WEST (N. California)

SOUTH AMERICA (Sao Paulo)

US-EAST (Virginia)

AWS GovCloud (US)

ASIA PAC

(Sydney)

ASIA PAC

(Singapore)

CHINA (Beijing)

Availability ZonesUS-WEST (Oregon)

EU-WEST (Ireland)

ASIA PAC (Tokyo)

US-WEST (N. California)

SOUTH AMERICA (Sao Paulo)

US-EAST (Virginia)

AWS GovCloud (US)

ASIA PAC

(Sydney)

ASIA PAC

(Singapore)

CHINA (Beijing)

Edge Locations

• $7B+ retail business

• 8,000+ employees

• A whole lot of servers

Every day, AWS adds enough

server capacity to power that

entire $7B enterprise

2004 2014

Compute

Storage &

Content

Delivery

AWS Global Infrastructure

Database

App Services

Deployment & Administration

Networking

Compute

Storage &

Content

Delivery

AWS Global Infrastructure

Database

App Services

Deployment & Administration

Networking

Amazon

CloudSearchAmazon

SQS

Amazon

SNS

Amazon

Elastic

Transcoder

Amazon SWFAmazon

SES

Amazon

DynamoDB

Amazon

RDS

Amazon

ElastiCache

Amazon

RedShift

AWS Storage

Gateway

Amazon S3

Amazon

Glacier

Amazon

CloudFrontAmazon

EC2

Amazon

EMRAmazon

VPC

Amazon

Route 53AWS

Direct

Connect

Amazon

Kinesis

Amazon

CloudWatch AWS IAM AWS

CloudFormation

Amazon Elastic

BeanstalkAWS

Data

Pipeline

AWS

OpsWorksAWS

CloudTrail

So let’s start from day

one, user one ( you )

Day One, User One:

• A single EC2 instance

– With a full stack on this host

• Web app

• Database

• Management

• etc.

• A single Elastic IP address

• Amazon Route 53 for DNS

EC2

instance

Elastic IP

address

Amazon

Route 53User

“We’re gonna need a bigger box”

• Simplest approach

• Can now leverage PIOPs

• High I/O instances

• High memory instances

• High CPU instances

• High storage instances

• Easy to change instance sizes

• Will hit an endpoint eventually

m3.xlarge

m1.small

i2.4xlarge

“We’re gonna need a bigger box”

m3.xlarge

m1.small

i2.4xlarge

• Simplest approach

• Can now leverage PIOPs

• High I/O instances

• High memory instances

• High CPU instances

• High storage instances

• Easy to change instance sizes

• Will hit an endpoint eventually

Day One, User One:

• We could potentially get to a few hundred to a few thousand depending on application complexity and traffic

• No failover

• No redundancy

• Too many eggs in one basket

EC2

instance

Elastic IP

address

Amazon

Route 53User

Day One, User One:

• We could potentially get to a few hundred to a few thousand depending on application complexity and traffic

• No failover

• No redundancy

• Too many eggs in one basket

EC2

instance

Elastic IP

address

Amazon

Route 53User

Day Two, User >1:

First, let’s separate out

our single host into

more than one:

• Web

• Database– Make use of a database

service?

Web

instance

Database

instance

Elastic IP

address

Amazon

Route 53User

Self-Managed Fully-Managed

Database server

on Amazon EC2

Your choice of

database running on

Amazon EC2

Bring Your Own

License (BYOL)

Amazon

DynamoDB

Managed NoSQL

database service

using SSD storage

Seamless scalability

Zero administration

Amazon RDS

Microsoft SQL,

Oracle, MySQL or

PostgreSQL as a

managed service

Flexible licensing

BYOL or License

Included

Amazon

Redshift

Massively parallel,

petabyte-scale, data

warehouse service

Fast, powerful and

easy to scale

Database Options

Scaling to 10M Users: A Story in Four Parts

• Intro and Initial Steps

• Building Blocks

• Tools and Monitoring

• 10M Users and Beyond

But how do I choose

the DB technology I

need? SQL? NoSQL?

Some folks won’t like

this. But…

Start with SQL

databases

Unless you already

have NoSQL skilled

staff…

Start with SQL

databases

Why start with SQL?

• Established and well-worn technology

• Lots of existing code, communities, books, background,

tools, etc.

• You aren’t going to break SQL DBs in your first 10 million

users. No really, you won’t*.

• Clear patterns to scalability

* Unless you are manipulating data at MASSIVE scale; even then, SQL will have a

place in your stack

AH HA! You

said “massive

amounts”, I

will have

massive

amounts!

If your usage is such that you will be

generating several TB ( >5 ) of data

in the first year OR have an

incredibly data-intensive workload…

you might need NoSQL

Regardless, why NoSQL?

• Super low latency applications

• Metadata driven datasets

• Highly non-relational data

• Need schema-less data constructs*

• Massive amounts of data (again, in the TB range)

• Rapid ingest of data ( thousands of records/sec )

• Already have skilled staff

*Need != “it is easier to do dev without schemas”

When NoSQL = Yes…

investigate use of DynamoDB

Amazon Dynamo DB

• Managed, provisioned throughput

NoSQL database

• Fast, predictable performance

• Fully distributed, fault tolerant

architecture

• Considerations for non-uniform

data

Feature Details

Provisioned

throughput

Dial up or down provisioned

read/write capacity

Predictable

performance

Average single digit millisecond

latencies from SSD-backed

infrastructure

Strong

consistency

Be sure you are reading the

most up-to-date values

Fault

tolerant

Data replicated across

Availability Zones

Monitoring Integrated to Amazon

CloudWatch

Secure Integrates with AWS Identity

and Access Management

(AWS IAM)

Amazon

EMR

Integrates with Amazon EMR

for complex analytics on large

datasets

But back to the main

path… Let’s see how

far SQL at the core

can grow

User >100:

First, let’s separate out

our single host into

more than one:

• Web

• Database– Use Amazon RDS to make

your life easier

Web

instance

Elastic IP

address

RDS DB

instance

Amazon

Route 53User

User > 1000:

Next, let’s address the

lack of failover and

redundancy issues:• Elastic Load Balancing

• Another web instance

– In another Availability Zone

• Enable Amazon RDS Multi-AZ

Web

instance

Amazon RDS DB instance

Active (Multi-AZ)

Availability Zone Availability Zone

Web

instance

Amazon RDS DB instance

Standby (Multi-AZ)

Elastic Load

Balancing

Amazon

Route 53User

• Create highly-scalable applications

Feature Details

Available Load balance across instances in multiple

Availability Zones

Health checks Automatically checks health of instances and

takes them in or out of service

Session stickiness Route requests to the same instance

Secure sockets layer Supports SSL offload from web and application

servers with flexible cipher support

Monitoring Publishes metrics to Amazon CloudWatch

Elastic Load

Balancing

Elastic Load Balancing

Scaling this horizontally

and vertically

will get us pretty far( 10s-100s of thousands )

User >10ks-100ks:

RDS DB Instance

Active (Multi-AZ)

Availability Zone Availability Zone

RDS DB Instance

Standby (Multi-AZ)

Elastic Load

Balancing

RDS DB Instance

Read Replica

RDS DB Instance

Read Replica

RDS DB Instance

Read Replica

RDS DB Instance

Read Replica

Web

instance

Web

instance

Web

instance

Web

instance

Web

instance

Web

instance

Web

instance

Web

instance

Amazon

Route 53User

This will take us pretty far,

honestly, but we care

about performance and

efficiency, so let’s clean up

with some components and

services

Shift some load around:

Think components and

services:

• Move static content from the

web instance to Amazon S3

and Amazon CloudFront

• Move session/state and DB

caching to Amazon

ElastiCache or Amazon

DynamoDB

• More on services later…

Web

instance

RDS DB Instance

Active (Multi-AZ)

Availability Zone

Elastic Load

Balancing

Amazon S3

Amazon

CloudFront

Amazon

Route 53User

ElastiCache

Amazon

DynamoDB

Working with Amazon S3

• Object-based storage for the web

• Designed for 11 9s of durability

• Good for things like:

– Static assets (css, js, images, videos)

– Backups

– Logs

– Ingest of files for processing

• “Infinitely scalable”

• Supports fine-grained permission control

• Ties in well with CloudFront

• Ties in with Amazon EMR

• Acts as a logging endpoint for Amazon

S3/CloudFront/Billing

• Supports encryption at transit and at rest

• Reduced redundancy 1/3 cheaper

• Amazon Glacier for super long term

storage

CloudFront

Amazon CloudFront is a web service for

scalable content delivery:

• Cache static content at the edge for faster delivery

• Helps lower load on origin infrastructure

• Dynamic and static content

• Streaming video

• Zone apex support

• Custom SSL certificates

• Low TTLs (as short as 0 seconds)

• Lower costs for origin fetches (between Amazon

S3 / Amazon EC2 and CloudFront)

• Optimized to work with Amazon EC2, Amazon S3,

Elastic Load Balancing, and Amazon Route 53

Re

spo

nse

Tim

e

Serv

er

Load

Res

po

nse

Tim

e

Serv

er

Load

Res

po

nse

Tim

e

Serv

er

Load

No CDN CDN for static

content

CDN for static &

dynamic content

0

10

20

30

40

50

60

70

80

8:00AM

9:00AM

10:00AM

11:00AM

12:00PM

1:00PM

2:00PM

3:00PM

4:00PM

5:00PM

6:00PM

7:00PM

8:00PM

9:00PM

Vo

lum

e o

f D

ata

D

eliv

ere

d (

Gb

ps

)

Shift some load around:

Let’s lighten the load on our

web and database

instances:

• Move static content from the

web instance to Amazon S3

and CloudFront

• Move session/state and DB

caching to ElastiCache or

Amazon DynamoDB

Web

instance

RDS DB Instance

Active (Multi-AZ)

Availability Zone

Elastic Load

Balancer

Amazon S3

Amazon

CloudFront

Amazon

Route 53User

ElastiCache

Amazon

DynamoDB

Shift some load around:Let’s lighten the load on our web and database instances:• Move static content from the

web instance to Amazon S3 and CloudFront

• Move dynamic content from the load balancer to CloudFront

• Move session/state and DB caching to ElastiCache or Amazon DynamoDB

Web

instance

RDS DB Instance

Active (Multi-AZ)

Availability Zone

Elastic Load

Balancer

Amazon S3

Amazon

CloudFront

Amazon

Route 53User

ElastiCache

Amazon

DynamoDB

Shift some load around:Let’s lighten the load on our web and database instances:• Move static content from the

web instance to Amazon S3 and CloudFront

• Move dynamic content from the load balancer to CloudFront

• Move session/state and DB caching to ElastiCache or Amazon DynamoDB

Web

instance

RDS DB Instance

Active (Multi-AZ)

Availability Zone

Elastic Load

Balancer

Amazon S3

Amazon

CloudFront

Amazon

Route 53User

ElastiCache

Amazon

DynamoDB

Scaling to 10M Users: A Story in Four Parts

• Intro and Initial Steps

• Building Blocks

• Tools and Monitoring

• 10M Users and Beyond

Now that our web tier is

much more lightweight,

we can revisit the

beginning of our talk…

Auto Scaling!

Automatic resizing of compute clusters

based on demand

Trigger auto-scaling policy

Feature Details

Control Define minimum and maximum instance pool sizes and when scaling and cool down occurs

Integrated to Amazon CloudWatch

Use metrics gathered by CloudWatch to drive scaling

Instance types Run Auto Scaling for On-Demand and Spot Instances; compatible with VPC

aws autoscaling create-auto-scaling-group--auto-scaling-group-name MyGroup--launch-configuration-name MyConfig--min-size 4--max-size 200--availability-zones us-west-2c

Auto ScalingAmazon

CloudWatch

Sunday Monday Tuesday Wednesday Thursday Friday Saturday

Typical weekly traffic to Amazon.com

Sunday Monday Tuesday Wednesday Thursday Friday Saturday

Typical weekly traffic to Amazon.com

Provisioned capacity

November traffic to Amazon.com

November

November traffic to Amazon.com

Provisioned capacity

November

November traffic to Amazon.com76%

24%

Provisioned capacity

November

November traffic to Amazon.com

November

Auto Scaling

lets you do this!

Auto Scaling can help from

one instance to thousands

and back down

User >500k+:

Availability Zone

Amazon

Route 53User

Amazon S3

Amazon

CloudFront

Availability Zone

Elastic Load

Balancing

Amazon

DynamoDBRDS DB Instance

Read Replica

Web

instance

Web

instance

Web

instance

ElastiCache RDS DB Instance

Read Replica

Web

instanceWeb

instance

Web

instance

ElastiCacheRDS DB Instance

Standby (Multi-AZ)RDS DB Instance

Active (Multi-AZ)

Use Tools:Managing your infrastructure will become an ever

increasing important part of your time. Use tools to

automate repetitive tasks.

• Tools to manage AWS resources

• Tools to manage software and configuration on your

instances

• Automated data analysis of logs and user actions

AWS Application Management Solutions

AWS

Elastic Beanstalk

AWS

OpsWorks

AWS

CloudFormation

Amazon EC2

Convenience Control

Higher-level services Do it yourself

Host-Based Configuration Management

Two big players:– Opscode Chef

– PuppetLabs Puppet

• Both do more or less the same thing

• Both have syntax that isn’t too dissimilar

• Use HBCM with one of the tools from the previous slide

• Spend the time required to learn them

• Can’t scale easily without HBCM

• Growing in popularity: Salt, Ansible

User >500k+:You’ll potentially start to run into issues with speed and

performance of your applications:

• Have monitoring/metrics/logging in place

– If you can’t build it internally, outsource it! (3rd party SaaS)

• Pay attention to what customers are saying works well

• Squeeze as much performance as you can out of each

service/component

HOST

LEVEL

METRICS

AGGREGATE

LEVEL

METRICS

LOG

ANALYSIS

EXTERNAL

SITE

PERFORMANCE

Not having proper

monitoring/metrics is like

flying a plane with an eye

mask on in a thunderstorm.

Oh, and your wing is on fire.

AWS Marketplace & Partners Can Help

• Customer can find, research,

and buy software

• Simple pricing, aligns with

Amazon EC2 usage model

• Launch in minutes

• AWS Marketplace billing

integrated into your AWS

account

• 1300+ products across 20+

categories

Learn more at: aws.amazon.com/marketplace

Scaling to 10M Users: A Story in Four Parts

• Intro and Initial Steps

• Building Blocks

• Tools and Monitoring

• 10M Users and Beyond

There are further

improvements to be

made in breaking apart

our web/app layer

SOA = Service Oriented Architecture

SOA’ing

Move services into their own

tiers/modules. Treat each of these

as 100% separate pieces of your

infrastructure and scale them

independently.

Amazon.com and AWS do this

extensively! It offers flexibility and

greater understanding of each

component.

Loose coupling sets you free!

• The looser they're coupled, the bigger they scale

– Independent components

– Design everything as a black box

– Decouple interactions

– Favor services with built-in redundancy and scalability rather than

building your own

Controller A Controller B

Controller A Controller B

Q Q

Tight coupling

Use Amazon SQS for buffers

Loose coupling

Loose coupling + SOA = winning

Examples:

• Email

• Queuing

• Transcoding

• Search

• Databases

• Monitoring

• Metrics

• Logging

Amazon

CloudSearchAmazon SQSAmazon SNS

Amazon Elastic

TranscoderAmazon SWF

Amazon SES

In the early days, if someone has a service for it already,

opt to use that instead of building it yourself.

DON’T RE-INVENT THE WHEEL

On re-inventing the wheel…

If you find yourself writing

your own: queue, DNS server,

database, storage system,

monitoring tool

Take a deep breath and

stop it. Now.

Back to SOA

User >1mil+:Reaching a million and above is going to require some bit of all the previous things:

• Multi-AZ

• Elastic Load Balancing between tiers

• Auto Scaling

• Service-oriented architecture

• Serving content smartly (Amazon S3/CloudFront)

• Caching off DB

• Moving state off tiers that auto-scale

User >1mil+:

RDS DB Instance

Active (Multi-AZ)

Availability Zone

Elastic Load

Balancing

RDS DB Instance

Read Replica

RDS DB Instance

Read Replica

Web

instance

Web

instance

Web

instance

Web

instance

Amazon

Route 53User

Amazon S3

Amazon

CloudFront

Amazon

DynamoDB

Amazon SQS

ElastiCache

Worker

instance

Worker

instance

Amazon

CloudWatch

Internal app

instance

Internal app

instanceAmazon SES

The next big steps

User >5mil – 10mil:You’ll potentially start to run into issues with your database

around contention on the write master.

How can you solve it?

• Federation ~ splitting into multiple DBs based on function

• Sharding ~ splitting one data set up across multiple hosts

• Moving some functionality to other types of DBs (NoSQL)

Shifting functionality to NoSQL

• Similar in a sense to federation

• Again, review the earlier points to determine need of NoSQL vs SQL

• Leverage hosted services like Amazon DynamoDB

• Some use cases:– Leaderboards/scoring

– Rapid ingest of clickstream/log data

– Temporary data needs ( cart data )

– “Hot” tables

– Metadata/lookup tables

Amazon

DynamoDB

…and there you have it.

10 Million

A Quick Review

Review

• Multi-AZ your infrastructure

• Make use of self-scaling services – Elastic Load Balancing, Amazon S3, Amazon SNS, Amazon SQS,

Amazon SWF, Amazon SES, etc.

• Build in redundancy at every level

• Most likely start with SQL

• Cache data both inside and outside your infrastructure

• Use automation tools in your infrastructure

Review (cont)

• Make sure you have good

metrics/monitoring/logging tools in place

• Split tiers into individual services (SOA)

• Use Auto Scaling when you’re ready for it

• Don’t reinvent the wheel

• Move to NoSQL when it really makes sense but

do your best not to administer it

Putting all this together

means we should now

easily be able to handle

10+ million users!

To infinity…..

User >10mil:

• More fine tuning of your application

• More SOA of features/functionality

• Going from Multi-AZ to multi-region

• Potentially needing to start building custom

solutions

• Deep analysis of your whole stack

Next Steps?

READ!

• aws.amazon.com/documentation

• aws.amazon.com/architecture

• aws.amazon.com/start-ups

Next Steps?

START USING AWS

aws.amazon.com/free

Next Steps?

ASK FOR HELP!

• forums.aws.amazon.com

• aws.amazon.com/support

• Your Account Manager

• A Solutions Architect

THANKS

FOR

LISTENING!

Mohan Vedula – mohanv@amazon.com

Brett Francis – brettf@amazon.com

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scaling on AWS for the

First 10 Million UsersMohan Vedula, Enterprise Solutions Architecture

Brett Francis, Enterprise Solutions Architecture

March 26, 2014

Thank you!