©2015,  Amazon  Web  Services,  Inc.  or  its  affiliates.  All  rights  reserved

Deep Dive: Maximizing EC2 & EBS Performance

Tom Maddox, Solutions Architect

What we’ll cover

•  Amazon EBS overview –  Volumes –  Snapshots

•  Performance –  Understanding I/O –  6 key considerations

•  Q&A

Amazon EBS Overview

For most builders AWS is get in and go!

A “normal” hard drive •  Vendor •  Model Number •  Average Data Rate,

Read / Write (MB/s) •  Spindle Speed •  Interface Options •  Supported SATA

Transfer Rates •  Bytes per Sector

•  Seek Average, Read (ms)

•  Seek Average, Write (ms)

•  Heads / Disks •  Max Sustained Data

Rate (MB/s) •  Cache Size (MB)

EBS !=

What is EBS?

•  It’s a service providing network block storage •  Volumes can be associated with EC2

instances in the same Availability Zone •  Designed for five nines of availability •  Provides point-in-time snapshots to Amazon

S3

A few definitions…

•  IOPS: Input/output operations per second (#)

•  Throughput: Read/write rate to storage (MiB/s)

•  Latency: Delay between request and completion (ms)

•  Capacity: Volume of data that can be stored (GiB)

•  Block size: Size of each I/O (KiB)

EBS volume types

•  General Purpose (SSD) •  Provisioned IOPS (SSD) •  Magnetic

When performance matters, use SSD-backed volumes

EBS General Purpose (SSD) volumes

•  Recommended for: default usage where high performance is occasionally required.

•  Throughput: up to 160 MB/sec

•  IOPS: 3 per GB provisioned up to 10,000. Smaller volumes can burst to 3,000 IOPS for periods relative to their size

•  Latency: 1-2ms

Understanding General Purpose (SSD) bursting

11

Baseline Performance 3 IOPS / GB

Always accumulating 3 I/O credits / GB / second

Max I/O credit 5.4 Million

Can consume up to (3,000 – 3*GB) I/O credits / second

Consumed IOPS

Comparing instance boot times

US East m3.medium boot time comparison Volume type Boot time Access time OS

GP2 3:31 4:33 Windows Server 2012

Magnetic 4:30 7:16 Windows Server 2012

GP2 0:36 0:45 CentOS6

Magnetic 0:57 1:16 CentOS6

Up to 40% reduction in boot times by using General Purpose SSD

Database volume selection

PIOPS

•  1x 1TB volume •  4,000 Provisioned IOPS

$526.40 per month

General Purpose

•  2x 500GB volumes •  3,000 baseline IOPS •  6,000 peak IOPS!

$102.40 per month

80% cost savings and 50% more peak I/O with multiple General Purpose SSD volumes

EBS PIOPS (SSD) volumes

•  Recommended for: I/O intensive databases that require highest consistency

•  Throughput: up to 320 MB/sec

•  IOPS: Provision up to 20,000 IOPS per volume (supports max IOPS:GB ratio of 30:1)

•  Latency: 1-2ms

•  Designed for 99.9% performance consistency

EBS Magnetic volumes

•  Recommended for: cold workloads (rarely accessed data that needs always on access)

•  Throughput: variable by workload (best effort to 10s of MBs)

•  IOPS: ~100 IOPS steady-state, with best-effort bursts

•  Latency: Varies

•  Reads typically ~20-40 ms

•  Writes typically ~2-10 ms

EBS volume types summary General Purpose (SSD) Provisioned IOPS (SSD) Magnetic

Recommend use cases Boot volumes

Small to med DBs Dev and test

I/O-intensive Large DBs Cold storage

Storage media SSD-backed SSD-backed Magnetic-backed

Volume size 1 GB - 16 TB 4 GB - 16 TB 1 GB - 1 TB

IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS

Burst < 1 TB to 3000 IOPS n/a n/a

Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MB/s

16k IOPS per EC2 node 48,000 48,000 48,000

Throughput per node 800 MB/s 800 MB/s 800 MB/s

Latency (random read) 1-2 ms 1-2 ms 20-40 ms

API Name gp2 io1 standard

Price* $.10/GB-month $.125/GB-month $.065/provisioned IOPS

$.05/GB-month $.05/ 1M I/O

Benefits of using EBS snapshots

•  More durable than an EBS volume –  Stored in Amazon S3

•  Differential (space-efficient) –  First snapshot includes all written blocks –  Subsequent snapshots include changed blocks –  Pay only for what you use

•  Availability Zone independent –  Clone to volumes in any neighboring AZ

•  Can be copied efficiently across regions

EBS Performance

Queuing theory – Little’s law

Little’s law is the foundation for performance tuning theory. (Mathematically proven by John Little in 1961)

L = A * W Where

L = Queue length (the average number of requests waiting) A = Request arrival rate W = Average amount of time requests are queued

EBS performance is reflective of this law

Performance optimization is measured by:

•  IOPS: Read/write I/O rate (IOPS)

•  Latency: Time between I/O submission and completion (ms)

•  Throughput: Read/write transfer rate (MB/s); throughput = IOPS X

I/O size

Tools available for performance tuning:

1.  EC2 instance: Network bandwidth (Mbps)

2.  EBS-optimized instance: EC2 instance option (On/Off)

3.  Workload: Block size, read/write ratio, serialization

4.  Queue depth: The number of outstanding I/Os

5.  RAID: Stripe volumes to maximize performance

6.  Pre-warming: Eliminate first-touch penalty

The life of EBS I/O operations

1. EC2 instance

2. I/O

4. EBS

3.Network link

EC2 instance type selection

Select the EC2 instance that has the right network, RAM, and CPU resources for your applications

C3 instance family overview Instance Type vCPU Memory (GiB) Networking

Performance EBS

Optimized c3.large 2 3.75 Moderate - c3.xlarge 4 7.5 Moderate Yes

c3.2xlarge 8 15 High Yes c3.4xlarge 16 30 High Yes c3.8xlarge 32 60 10 Gigabit -

EBS-optimized

•  Most instance families supports the EBS-optimized flag

•  EBS-optimized instances now support up to 4 Gb/s

–  Drive 32,000 16K IOPS or 500 MB/s

•  EC2 *.8xlarge instances support 10 Gb/s network

•  Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

Use EBS-optimized instances for consistent EBS performance

Workload •  I/O size:

–  4 KB to 64 MB

•  I/O pattern: –  Sequential and random

•  I/O type: –  Read and write

•  I/O concurrency: –  Number of concurrent I/O

EBS SSD-backed volumes measure I/O size up to 256 KB

EBS SSD-backed volumes deliver same performance for read and write

EBS IOPS and throughput limits

20,000 IOPS PIOPS volume

20,000 IOPS

320 MB/s throughput

You can achieve 20,000 IOPS when driving smaller I/O

operations

You can achieve up to 320 MB/s when driving larger I/O

operations

EBS IOPS and throughput limits

8,000 IOPS PIOPS volume

8,000 IOPS

320 MB/s throughput

16,000 X 8 KB = 128 MB/s 8,000 X 8 KB = 64 MB/s 8,000 X 16 KB = 128 MB/s 8,000 X 32 KB = 256 MB/s 8,000 X 64 KB = 512 MB/s

1,250 x 256 KB = 320 MB/s

Block (I/O) size determines whether your application is IOPS bound or

throughput bound

Queue depth

An I/O operation

EBS

After it’s gone, it’s gone

EC2

Queue depth is the number of pending I/O operations for a volume

I/O latency •  Elapsed time between I/O submission and its completion

time

•  Performance requirements may be driven by IOPS or latency or both

•  There is an interdependency among IOPS, queue depth, and latency

Random read latency

0.075

35.1

0

5

10

15

20

25

30

35

40

1 4 8 12 16 20 24 28 32

Late

ncy

TP90

(ms)

Queue depth

Random read latency across various queue depths

Latency (TP90)

Read latency linearly increases with increase in queue depth

Random read latency

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

Late

ncy

TP9

0 (m

s)

Queue depth

Random read 16 KB IOPS, latency across various queue depths

Latency (TP90) Avg Read IOPS

IOPS

Queue depth of 1 has the lowest latency, but also has the lowest IOPS

Random read latency

0.075

35.1

2.09

1,865

4,152 3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

Late

ncy

TP9

0 (m

s)

Queue depth

Random read 16 KB IOPS, latency across various queue depths

Latency (TP90) Avg Read IOPS

IOPS

Key EBS Volume Metrics

•  Average queue depth •  Throughput percentage •  Average read/write size •  Consumed read/write IOPS •  Average read/write latency

Latency: baseline

Log scale

Median

p75

p25

Latency: General Purpose SSD volume

Latency: Latest generation instance

Optimal queue depth to achieve lower latency and highest IOPS is typically between 4-8; ~1 queue depth per 500 IOPS

EBS-optimized instances provide consistent latency experience

Use SSD volumes with latest-generation EC2 instances

RAID (or volume striping)

Do

Use to extend beyond 16TB volume capacity Use to surpass 320MB/s or 20k IOPS single volume performance limits

Don’t

Mix volume types

Configure RAID for redundancy (e.g. RAID-1)

Maximum performance per instance

How should you think about taking snapshots on a striped volume?

–  Quiesce file systems and take snapshot

–  Unmount file system and take snapshot

–  Use OS-specific tools

12×400 GB PIOPS, pre-warmed, RAID 0 LVM, stripe size 128 KB, attached to CR1 instance I/O Pattern Block

Sizes Thread Write

IOPS Write BW (MB)

Read IOPS

Read BW (MB)

Sequential 4K 8 33,500 134 48,250 193

16K 8 13,875 222 48,063 769 1M 1 247 247 823 823

Pre-warming •  EBS volumes may incur first-access penalty (5-50%

performance reduction in IOPS and latency) whilst provisioning continues in background.

•  Pre-warming is the name given to intentionally accessing each block; ensuring a consistent performance level is reached.

•  Pre-warm new volumes before benchmarking by using: –  Linux: DD write –  Windows: NTFS full format

•  Always check latest documentation http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

Recap: Tools available for performance tuning

1.  EC2 instance: Network bandwidth (Mbps)

2.  EBS-optimized instance: EC2 instance option (On/Off)

3.  Workload: Block size, read/write ratio, serialization

4.  Queue depth: The number of outstanding I/Os

5.  RAID: Stripe volumes to maximize performance

6.  Pre-warming: Eliminate first-touch penalty

LONDON