The Secret Guide to Cloud Performance - Cloudlook

The Secret Guide to Cloud Performance Jan 30 2014 @ DevCon TLV

Gideon Greenspan Co-Founder, Cloudlook

Types of hosting

Secret Guide to Cloud Performance Slide 2 Cloudlook

Hardware

OS + web server + DBMS

Website + Database

Dedicated

Hardware

OS + web server + DBMS

Site

/DB

Shared

Site

/DB

Site

/DB

Host OS Hardware

OS

etc.

Cloud / VPS

Site

/DB

Site

/DB

Site

/DB

OS

etc.

OS

etc.

Hypervisor

“The Cloud” vs Old-School VPS

•  Deploy/decommission in minutes

•  Scale up/down in minutes

•  Management APIs

•  Automatic scaling

•  Billing by the hour

•  Migration on hardware failure


Some public cloud providers


Cloud specifications — AWS EC2


Cloud specs — Rackspace


The trouble with the cloud

•  Vague system specifications

•  Variation between instances – Noisy neighbors?

– Diverse underlying hardware

•  Variation over time within an instance

•  One-time benchmarks insufficient


Cloudlook: continuous benchmarking

•  6 cloud providers, 35 instance types

•  15,000 instances per month

•  50 samples per instance over 1 hour

•  All 64-bit CentOS (even on Azure!)

•  $2,000 per month

•  Available free for your server too!


What does Cloudlook measure?

•  Integer CPU performance

•  Concurrency

•  Memory bandwidth

•  Sequential disk access

•  Random disk access

•  Host disk cache effect


All done with…

The Secret Guide

•  Introduction •  Benchmarks •  Conclusions


Measuring integer CPU

•  PHP scripts run as interpreted bytecode

•  Built-in PHP functions are compiled C

•  We benchmarked 14 heavy functions across 66 versions of PHP ≥ 5.0.4

•  Two performed consistently (±0.2%) – sha1() — hash function

– levenshtein() — deep string comparison


CPU results

Core i7 equivalent

Between instances

Within instances

1 GoGrid (all) 2.05 GHz ± 6% ± 4%

2 HP Cloud (all) 2.01 GHz ± 12% ± 10%

3 Rackspace Performance 1 (all) 2.01 GHz ± 4% ± 4%

4 Rackspace Standard (all) 1.93 GHz ± 6% ± 5%

5 Windows Azure (all) 1.74 GHz ± 3% ± 11%

6 Digital Ocean ≥1GB 1.62 GHz ± 19% ± 12%

7 Amazon m1.medium 1.48 GHz ± 6% ± 5%


CPU variation per instance


Digital Ocean 2GB (sometimes)

Rackspace Cloud Performance 1 2GB

Measuring concurrency

•  Parallel CPU-heavy HTTP requests

•  Precise client-side synchronization – microtime() + usleep()

•  Avoid CPU temperature bias


1 2 3 4

Parallel HTTP

requests

Concurrency results


1

2

3

4

5

1 2 3 4 5 6 7 8

Mea

sure

d co

ncur

renc

y

Specified vCPUs / cores

Windows Azure Digital Ocean + HP Cloud Rackspace Performance 1 GoGrid Amazon EC2 Rackspace Standard

Measuring memory bandwidth

•  Create a 64 MB string in memory – Largest L1+L2+L3 cache currently ~41 GB

•  substr() to read successive windows – memcpy() in C does the actual work

•  Window length = 128 kb

– Too short ⇒ time in bytecode interpreter

– Too long ⇒ copy too big for L1+L2 cache


Memory results


Memory bandwidth

Between instances

Within instances

1 GoGrid 6.7 GB/s ± 10% ± 7%

2 Rackspace Performance 1 6.7 GB/s ± 7% ± 4%

3 Digital Ocean ≥1GB 6.3 GB/s ± 10% ± 8%

4 HP Cloud 5.9 GB/s ± 9% ± 12%

5 Amazon m1.medium 5.2 GB/s ± 25% ± 4%

6 Rackspace Standard 4.1 GB/s ± 16% ± 7%

7 Windows Azure 3.6 GB/s ± 5% ± 6%

Memory performance variation


HP Cloud Standard Medium

Digital Ocean 4GB Amazon m1.large

Rackspace Cloud Standard 4GB

Measuring disk performance

•  Write benchmark files in /tmp

•  Avoid operating system cache – Total size > instance memory

– Explicitly clear cache (our instances)

– Read each block once (our instances)

•  “Random” is pseudo-random

– Fibonacci hashing scheme


Host cache interference


Host OS Hardware

OS

etc.

Si

te/D

B

Site

/DB

Site

/DB

OS

etc.

OS

etc.

Hypervisor

•  Results can be skewed

•  Big instances: write lots of other data inbetween

•  Small instances: wait a week and pray…

•  But still worth measuring because it’s a good thing!

Disk results (no host cache)


10

100

1,000

10,000

10 100 1000

Dis

k Se

eks

— o

ps/s

Disk Throughput — MB/s

EC2 m1.medium Primary EC2 m1.medium Secondary Rackspace Standard >= 2GB Rackspace Performance 1 >= 2GB Both Azure >= 1.5 GB Primary Azure >= 1.5 GB Secondary HP Cloud >= 2GB Both Digital Ocean >= 1GB GoGrid >= 2 GB

Who has a host cache?


Primary disk rate (no host cache)

Primary disk rate (host cache)

Host cache speedup

HP Cloud Small 109 MB/s 528 MB/s 4.8 ⤬

HP Cloud Medium 124 MB/s 320 MB/s 2.6 ⤬

HP Cloud Large 110 MB/s 117 MB/s 1.1 ⤬

Digital Ocean 1 GB 244 MB/s 709 MB/s 2.9 ⤬




The Secret Guide

•  Introduction •  Benchmarks •  Conclusions


Same instance, different stats


EC2 m1.medium GoGrid 4 GB Rackspace

Performance 1 4 GB

Integer CPU 1.4 : 1 1.2 : 1 1.5 : 1

Concurrency 1.1 : 1 1.5 : 1 1.5 : 1

Memory 3.8 : 1 1.6 : 1 1.7 : 1

Disk throughput 3.0 : 1 10.0 : 1 3.0 : 1

Disk seeks 15.8 : 1 4.2 : 1 3.3 : 1

Dedicated vs Cloud


Hetzner EX-40 GoGrid 2 GB GoGrid 8 GB

Integer CPU 3.49 GHz 2.04 GHz 2.09 GHz

Concurrency 6.82 ⤬ 1.83 ⤬ 4.61 ⤬

Memory 12.9 GB/s 6.6 GB/s 6.9 GB/s

Disk throughput 155 MB/s 387 MB/s 381 MB/s

Disk seeks 134 ops/s 127 ops/s 112 ops/s

Memory size 32 GB 2 GB 8 GB

Disk size 1000 GB 100 GB 400 GB

Price $67/mo $66/mo $263/mo

Bad reasons to host in the cloud

•  Hardware abstraction (peace of mind) – Last week Rackspace Cloud lost my data

•  Easy resizing – Dedicated is at top of scale anyway

– Can’t resize Rackspace Performance servers

•  Lower costs

– No. That’s why hosting companies like it…


Some good reasons

•  Small instances

•  Fast deploy/decommission

•  Tools and APIs

•  Other cloud services

– Storage and CDN

– Databases and messaging


So: Beware of fashions…


Imag

e fr

om h

ttp:

//w

ww

.kap

oww

owob

ject

s.co

m

… and measure your own



www.cloudlook.com

Thank you!

Technology

The Secret Guide to Cloud Performance - Cloudlook