How to Handle 1,000,000 Daily Users Without Using a Cache (RailsWayCon 2012)

HOW TO HANDLE

1,000,000 DAILY USERS(without using a cache)

Jesper Richter-‐Reichhelm, @jrireiTuesday, June 5, 2012

Tuesday, June 5, 2012

The overall architecture is not that complex

BackendFlash client



Game Session

Asynch. Communica2on

Flash client


Backend


State Changes

ValidaMon

Persistence


But the scale is interesMng

14 billion requests / month







>100,000 DB operaMons / second




>100,000 DB operaMons / second

>50,000 DB updates / second


A journey to 1,000,000 daily users

Start of the journey

6 weeks of pain

Paradise

Conclusion


October 2009: wooga’s first simulaMon game


Instead of PHP we used Ruby


Our database was MySQL


Our database was MySQL

even user ids odd user ids


And we went into the cloud


Master-‐slave replicaMon for DBs worked fine

app app app

lb

db db


We added a few applicaMon servers over Mme

app app app app app app app app app

lb

db db


250K daily users and no problems

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life was good


Life was well and I went on a nice vacaMon

<picture: Jesper in clot canyon>

TO DO



Our bane: MySQL hiccups

!"#

$!"#

%!"#

&!"#

'!"#

(!!"#

!# )# (!# ()# $!# $)# *!# *)# %!#



!"#

$!"#

%!"#

&!"#

'!"#

(!!"#

!# )# (!# ()# $!# $)# *!# *)# %!#



!"#

$!"#

%!"#

&!"#

'!"#

(!!"#

!# )# (!# ()# $!# $)# *!# *)# %!#




6 weeks of pain

Paradise

Conclusion


SQL queries generated by Rubyamf gem

AMF responses to Flash client




Wrong config...

... so associated data was included, too




Wrong config...

... so associated data was included, too

=> Easy to fix


More traffic using the same cluster


lb

db db


Config tweaks brought us to 300K DAU

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Config fixes


AcMveRecord’s checks caused 20% extra DB

Checking connecMon state

MySQL process list full of ‘status’ calls


AcMveRecord’s checks caused 20% extra DB

Checking connecMon state

MySQL process list full of ‘status’ calls

=> Fixed by 1 line of code


I/O on MySQL masters sMll was the bojleneck

New Relic: 60% of all UPDATEs on ‘Mles’ table


Tiles are part of the core game loop

Core game loop1) plant2) wait3) harvest


We started to shard on model, too

Adding new shards

old master

old slave



Adding new shards1) Setup new masters as slaves of old ones

old master

old slave

new master



Adding new shards1) Setup new masters

old master

old slave

new master

new slave



Adding new shards1) Setup new masters2) Start using new masters

old master

old slave

new master

new slave



Adding new shards1) Setup new masters2) Start using new masters3) Cut replicaBon

old master

old slave

new master

new slave



Adding new shards1) Setup new masters2) Start using new masters3) Cut replicaBon4) Truncate

old master

old slave

new master

new slave


4 DB masters and a few more servers

app app

app app app app app app app app

app appapp

lb

Blesdb

Blesdb

db db

app app app


Sharding by model brought us to 400K DAU

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Shard by model


We improved our MySQL setup

RAID-‐0 of EBS volumes




Using XtraDB




Using XtraDB

Tweaking my.cnf


Sharding gem circumvented AR’s internal cache

AcMveRecord caches SQL queries...




... only in our development environment!




... only in our development environment!

=> Fixed by 2 lines of code


I/O sMll was not fast enough

If 2 + 2 is not enough, ...


I/O sMll was not fast enough

If 2 + 2 is not enough, ...

… perhaps 4 + 4 masters will do?


It’s no fun to handle 8+8 MySQL DBs

app app app app appapp app


appapp

lb

Blesdb

Blesdb

db db


It’s no fun to handle 8+8 MySQL DBs

app app app app appapp app


appapp

lb

Blesdb

Blesdb

Blesdb

Blesdb

db db db db


At 500K DAU we were at a dead end

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


At 500K DAU we were at a dead end

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


I/O remained bojleneck for MySQL UPDATEs

Each DB master could do

about 1000 DB write/s.


I/O remained bojleneck for MySQL UPDATEs

Each DB master could do

about 1000 DB write/s.

That’s not enough!


Pick the right tool for the job!


Pick the right tool for the job!


Redis is fast but goes beyond simple key/value

Redis is a key-‐value storeHashes, Sets, Sorted Sets, ListsAtomic operaBons like set, get, increment


Redis is fast but goes beyond simple key/value

Redis is a key-‐value storeHashes, Sets, Sorted Sets, ListsAtomic operaBons like set, get, increment

50,000 transacMons/s on EC2Writes are as fast as reads


We could learn from another team using Redis


We could learn from another team using Redis


Shelf Mles : An ideal candidate for Redis

Shelf 2les:{ plant1 => 184,plant2 => 141,plant3 => 130,plant4 => 112,

… }


Shelf Mles : An ideal candidate for using Redis

Redis HashHGETALLHGETSETHINCRBY…


On-‐demand migraMons from MySQL to Redis
















Typical migraMon throughput over 3 days


Migrate on the fly -‐ and clean up later

1. Let migraMon run unMl everything cools down




2.Migrate the rest manually





3. Remove migraMon code






4.Wait unMl no fallback necessary






4.Wait unMl no fallback necessary

5. Remove SQL table




6 weeks of pain

Paredise (or not?)

Conclusion


Again: Tiles are part of the core game loop

Core game loop1) plant2) wait3) harvest


Size majers for migraMons



MigraMon check overload



MigraMon check overloadMigraBon only on startup




Overlooked an edge case




Overlooked an edge caseOnly migrate 1% of users




Overlooked an edge caseOnly migrate 1% of usersConBnue if everything is ok


In-‐memory DBs don’t like dumping to disk

Dumping to disk



Dumping to diskSAVE is blocking



Dumping to diskSAVE is blockingBGSAVE needs free RAM




Latency increase by 100%




Latency increase by 100%

=> BGSAVE on slaves every 15 minutes


Redis replicaMon starts with a BGSAVE

StarMng up a new slave by replicaMon



StarMng up a new slave by replicaMonBGSAVE on master



StarMng up a new slave by replicaMonBGSAVE on masterSlave imports dumped file



StarMng up a new slave by replicaMonBGSAVE on masterSlave imports dumped file

=> No RAM means no new slaves


Redis had a memory fragmenMon problem

24 GB

44 GB

in 8 days



24 GB

38 GB

in 3 days



24 GB

38 GB

in 3 days

Fixed in

v2.2


If MySQL is a truck

Fast enough

Disk based

Robust

Fast enough disk based robust


If MySQL is a truck, Redis is a race car

Super fast

RAM based

Fragile

Super fast RAM based fragile


Big and staMc data in MySQL, rest goes to Redis

60 GB data

50% writes

256 GB data

10% writesh"p://www.flickr.com/photos/erix/245657047/


http://www.flickr.com/photos/erix/245657047/

http://www.flickr.com/photos/erix/245657047/

Lots of boxes, but automaMon helps a lot!

app app app app app app app app app app app appapp



lb lb

redis redis redis redis redisdb db db db db


We reached 1 million daily users!

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

1,000,000 -‐ Big party!


We started archiving inacMve users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

50% DBreduc2on


We even survived a complete data center loss

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

EBS nomore!


We improved our MySQL schema on-‐the-‐fly

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

30% DBreduc2on


Meanwhile we have more than 2M daily users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"




6 weeks of pain

Paredise (or not?)

Conclusion


of sotware

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

EvoluMon every week

EVOLUTION


of sotware

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

EvoluMon every week

EVOLUTION


of sotware

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

EvoluMon every week

EVOLUTIONREVOLUTION


of sotware

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

EvoluMon every week

EVOLUTIONREVOLUTION


!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

of sotware

EvoluMon every week

EVOLUTIONREVOLUTION


Works for teams ...


Each new game is a revoluMon










Works for teams and for companies

!""#$%&"'()"*+,


Ques2ons?

Jesper Richter-‐Reichhelm@jrirei

slideshare.net/woogawooga.com/jobs



Technology

How to Handle 1,000,000 Daily Users Without Using a Cache (RailsWayCon 2012)