41
S Scale In after Scaling Out WHY, WHEN AND HOW

Shard-In after Sharding Out with SSD

Embed Size (px)

DESCRIPTION

The reason why we shard and Tips on how to combine shards from both Code perspective and Operational perspective

Citation preview

Page 1: Shard-In after Sharding Out with SSD

S

Scale In after Scaling Out

WHY, WHEN AND HOW

Page 2: Shard-In after Sharding Out with SSD

Who am I

Been using mySQL since 1999

Worked for FriendFinder, Friendster, Flickr, Rockyou, SchoolFeed, Weebly

Presented on Flickr Architecture: Doing Billions of Queries Per Day. Record Every Referral For Flickr Real-time. Scaling to 200K TPS per second with Open Source. Scaling a Widget Company, MySpace Vrs Facebook API Load Patterns. University Of Utah Presentation and various others.

Page 3: Shard-In after Sharding Out with SSD

Patterns from Start to Scale

Start a project with a single mySQL DB.

Get some users add more disks to the mySQL DB

Get some more users add a slave

Add more Slaves

Then need to split up the master

Page 4: Shard-In after Sharding Out with SSD
Page 5: Shard-In after Sharding Out with SSD

Patterns: Continued

Master is not strong enough

Put tables on other servers with slaves

Constantly battle slave lag

Page 6: Shard-In after Sharding Out with SSD

Big TableAll Tables Minus the Big Table

Page 7: Shard-In after Sharding Out with SSD

Still not scaling Horizontally

Let’s Shard

Page 8: Shard-In after Sharding Out with SSD

Federation

User 1’s DataUser 2’s DataUser 3’s Data

….User N’s Data

User 1’s Data

User 2’s Data

User 3’s Data

User N’s Data

Page 9: Shard-In after Sharding Out with SSD

Assign a section of the Database as a whole to a Server

Have a Layer to tell the connector what to connect to

Add slaves for redundancy

Go Master-Master

Do this N times

Finally provided stable service

Page 10: Shard-In after Sharding Out with SSD

Federation

This Increases

WriteThroughput

Page 11: Shard-In after Sharding Out with SSD

Now you have the ability to scale Horizontally

What problem was solved at its lowest levels? Lack of IOPS solved Handling of concurrency solved

Page 12: Shard-In after Sharding Out with SSD

Problems introduced

Lots of power used

Lots of servers to manage

Lots of rack space used

Some less then optimal hardware usage

Page 13: Shard-In after Sharding Out with SSD

SSD

SSD use NAND flash chips, each chip holds millions of Cells.

SLC can hold a single data bit, MLC can hold multiple data bits yielding a higher density or more disk space.

Typically MCL provides Slower throughput than SLC due to more complicated error correction algorithms and false positive reads.

Page 14: Shard-In after Sharding Out with SSD
Page 15: Shard-In after Sharding Out with SSD

MLC is not all bad major leaps in Firmware improved it

More enterprises are using MLC

Its cheaper

Fast Enough

Endurance improved

Write amplification improved (erasure and data wad resends)

Stay on top of Firmware changes

TRIM improvements which solves the progressively slower writes to blocks over and over.

Page 16: Shard-In after Sharding Out with SSD

We use Intel SSDSA2CW160 320 Series MLC SSD

It’s FAST

It’s Reliable

It has advance power protection features Really big capacitors to flush buffered data

Low power usage

We consider it the best

It’s no longer made

Everyone wants it

Page 17: Shard-In after Sharding Out with SSD

Speed of single SSD verses Single Spinning Metal Drive

20K IOPS writes reported - SSD

35K IOPS reads reported – SSD

200 IOPS for SEAGATE ST9146852SS HT043TB0584C

Page 18: Shard-In after Sharding Out with SSD

Now that we have more IOPS need Space

RAID-5 gives the best space performance we can use.

8 160GB SSD gives 1TB 613GB of usable space Raw size is 149 GB per disk Reserved for wear

Page 19: Shard-In after Sharding Out with SSD

Now that we have space we can combine Shards

First get the IOP usage of the current shards is 12K IOPS

Next get Disk space requirements Do not use more then 50% so you have room for

growth I use now 56% of space

Depending on the Replication Traffic per shard you may need another plan

Page 20: Shard-In after Sharding Out with SSD

S

How to combine Data

Page 21: Shard-In after Sharding Out with SSD

Code Steps

Have a program that keeps a hashmap of tablename to federated column

Lock the federated entity by throwing an error in the application that says this federated entity is not available

SELECT ALL Federated data (in chunks) and add it to the new combined table.

Update pointers

Error if any step fails and keep the data locked otherwise Unlock

Page 22: Shard-In after Sharding Out with SSD

Another way to combine, more Operational

Take a copy of the shard.

Configure multi instance mysql

Run that shard off a different port

Page 23: Shard-In after Sharding Out with SSD

I choose to do both methods here is how

Let’s take a case.

Support 20 million websites

90% of all sites get 1 or more hits but less then 1000 hits per day

Less then 10% of sites gets more then 1000 hits per day

8 shards to handle 12K IOPS 64 CPU Threads 288 GB of memory 64 2.5” drives Roughly $40K Of hardware Multiply * 2 for redundancy

Page 24: Shard-In after Sharding Out with SSD

Replication was lagging

Simply combining the data onto one server will not work

Master needs 10K IOPS replication with some tricks can use 2.5K IOPS

Innodb_fake_changes did not work

Facebook:faker helped but CPU was underpowered thus not really good to saturate IOPS and keep it in sync

Page 25: Shard-In after Sharding Out with SSD

Multi_Mysql is the answer

Set up 4 mySQL instances

Instead of 1 Replication thread I now have 4

Instead of limiting 2.5K IOPS ON SSD from single Replication thread I now have 10K IOPS

Master produces 10K IOPS from ETL that runs on all Front End

I don’t need much memory in fact each instance only has a 4GB buffer pool

Page 26: Shard-In after Sharding Out with SSD

S

Let’s look at the details

Page 27: Shard-In after Sharding Out with SSD

All tables are compressed

KEY_BLOCK_SIZE=8K with INNODB

Page 28: Shard-In after Sharding Out with SSD

Consistent Hash for Hostname to bigint

Remove Lookup in exchange for a small CPU computation

Md5 based off 1st 16 bits of hex number to produce 8byte bigint

Primary KEY is HashId + Hostname(10)

HashId maps to ShardId with range blocks

Page 29: Shard-In after Sharding Out with SSD
Page 30: Shard-In after Sharding Out with SSD

S

Run through all hostnames to assign

to a shard

Page 31: Shard-In after Sharding Out with SSD

Test Hashing is even

X = #

Shards

Y = PV/Site

Page 32: Shard-In after Sharding Out with SSD
Page 33: Shard-In after Sharding Out with SSD

Shard on a Single Server

There are 8 Databases per Server Instance

A database represents a shard

There are 4 mySQL Server Instances

32 Shards Total

Can isolate a single DB to a single Server

Can isolate a single Host to a single Shard

Page 34: Shard-In after Sharding Out with SSD

Base on a Range go to the correct server, port and database

Page 35: Shard-In after Sharding Out with SSD

How to switch

Write in both locations

Log if write fails in a single location (none happened)

Backfill old data to new format

Switch reads over to the new format once data is verified as correct

Page 36: Shard-In after Sharding Out with SSD

In Staging switch Reads to new format

Verify that Production and Staging render the same graph

Verify that Production and Staging have the same referrers

Sample random Pro accounts and make sure numbers match

Page 37: Shard-In after Sharding Out with SSD

Roll out with a switch to rollback

There was a bug where some website where passed as user input to the lookup method, yet I stored everything lowercase names

Turn off new reads with Application config switch

Fix issue and turn on new reads

Page 38: Shard-In after Sharding Out with SSD

Clean Up

Once fully over on new format

Kill old format

Repurpose servers

Profit

Page 39: Shard-In after Sharding Out with SSD

Some Stats

$80K potential in server cost reduced to $7K

Utilize all the CPU

Less memory per server but more IOPS

All replicas stay in sync because there is now more then 1 replication thread per physical server. (There are 4)

Page 40: Shard-In after Sharding Out with SSD

Next Generation

Fusion I/O PCIe SSD Card

1U Form factor

Less power 40W-50W

No need to RAID

Page 41: Shard-In after Sharding Out with SSD

Questions

Twitter @dathanvp

http://mysqldba.blogspot.com

http://facebook.com/dathan

http://linkedIn.com/in/dathan

http://about.me/dathan

mailto:[email protected]