Ensuring Consistency in a Replicated World

Ensuring Consistency in a Replicated World

Josh  Snyder  2014-­‐09-­‐30

2

what is Yelp?

• we operate in a bunch of markets • aim to be globally distributed • our users should never see stale content • our developers should be able to design an application resilient to

replication delay

3

goals

4

a sample architecture

• a small set of moving parts • enables us to do more with fewer shards • masks geographic traffic split from users and developers • enhanced tolerance to replication delay • ability to

– perform online replication hierarchy changes – batch-load data

5

our toolset

6

cookies

• give the client a short-lived “dirty session” cookie • encode the time of the latest interaction between you and them • expire or ignore the cookie after replicas have caught up

7

cookies

• load balancer: • POST? • GET? -> cookie?

• routes the request into the appropriate datacenter • adds headers to requests

8

request routing

• users get read-after-write consistency • routing a user’s request between datacenters increases latency !

• getting it wrong: increased load on the master database

9

tradeoffs

• we need to be assured that a user’s request falls back to a datacenter that has all of their data

10

tradeoffs

• we need a clear picture of it • never underestimate replication delay, always overestimate

11

replication delay

• made of lies (for this purpose) • underestimates most of the time • overestimates some of the time

12

Seconds_Behind_Master

http://bugs.mysql.com/bug.php?id=66921

13

heartbeats

• insert known data on the master • wait until you see it on the slave • time waited is replication delay

14

heartbeats

15

clocks are evil

16

clocks are evil (2)

17

pt-heartbeat

18

yelp_heartbeat

19

the secret sauce

• A sensu check:

20

what does that get us? (pt 1)

21

why that way?

22

time is hard

http://bugs.mysql.com/bug.php?id=48326

• aggregates heartbeat information • provides it to the webapp • determines when to expire the dirty session cookie

23

repl_delay_reporter

• Wait for replication: • “I inserted some data; when will it be available on all replicas?”

• Throttle to replication: • “I want to bulk insert data. Will doing so cause too much replication delay?”

24

operations

• insert some data • ask the master database “what’s the heartbeat right now?”

• ask the repl_delay_reporter “what’s the lowest heartbeat right now?” • wait a bit

• loop until the lowest heartbeat exceeds the original master heartbeat

25

wait for replication

• determines when to expire the dirty session cookie • relies on only 1 clock, and only for monotonicity • used heavily by batches

– provides read-after-write consistency

26

wait for replication

• prevents batches from causing excessive replication delay • operates before the beginning of each transaction

– batches ask “is replication delay low enough for me to write right now?”

• batches are required to keep their transactions reasonably-sized

27

throttle to replication

• load on masters • laggards • over-throttling

28

gotchas

• batch data can reside on the same shards that serve OLTP requests • support databases with heterogenous SLAs • automatic load-shedding when there is a replication issue

29

what this gets us

• shunting of nearly ALL reading and reporting off of the master • better mileage out of the Percona toolkit • on-line replication hierarchy changes

30

what this gets us