Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB

8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB

http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 1/80

SHORTCUTS AROUND THEMISTAKES I’VEMADE SCALINGMONGODB

Theo, Chief

Architect

at

onsdag 21 september 11



What we doWe want to revolutionize the digital advertising

industry by showing that there is more to adanalytics than click through rates.




Ads




Data




Assembling sessionsexposure

ping

ping

ping ping

ping

event

event

ping

session➔ ➔




Crunching

session

session

session

session

sessionsession

sessi on sessi on

session

session

session

session

session

➔ ➔ 42




Reports




What we do Track ads, make pretty reports.




That doesn’t sound so hard




That doesn’t sound so hardWe don’t know when sessions end





There’s a lot of data





There’s a lot of data

It’s all done in (close to) real time




Numbers




Numbers40 Gb data




Numbers40 Gb data

50 million documents




Numbers40 Gb data

50 million documents

per day




How we use MongoDB




How we use MongoDB“Virtual memory” to offload data while we wait

for sessions to finish





for sessions to finishShort time storage (<48 hours) for batch jobs





for sessions to finishShort time storage (<48 hours) for batch jobs

Metrics storage




Why we use MongoDB




Why we use MongoDBSchemalessness makes things so much easier,

the data we collect changes as we come upwith new ideas






Sharding makes it possible to scale writes







Secondary indexes and rich query language aregreat features (for the metrics store)







Secondary indexes and rich query language aregreat features (for the metrics store)

It’s just… nice




Btw.




Btw.We use JRuby, it’s awesome




A story in 7 iterations




secondary indexes and updates1st iteration




secondary indexes and updates1st iterationOne document per session, update as new

data comes alongOutcome: 1000% write lock




#1Everything is aboutworking around theGLOBALWRITE

LOCK




MongoDB 2.0.0

db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)

db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)




MongoDB 1.8.1

db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)

db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)




using scans for two step assembling2nd iterationInstead of updating, save each fragment, then

scan over _id to assemble sessions




using scans for two step assembling2nd iterationOutcome: not as much lock, but still not great

performance. We also realised we couldn’tremove data fast enough





LOCK




#3Give a lot of

thought to yourPRIMARY KEY




partitioning

3rd iteration




partitioning

3rd iterationWe came up with the idea of partitioning the

data by writing to a new collection every hour




partitioning

3rd iterationWe came up with the idea of partitioning the

data by writing to a new collection every hourOutcome: lots of complicated code, lots of

bugs, but we didn’t have to care about

removing data




#4Make sure you can

REMOVE

OLD DATA




sharding

4th iteration




sharding

4th iteration To get around the global write lock and get

higher write performance we moved to a

sharded cluster.




sharding

4th iteration To get around the global write lock and get

higher write performance we moved to a

sharded cluster.

Outcome: higher write performance, lots of

problems, lots of ops time spent debugging





LOCK




#6

SHARDINGIS NOT A

SILVER BULLETand it’s buggy,

if you can, avoid it







#7

IT WILL FAILdesign for it










moving things to separate clusters5th iteration




moving things to separate clusters5th iterationWe saw very different loads on the shards and

realised we had databases with very different

usage patterns, some that made autosharding

not work. We moved these off the cluster.




moving things to separate clusters5th iterationWe saw very different loads on the shards and

realised we had databases with very different

usage patterns, some that made autosharding

not work. We moved these off the cluster.

Outcome: a more balanced and stable cluster




#8Everything is aboutworking around theGLOBALWRITELOCK




#9

ONE DATABASEwith one usage patternPER CLUSTER




#10

MONITOR

EVERYTHINGlook at your healthgraphs daily




monster machines6th iteration




monster machines6th iterationWe got new problems removing data and

needed some room to breathe and think






Solution: upgraded the servers to High-

Memory Quadruple Extra Large (with cheese).






Solution: upgraded the servers to High-

Memory Quadruple Extra Large (with cheese).

♥I




#11Don’t try to scale upSCALE OUT




#12When you’re out of ideas

CALL THE EXPERTS




partitioning (again) and pre-chunking7th iteration




partitioning (again) and pre-chunking7th iterationWe rewrote the database layer to write to a

new database each day, and we created all

chunks in advance. We also decreased the size

of our documents by a lot.




partitioning (again) and pre-chunking7th iterationWe rewrote the database layer to write to a

new database each day, and we created all

chunks in advance. We also decreased the size

of our documents by a lot.

Outcome: no more problems removing data.




#13Smaller objects means a smaller database, and a smaller database meansLESS RAM NEEDED




#14Give a lot of

thought to yourPRIMARY KEY












KTHXBAI

@iconara

architecturalatrocities.comburtcorp.com




Since we got time…




Safe modeTips




Safe modeTips

Run every Nth insert in safe mode




Safe modeTips

Run every Nth insert in safe mode

This will give you warnings when bad thingshappen; like failovers




Avoid bulk insertsTips




Avoid bulk insertsTips

Very dangerous if there’s a possibility of

duplicate key errors




EC2

Tips




EC2

Tips

You have three copies of your data, do you

really need EBS?




EC2

Tips


really need EBS?

Instance store disks are included in the price

and they have predictable performance.




EC2

Tips


really need EBS?

Instance store disks are included in the price

and they have predictable performance.

m1.xlarge comes with 1.7 TB of storage.

Documents

Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB