Upload
mongodb
View
901
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
20M crashes/day
2TB data / 50GB index
11 nodes / 3* availability zones
I don't care if MongoDB is slowI care if my app is slow
How to make my app fast again if it's slow because of the way I'm using MongoDB
What does slow mean?
Performance over time
Performance over time
It's slow :(
Solution 1
Denormalize
It's fast :)
It's slow :(
db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)
db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()
{ "cursor" : "BtreeCursor _id_ reverse", "isMultiKey" : false, "n" : 0, "nscannedObjects" : 227756, "nscanned" : 227756, "nscannedObjectsAllPlans" : 227756, "nscannedAllPlans" : 227756, "scanAndOrder" : false, "indexOnly" : false, "nYields" : 1779, "nChunkSkips" : 0, "millis" : 461, "indexBounds" : { "_id" : [ [ { "$maxElement" : 1 }, { "$minElement" : 1 } ] ] }, "server" : "Jaroussky.local:27017", "filterSet" : false}
{"cursor" : "BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] }}
{"cursor" : "BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] }}
Solution 2
Index
db.errors.ensureIndex( {project_id: 1, id: -1})
{"cursor" : "BtreeCursor project_id_1__id_1","nscanned" : 6,"indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]]}}
It's fast :)
It's slow :(
Only 1 query...Indexed properly...
mongostat
insert query update delete getmore command flushes mapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g
vsize res faults locked db idx miss % qr|qw ar|aw 641g 8.65g 7 bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2 bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3 bugsnag:23.4% 0 0|0 0|0
netIn netOut conn set repl time 47k 73k 145 bugsnag1 PRI 06:34:05 63k 99k 146 bugsnag1 PRI 06:34:06 98k 124k 146 bugsnag1 PRI 06:34:07
command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07
Solution 3
Shard
sh.shardCollection("errors"{ project_id: 1, _id: -1})
command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14
It's fast :)
It's slow :(
Only 1 query...Indexed properly...
Lock % ok...
iostat
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-szxvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41
avgqu-sz await r_await w_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40
rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60
db.stats()['indexSize']= 3852071824
= 3932614656free -b (m3.medium)
Solution 4
Scale
db.stats()['indexSize']= 3852071824
= 15775363072free -b (r3.large)
It's fast :)
It's slow :(
Main query seems fine...
"Quick" queries sometimes slow...
mongotop
ns total read write 2014-06-22T19:11:35 bugsnag.events 35ms 0ms 35ms bugsnag.errors 30ms 6ms 24msbugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms
ns tot r w bugsnag.events 35 0 35 bugsnag.errors 30 6 24bugsnag.projects 12 5 7 bugsnag.users 15 7 8
Solution 5
Tag shards
It's fast :)
Solution 1
Denormalize
Solution 2
Index
Solution 3
Shard
Solution 4
Scale
Solution 5
Tag shards
Solution 6
...
NewRelic / Skylight
explain()
mongostat
iostat
...
It will be slow
You can speed it up
Thanks!@ConradIrwin