Upload
alvin-john-richards
View
219
Download
0
Embed Size (px)
Citation preview
8/7/2019 MongoAsia - Scaling
1/44
Scaling with MongoDBAlvin Richards
8/7/2019 MongoAsia - Scaling
2/44
Agenda
Vertical Scaling Horizontal Scaling with MongoDB
Schema & Index design Auto Sharding Replication
8/7/2019 MongoAsia - Scaling
3/44
Scaling
Operations/sec go up Storage needs go up
Capacity IOPs
Complexity goes up Caching
8/7/2019 MongoAsia - Scaling
4/44
Optimization & Tuning Schema & Index Design O/S tuning Hardware conguration
Vertical scaling Hardware is expensive Hard to scale in cloud
How do you scale now?
$$$
throughput
8/7/2019 MongoAsia - Scaling
5/44
MongoDB Scaling - Single Node
write
read
node_a1
8/7/2019 MongoAsia - Scaling
6/44
Read scaling - add Replicas
write
read
node_b1
node_a1
8/7/2019 MongoAsia - Scaling
7/44
Read scaling - add Replicas
write
read
node_c1
node_b1
node_a1
8/7/2019 MongoAsia - Scaling
8/44
Write scaling - Sharding
write
read
shard1
node_c1
node_b1
node_a1
8/7/2019 MongoAsia - Scaling
9/44
Write scaling - add Shards
write
read
shard1
node_c1
node_b1
node_a1
shard2
node_c2
node_b2
node_a2
8/7/2019 MongoAsia - Scaling
10/44
Write scaling - add Shards
write
read
shard1
node_c1
node_b1
node_a1
shard2
node_c2
node_b2
node_a2
shard3
node_c3
node_b3
node_a3
8/7/2019 MongoAsia - Scaling
11/44
Scaling with MongoDB
Schema & Index Design Sharding Replication
8/7/2019 MongoAsia - Scaling
12/44
Schema
Data model e f ects performance Embedding versus Linking
Roundtrips to database Disk seek time Size if data to read & write
Partial versus full document writes
Performance problems can be solved by changingschema
not schema less - dynamic schemaschema is just as important, or more important than relationalunderstand write vs read tradeo f s
8/7/2019 MongoAsia - Scaling
13/44
Indexes
Index common queries Do not over index
(A) and (A,B) are equivalent, choose one Right-balanced indexes keep working set small
most common performance problemwhy _id index can be ignored
8/7/2019 MongoAsia - Scaling
14/44
Random Index Access
Entire indexin ram
8/7/2019 MongoAsia - Scaling
15/44
Right-Balanced Index Access
Small portionin ram
8/7/2019 MongoAsia - Scaling
16/44
What is Sharding
Ad-hoc partitioning
Consistent hashing Amazon Dynamo
Range based partitioning Google BigTable Yahoo! PNUTS MongoDB
sharding isnt new
8/7/2019 MongoAsia - Scaling
17/44
MongoDB Sharding
Automatic partitioning and management
Range based
Convert to sharded system with no downtime
Almost no functionality lost over single master
Fully consistent
8/7/2019 MongoAsia - Scaling
18/44
How MongoDB Sharding works
> db.runCommand( { addshard : "shard1" } );
- +
Range keys from - to +
Ranges are stored as chunks
8/7/2019 MongoAsia - Scaling
19/44
How MongoDB Sharding works
> db.posts.save( {age:40} )
- +
- 40 41 +
Data in inserted
Ranges are split into more chunks
8/7/2019 MongoAsia - Scaling
20/44
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )
- +
- 40 41 +
41 50 51 +
More Data in inserted
Ranges are split into morechunks
8/7/2019 MongoAsia - Scaling
21/44
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )
- +
- 40 41 +
41 50 51 +
61 + 51 60
8/7/2019 MongoAsia - Scaling
22/44
- +
41 +
51 +
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )
- 40
41 50
61 + 51 60
8/7/2019 MongoAsia - Scaling
23/44
How MongoDB Sharding works
- 40
41 50
61 +
51 60
shard1
8/7/2019 MongoAsia - Scaling
24/44
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
- 40
41 50
61 +
51 60
8/7/2019 MongoAsia - Scaling
25/44
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
- 40
41 50
61 +
51 60
shard1
8/7/2019 MongoAsia - Scaling
26/44
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
- 40
41 50
61 +
51 60
shard1 shard2
8/7/2019 MongoAsia - Scaling
27/44
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
- 40
41 50
61 +
51 60
shard1 shard2
> db.runCommand( { addshard : "shard3" } );
shard3
8/7/2019 MongoAsia - Scaling
28/44
Sharding Key Examples
Good : {server:1} All data for one server is in a single chunk Chunk cannot be split any smaller
Better : {server:1,time:1} Chunk can be split by millisecond
{server : "ny153.example.com" ,application : "apache" ,time : "2011-01-02T21:21:56.249Z" ,level : "ERROR" ,msg : "something is broken"
}
8/7/2019 MongoAsia - Scaling
29/44
Sharding Key Examples
Good : {time : 1} Time is an increasing number All data will be rst written to a single shard Data balanced to other shards later
Better : {server:1,application:1,time:1}
More key values to enable writes to all shards
{server : "ny153.example.com" ,application : "apache" ,time : "2011-01-02T21:21:56.249Z" ,level : "ERROR" ,msg : "something is broken"
}
8/7/2019 MongoAsia - Scaling
30/44
Sharding Features
Shard data without no downtime Automatic balancing as data is written Commands routed (switched) to correct node
Inserts - must have the Shard Key Updates - must have the Shard Key Queries
With Shard Key - routed to nodes Without Shard Key - scatter gather
Indexed Queries With Shard Key - routed in order
Without Shard Key - distributed sort merge
8/7/2019 MongoAsia - Scaling
31/44
MongoDB Replication
MongoDB replication like MySQL replicationAsynchronous master/slave
Variations:Master / slaveReplica Sets
8/7/2019 MongoAsia - Scaling
32/44
A cluster of N servers Any (one) node can be primary Consensus election of primary Automatic failover Automatic recovery All writes to primary Reads can be to primary (default) or a secondary
Replica Set features
8/7/2019 MongoAsia - Scaling
33/44
How MongoDB Replication works
Member 1
Member 2
Member 3
Set is made up of 2 or more nodes
8/7/2019 MongoAsia - Scaling
34/44
How MongoDB Replication works
Member 1
Member 2PRIMARY
Member 3
Election establishes the PRIMARY
Data replication from PRIMARY to
SECONDARY
8/7/2019 MongoAsia - Scaling
35/44
How MongoDB Replication works
Member 1
Member 2DOWN
Member 3
negotiatenew master
PRIMARY may fail
Automatic election of new PRIMARY
8/7/2019 MongoAsia - Scaling
36/44
How MongoDB Replication works
Member 1
Member 2DOWN
Member 3PRIMARY
New PRIMARY elected
Replication Set re-established
8/7/2019 MongoAsia - Scaling
37/44
How MongoDB Replication works
Member 1
Member 2RECOVERING
Member 3PRIMARY
Automatic recovery
8/7/2019 MongoAsia - Scaling
38/44
How MongoDB Replication works
Member 1
Member 2
Member 3PRIMARY
Replication Set re-established
8/7/2019 MongoAsia - Scaling
39/44
Using Replicas
slaveOk()
- driver will send read requests to Secondaries- driver will always send writes to Primary
Java examples- DB.slaveOk()- Collection.slaveOk()- find(q).addOption(Bytes.QUERYOPTION_SLAVEOK);
8/7/2019 MongoAsia - Scaling
40/44
Creating a Replica Set
> cfg = {... _id : "acme_a",... members : [... { _id : 0, host : "sf1.acme.com" },... { _id : 1, host : "sf2.acme.com" },... { _id : 2, host : "sf3.acme.com" } ] }> use admin> db.runCommand({replSetInitiate:cfg})
8/7/2019 MongoAsia - Scaling
41/44
Replica Set Member Types
Normal {priority:1}
Passive {priority:0} Cannot be elected as PRIMARY
Arbiters Can vote in an election Do not hold any data
8/7/2019 MongoAsia - Scaling
42/44
Replication features
Reads from Primary are always consistent
Reads from Secondaries are eventually consistent
Automatic failover if a Primary fails
Automatic recovery when a node joins the set
8/7/2019 MongoAsia - Scaling
43/44
Summary
Schema & Index design Simplest way to scale
Sharding Automatically scale writes
Replication Automatically scale reads
8/7/2019 MongoAsia - Scaling
44/44
@mongodb
conferences, appearances, and meetupshttp://www.10gen.com/events
http://bit.ly/mongoJFacebook | Twitter | LinkedIn
http://linkd.in/joinmongo
download at mongodb.org