Approaching 1 Billion Documents in MongoDB

David Myttondavid@boxedice.com / www.mytton.net1/25

Server Density Monitoring

Processing Database UI

www.serverdensity.com2/25

db.stats()

Documents 981,289,332

Collections 47,962

Indexes 39,684

Data size 369GB

Index size 241GB

As of 25th Apr 2010.3/25

10 months

Why we moved: http://bit.ly/mysqltomongo4/25

Initial Setup

MasterDC1

8GB RAM

SlaveDC2

8GB RAM

Replication

5/25

Vertical Scaling

MasterDC1

72GB RAM

SlaveDC2

8GB RAM

Replication

6/25

Tip #1

Keep your indexes in memory at all times.

7/25

db.stats()

Manual Partitioning

Master ADC1

16GB RAM

Slave ADC2

16GB RAM

Replication

Master BDC1

16GB RAM

Slave BDC2

16GB RAM

Replication

8/25

Database vs collections

• Many databases = many data files (small but quickly get large).

• Many collections = watch namespace limit.

9/25

Namespaces = Number of collections + number of indexes

10/25

Tip #2

Monitor the 24,000 namespace limit.

11/25

Using Server Density

12/25

Console

db.system.namespaces.count()

13/25

Replica Pairs = Failover

Master ADC1

16GB RAM

Slave ADC2

16GB RAM

Replica Pair

Master BDC1

16GB RAM

Slave BDC2

16GB RAM

Replica Pair

14/25

Tip #3

Pre-provision your oplog files.

15/25

for i in {0..40} do echo $i head -c 2146435072 /dev/zero > local.$i done

A shell script to generate 75GB oplog files

16/25

Tip #4

Expect slower performance during initial replica sync.

17/25

Tip #5

You can rotate your log files from the console.

18/25

Rotating your log files

db.runCommand("logRotate")

19/25

Tip #6

Index creation blocks by default. Use background

indexing if necessary.

MongoDB Manual: http://bit.ly/mongobgindex20/25

Tip #7

Increase your OS file descriptor limit + use

persistent connections.

21/25

Too many open files!

mongo hard nofile 10000mongo soft nofile 10000

/etc/security/limits.conf

UsePAM yes

/etc/ssh/sshd_config

22/25

user type limit

Space is not reused

23/25

Tip #8

10gen commercial support is worth paying for.

24/25

Summary1. Keep indexes in memory.

2. Monitor the 24k namespace limit.

3. Pre-provision oplog files.

4. Expect slower performance on replica sync.

5. Rotate logs from the console.

6. Index creation blocks by default.

7. OS file descriptor limit + persistent connections.

8. Commercial support is worth it.25/25