MongoDB: Awesomely Dangerous

MongoDB:Awesomely Dangerous

Twin Cities Code CampOctober 2010

Sunday, October 10, 2010

Ethan Gundersonhttp://ethangunderson.com

Twitter & Github: ethangunderson


http://ethangunderson.com





Our agenda1) Intro to MongoDB

2) Advanced Features

3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


30,000 foot overview

• Schema-less

• Scalable

• High-Performance

• Open Source

• Document Database


Driving Principles

1) Performance

2) Performance

3) Scalability


Documents

• Similar to traditional rows

• First attribute is _id, which is an ObjectId


ObjectID

1) Timestamp

2) Machine Id

3) Process Id

4) Counter

4b6857a07613c367094426b2


Embedded Documents• Documents in Documents

• Indexable

• Queryable


Max Document Size

• 4MB limit on individual documents

• Realistically ~250kb

• Used to force better data modeling


Collections• Similar to traditional tables

• Collections of like documents

• Schema-less


Capped Collections

• Fixed size collections

• Must be explicitly created

• Limited functionality (no deletes, limited updates)



BSONA language independent data interchange format

• The language of Mongo

• Similar to JSON, but BETTER

• Fast

• 10gen driver support for: C, C++, Java, Javascript, Perl, PHP, Python, Ruby

• Community support for:REST, C#, Clojure, Coldfusion, Scala, and a lot more


B-Tree Indexes• Similar to a traditional RDBMS

• Can index any field, including arrays

• Missing keys in a unique will be given a value of null

• Blocking by default



Inserting


Updating


Modifier Operators

$inc$set$push$pushAll$pop$pull$pullAll$addToSet

Conventional updates do work, they’re just not as fast.


Querying


Query Operators$in$nin $all $ne $gt$gte$lt

$lte$size$where$limit $offset$sort$slice




3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


Map / Reduce• Replaces GROUP BY in SQL

• Similar in spirit to Hadoop with all info coming from a collection and going to a collection

• Runs in parallel on all shards, but only one thread per node

• map and reduce functions written in Javascript


GIS

• Built with location based queries in mind

• Assumes a flat map model of the Earth(!)


Near Queries


Bounded Queries


GridFS• How you store large files in Mongo

• Spreads data to multiple 256kb documents in a ‘chunks’ collection

• Meta data about the file is stored in the files collection

• Permits range operations (x bytes from file)




3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


The journey of a write


Memory Mapped File

Save!Success!



Safe Mode

• Allows you to determine the durability of a write per query

• Sacrifice performance for safety

• Options for each stage of the write


Safe mode query


Memory Mapped File

Save!

Success!


FsyncEvery 60 seconds, or when the kernel forces it


Save mode with fsync


Memory Mapped File

Save!

Success!


Safe mode with replication flag


Memory Mapped File

Save!

Success!


Save Flag with fsync and replication


Memory Mapped File

Save!

Success!


Low and High Value

• Useful to determine when building queries

• Allows you to be more careful(and slow), with data that is more important




3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As



Single Server DurabilityIt’s not



What 10gen has to say...

http://blog.mongodb.org/post/381927266/what-about-durability

True single server durability is almost never done correctly.First, there are many scenarios in which that server loses all its data no matter what. If there is water damage, fire, some hardware problems, etc… no matter how durable the software is, data can be lost.

The path to true durability is replication.






3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


ScalingSince we’re forced to think about it


Determining how to scale

Reads

Writes


Replica Sets

• Distribute reads across the cluster

• Replaces the traditional Master/Slave setup

• Replication is done via an ops log

• Auto failover

• Rack and Datacenter aware

• Smart, very smart


Master

SlaveSlave Slave


Master

SlaveSlave Slave

X


Now a slave

New Master

Slave Slave

X


Slave

New Master

Slave Slave


Slave

Master

SlaveSlave


Slave

Master

SlaveSlave

X


Master

Slave Slave

X

Me! Me!


Master

Slave Slave

X

Me! Me!

Web Slice(Arbiter)

You


Determining how to scale

Reads

Writes


Being write heavyCurrently, Mongo can only process one concurrent write.

Usually not a problem, as writes are wicked fast


Auto-Sharding• Partitions data across the cluster in an

order preserving manner

• No support for load based partitioning

• Automatic failover and balancing of nodes

• Distributes writes across the cluster

• Based very heavily off of Yahoo!’s PNUTS and Google’s BigTable






3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


Takeaways• Mongo is fast, but it does interesting things to

be that fast

• Mongo is not SQL. You will need to learn new things




3) Write Path

4) Durability

5) Scaling

6) Takeaways

7) Qs and As


Qs and As

http://spkr8.com/t/4756




ResourcesOfficial MongoDB sitehttp://mongodb.org

BSON sitehttp://bsonspec.org

Comprehensive writeup of mongo featureshttp://www.markus-gattol.name/ws/mongodb.html


http://mongodb.org

http://mongodb.org

http://bsonspec.org

http://bsonspec.org

http://www.markus-gattol.name/ws/mongodb.html#virtual_collection

http://www.markus-gattol.name/ws/mongodb.html#virtual_collection

Technology

MongoDB: Awesomely Dangerous