Performance Tuning and Monitoring Using MMS

Technical Support Manager, North America @ 10gen

Nicholas Tang

#MongoNYC

Performance Tuning and Monitoring Using MMS

Performance Tuning and Monitoring Using MMS, Nicholas Tang

Agenda

• What is MMS?

• Why use it?

• Setting it up and getting around

• Performance and monitoring (the fun stuff)

• Wrap up

What is MMS?


What is MMS?

• The MongoDB Monitoring Service: a free service (or software) for monitoring and management


Metric collection and reporting


Alerting


Event Tracking


Logs and Profile data


Hardware stats (CPU, disk)


DB stats


Basic user management

What’s in it for me?


Why?

• Great high level view + detailed metrics

• Low effort, high-return

• Makes it easier for us to help you!

• Makes you more attractive, promotes bone strength and muscle tone *

* - these last points still under review

How do I use this crazy thing?


Setting it up

http://mms.10gen.com/help/monitoring/tutorial/

• Setup an account

• Install the agent

• Add your hosts

• Optional: hardware stats through munin-node

• Optional: enable logging and profiling

• More info: http://mms.10gen.com/help/monitoring/install/



http://mms.10gen.com/help/monitoring/install/

http://mms.10gen.com/help/monitoring/install/


Notes

• Agent written in Python (moving to Go)

• Failover: run multiple agents (1 primary)

• Hosts: use CNAMEs, especially on AWS!

• You can use a group per env (each needs an agent)

• Connections are over SSL

• On-Premise solution for Enterprise customers that don’t want to use the hosted service

Performance tuningand monitoring


Finding the bottleneck

Source: http://www.flickr.com/photos/laenulfean/462715479/

http://www.flickr.com/photos/laenulfean/462715479/

http://www.flickr.com/photos/laenulfean/462715479/


What is performance tuning?1. Assess the problem and establish acceptable

behavior

2. Measure the current performance

3. Find the bottleneck*

4. Remove the bottleneck

5. Re-test to confirm

6. Wash, rinse, repeat

* - (This is the hard part)

(Adapted from http://en.wikipedia.org/wiki/Performance_tuning )

http://en.wikipedia.org/wiki/Performance_tuning

http://en.wikipedia.org/wiki/Performance_tuning


Pro-Tip: know thyself

You have to recognize normal to know when it isn’t.

Source: http://www.flickr.com/photos/skippy/6853920/

http://www.flickr.com/photos/skippy/6853920/

http://www.flickr.com/photos/skippy/6853920/


Some handy metrics to watch

• Memory usage

• Opcounters

• Lock %

• Queues

• Background flush average

• Replication stats


Example: replication lag

150,000s of lag == almost 2 days of lag!



Some common causes of replication lag:

• Secondaries underspecced vs primaries

• Access patterns between primary/ secondaries

• Insufficient bandwidth

• Foreground index builds on secondaries


Fun fact: oplog idempotencyOperations in the oplog only affect the value once, so they can be run multiple times safely.

Example: If you increment n from 2 to 3, n = 3 is fine; n + 1 is not.

Frequent, large updates means a big oplog to sync.



• Secondaries underspecced vs primaries

• Access patterns between primary/ secondaries

• Insufficient bandwidth

• Foreground index builds on secondaries

“…when you have eliminated the impossible, whatever remains, however improbable, must be the truth…” -- Sherlock Holmes

Sir Arthur Conan Doyle, The Sign of the Four



Example:

• ~1500 ops per minute (opcounters)

• 0.1 MB per object (average object size, local db)

~1500 ops/min / 60 seconds * 0.1 MB/op * 8b/B =~ 20 mbps required bandwidth


Use alerts!

Don’t wait until your secondaries fall off your oplog!


Examining memory and disk

Memory: resident vs virtual vs (non-)mapped



Page faults and Record Stats



Background flush and Disk IO

(Check out http://www.wmarrow.com/strcalc/ )

http://www.wmarrow.com/strcalc/


Monitoring: watch for warnings

MMS warns you if your systems have startup warnings or if they are running outdated versions.

Don’t ignore these!

Wrapping up


What’s next?

• Visual update - 3 weeks ago (June 3rd)

• Backup service (join the queue!)

• More UI/ UX improvements:– Enhanced dashboards– Improved cluster view


Summary

• MMS is a great, free service

• Setup is easy

• Metrics are awesome, preventing failures even more awesome

• There’s more functionality coming soon!


Questions?

• Ask me now…

• …or check out our special Ask the Experts session all about MMS and our new Backup Service!

Performance Tuning and Monitoring with MMS, Nicholas Tang

Next Sessions at 11:555th Floor:

West Side Ballroom 3&4: Basic Replication in MongoDB

West Side Ballroom 1&2: Advanced Sharding Features in MongoDB 2.4

Juilliard Complex: Business Track: How Criteo Scaled and Supported Massive Growth with MongoDB

Lyceum Complex: Ask the Experts Online Education Session

Empire Complex: Right Ways and Wrong Ways to Implement MongoDB

SoHo Complex: Automated Slow Query Analysis: Dex the Index Robot

Documents

Performance Tuning and Monitoring Using MMS