Confidential

MongoDB Monitoring

Become a MongoDB DBA - What to Monitor (If you’re really a MySQLer)

Art van Scheppingen

Senior Support Engineer, Severalnines

Confidential

Logistics

☐ Webinar is recorded☐ Replay available soon☐ Feel free to ask questions at any time☐ Use your control panel to contact us☐ Or email us as well: info@severalnines.com

Confidential

Agenda

☐ Trending and alerting☐ Why do we collect data?☐ How does MongoDB monitoring compare to MySQL☐ Key MongoDB metrics to know about☐ Available open source MongoDB monitoring tools☐ How to monitor MongoDB using ClusterControl☐ Demo

Confidential

Trending and alerting

Confidential

Why do you need a good trending system?

Confidential

Confidential

Monitoring vs trending

☐ Monitoring system (i.e. Nagios)

☐ Checks if services are healthy

☐ Sends pages

☐ Trending system (i.e. Cacti, Graphite)

☐ Collects metrics

☐ Generate graphs

Confidential

Monitoring: Availability

☐ Do more than just opening a connection☐ Measure true status of nodes and cluster☐ Test read/write☐ Open essential databases and collections☐ Keep an eye on the replication lag☐ Increase oplog size?

☐ Check the full topology

Confidential

Trending: Why do we need trends?

☐ Trending☐ Plot trends of key (performance) metrics☐ Find problems before they arise☐ Pre-emptive problem management

☐ Trending tools☐ Granularity of sampling☐ More datapoints = better

Confidential

Why do we collect data?

Confidential

Why do we collect data?

☐ Periodical (daily/weekly) healthchecks

☐ Insight into all aspects of the database operations

☐ Post mortem and proactive monitoring

☐ Capacity planning

Confidential

Healthchecks

☐ Healthchecks are a pain

☐ You want to see aggregated

data

☐ You want to be able to drill down

to a particular host

☐ You want to see the most

important data first and dig in

later on

Confidential

Post mortem and proactive monitoring

☐ Ability to dig into past data

☐ Even less than 5s of data

granularity (hardware-

dependent)

☐ Low granularity allows you to

catch the issue as it evolves -

no need to wait 5 minutes for a

graph to refresh

Confidential

Insight into internals, capacity planning

☐ Graphs based on MySQL

status counters

☐ Overall status and per-node

graphs

☐ Ability to get a timeshifted

graphs - useful for comparing

workload changes across the

time

Confidential

How does MongoDB monitoring compare to MySQL

Confidential

Type of metrics to collect

☐ Quite similar to MySQL☐ Host metrics☐ Operational metrics☐ Storage engine metrics☐ Replication metrics

Confidential

Host Metrics: what for?

☐ Similar to most other databases

☐ Understand the utilization of the hardware

☐ Capacity planning

☐ Determine the type of an issue

☐ I/O related?

☐ CPU related?

☐ Network related?

Confidential

Host Metrics: what to look for?

☐ CPU utilization (should I add more nodes to the cluster?)

☐ Network utilization (am I running out of bandwidth?)

☐ Ping (how badly latency affects my MongoDB cluster?)

☐ Disk throughput and IOPS (am I within my hardware limits?)

☐ Disk space (do I have to plan for larger disks?)

☐ Memory utilization (do I suffer from a memory leak?)

Confidential

Operational metrics

☐ Similar to most other databases

☐ Throughput of the cluster☐ Relate throughput to cluster performance☐ Determine the type of an issue

☐ Request spikes?

☐ Write amplification related?

☐ Queueing?

Confidential

Storage engine metrics

☐ Storage engine specific

☐ MMAP

☐ Wired Tiger

☐ MongoRocks

☐ Insight in how the engine performs☐ Internal congestion

Confidential

Replication Metrics

☐ Throughput of the replication

☐ Durability of the oplog

☐ Replication lag

☐ Comparable to Galera replication

☐ Quorum based

☐ At least one secondary needs to acknowledge

Confidential

Eventual consistency

Confidential

Key MongoDB metrics to know about

Confidential

Oplog

☐ Similar to MySQL binary logs☐ Oplog: a special collection

☐ Limited size☐ Eviction of transactions (FIFO)

☐ Replication window☐ Time between first and last transaction in the oplog

Confidential

Oplog

mongo_replica_0:PRIMARY> db.getReplicationInfo()

{

"logSizeMB" : 1895.7751951217651,

"usedMB" : 419.86,

"timeDiff" : 281419,

"timeDiffHours" : 78.17,

"tFirst" : "Fri Jul 08 2016 10:56:01 GMT+0000 (UTC)",

"tLast" : "Mon Jul 11 2016 17:06:20 GMT+0000 (UTC)",

"now" : "Mon Jul 11 2016 17:15:06 GMT+0000 (UTC)"

}

Confidential

Replication lag

☐ CPU, IO or lock related☐ Outcome: similar to Galera

☐ Lagging behind could cause a full sync

Confidential

Replication lagmy_mongodb_0:PRIMARY> db.runCommand( { replSetGetStatus: 1 } ) {

…

"members" : [

{

"_id" : 0,

"name" : "10.10.32.11:27017",

"stateStr" : "PRIMARY",

"optime" : {

"ts" : Timestamp(1466247801, 5),

"t" : NumberLong(1)

},

},

{

"_id" : 1,

"name" : "10.10.32.12:27017",

"stateStr" : "SECONDARY",

"optime" : {

"ts" : Timestamp(1466247801, 5),

"t" : NumberLong(1)

},

},

…

],

"ok" : 1

}

Confidential

Connections

☐ Similar to MySQL when handling connections☐ Client drivers may support connection pooling

☐ Multiple non-blocking queries can use the same connection☐ Spawns new connections when low on threshold

☐ Increase of connections☐ Locking issues☐ Application request bursts

Confidential

Connections

mongo_replica_0:PRIMARY> db.serverStatus().connections

{ "current" : 25, "available" : 794, "totalCreated" : NumberLong(122418) }

Confidential

Transactions

☐ Atomicity on document level☐ No “real” transactions☐ Write data with the $isolated operator

☐ Similar to READ UNCOMMITTED☐ No rollback☐ Does not work on shards

Confidential

Transactions

mongo_replica_0:PRIMARY> db.serverStatus().opcounters

{

"insert" : 1355272,

"query" : 20712,

"update" : 8995,

"delete" : 0,

"getmore" : 400791,

"command" : 2405749

}

Confidential

Locks (generic)

☐ Three levels of (generic) locking☐ Global☐ Database☐ Collection

Confidential

Locks (generic)

mongo_replica_0:PRIMARY> db.serverStatus().locks

{

"Global" : {

"acquireCount" : {

"r" : NumberLong(6050583),

"w" : NumberLong(2416551),

"R" : NumberLong(1),

"W" : NumberLong(7)

},

"acquireWaitCount" : {

"r" : NumberLong(1),

"w" : NumberLong(1),

"W" : NumberLong(1)

},

…}

Confidential

Locks (WiredTiger)

☐ Document level locking☐ Tickets (threads)

☐ Read☐ Write

Confidential

Locks (WiredTiger)

mongo_replica_0:PRIMARY> db.serverStatus().wiredTiger.concurrentTransactions

{

"write" : {

"out" : 0,

"available" : 128,

"totalTickets" : 128

},

"read" : {

"out" : 0,

"available" : 128,

"totalTickets" : 128

}

}

Confidential

Cache

☐ MongoDB uses three tiers of cache☐ Filesystem☐ Active memory☐ Storage engine (WiredTiger / MongoRocks)

☐ Page faults☐ Evictions

Confidential

Page faults and cache usage

mongo_replica_0:PRIMARY> db.serverStatus().extra_info.page_faults

37912924

mongo_replica_0:PRIMARY> db.serverStatus().wiredTiger.cache

{

"bytes currently in the cache" : 887889617,

"modified pages evicted" : 561514,

"tracked dirty pages in the cache" : 626,

"unmodified pages evicted" : 15823118

}

Confidential

Available open source MongoDB monitoring tools

Confidential

Alerting Solutions

☐ Open Source☐ Nagios☐ Zabbix

☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl

Confidential

Nagios

☐ Nagios-MongoDB☐ https://github.com/mzupan/nagios-plugin-mongodb/☐ Performs some very important checks☐ Replication lag☐ Lock time percentage☐ Index miss ratio

Confidential

Zabbix

☐ MongoDB Zabbix monitoring plugin☐ https://github.com/nightw/mikoomi-zabbix-mongodb-monitoring☐ All the necessary metrics and more☐ Entries in oplog

☐ Pre-canned triggers

Confidential

Trending Solutions

☐ Trending tools☐ Statsd/Grafana☐ Cacti☐ Zabbix

☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl

Confidential

Cacti

☐ Percona MongoDB Monitoring Templates☐ https://www.percona.com/doc/percona-monitoring-plugins/1.

1/cacti/mongodb-templates.html

Confidential

Cacti

Confidential

Orchestration systems: Percona Monitoring & Management

☐ PMM☐ https://www.percona.com/doc/percona-monitoring-and-management/

☐ Open Source Monitoring & Management framework☐ Can deploy, manage and monitor MySQL & MongoDB☐ Uses Prometheus and Grafana☐ Still in Beta

Confidential

How to monitor MongoDB using ClusterControl

Confidential

Orchestration Systems: ClusterControl

☐ ClusterControl☐ http://www.severalnines.com☐ Deploy Mongo shards & replicasets☐ Monitor and trend☐ Manage configuration and backups☐ Scale

☐ Community edition

Confidential

Deploy Mongo replicaSets and Shards

Confidential

Monitor and Trend

Confidential

Management

Confidential

Scale

Confidential

Demo

Confidential

Q & A

Confidential

Additional Resources

☐ Blog series: Become a MongoDB DBA☐ http://severalnines.com/blog-categories/mongodb

☐ Webinar series: Become a MongoDB DBA☐ http://severalnines.com/upcoming-webinars

☐ Visit our website for more resources!☐ http://www.severalnines.com