Advanced Administration, Monitoring and Backup

Advanced AdministrationMonitoring and Backup at Scale

Dr. Jeffrey BergerLead Database Engineer - Sailthru

Scale The Universe!

Scale The Universe!

Flat FRW Metric for isotropic cosmological geometry

Scale The Universe!

Flat FRW Metric for isotropic cosmological geometry

Scale Factor

Scale The Universe!

Flat FRW Metric for isotropic cosmological geometry

Scale Factor

Related to the hubble constant for an expanding universe, this does a great job of actually scaling our universe.

In fact the rate of expansion is continuing to grow and accelerate!

Scale The Universe!

Uhh maybe just the galaxy

The world is a lot still

Keep zooming in...

‘Big Data’...?

Sailthru

Sailthru

● Extremely early adopter of MongoDB ~2009

● 4 Clusters and 9 Stand-Alone RS

● Largest is 32 shards and 5.5TB with ~1.5 billion profiles

● All production systems are housed in a colo data center

on hardware owned and operated by Sailthru

Sailthru

● 4 DB Team Members

○ Me

○ Dr. Joshua Wickman

○ Chandrakant Gopalan

○ Tim Burrington

Sailthru

● Our systems are composed of replica sets of 2 live nodes and 1 arbiter

● Many of our systems are ‘microsharded’

PRIMARY

ARBITER

SECONDARY

PRIMARY

ARBITER

SECONDARY

Two tales of DBA struggle

No DBEs to DB team Mass Migration

What do you do if you have to move data from one data-center to another, while moving 17 replica sets into a single sharded cluster

with no (minimal) downtime?

What do you do when you join an organization which has been using

MongoDB without any DBA oversight?

Welcome to the DB team

What are the most important things for a DB to set up?

MONITORING BACKUPS

Monitoring

Microsharded systems are not easy to monitor!

● Multiple replica sets on a single machine

● Primaries and Secondaries often sharing hardware

● Monitoring systems for Mongo are at a instance level not server level

SHARD 1 PRIMARY

SHARD 2 SECONDARY

MEMORY

DISK IO

NETWORK IO

Monitoring - MMS

MMS is a great tool for all Mongo deployments

● Built in user level permissions● Automatic topology discovery● Graphs and time series data● Breakdown by replica set for

clusters● Pulls a wealth of data

Monitoring - MMS

● Built in alerting

● Many variable alerting criteria

● Integration with email, SMS, Pagerduty and more

Monitoring - MMS

MMS is our backup monitoring system

● Alerting time sometimes lags behind issue time

● Organizational decision not to host MMS and that we need an internal monitoring system as our main monitor

Monitoring - MMS

What we are looking forward to:● Proactive Support has some great features coming

through MMS● Enhanced monitoring and alerting options● Logging long queries? Non-indexed queries?● Perhaps we can run custom scripts and checks against

the system eventually!

Monitoring - Zabbix

“Quis custodiet ipsos custodes?” - ZABBIX

Monitoring - Zabbix

Monitoring mongo with Zabbixhttps://github.com/sailthru/mongodb-zabbix

● Number of voting members● Long query logging● Chunk distribution in a sharded cluster● Fsync lock status● Failover notification

Monitoring - Zabbix

Custom checks and graphs - cluster monitoring

Monitoring - Zabbix

Long Query Logging

Monitoring - Zabbix

Zabbix does not have any automated topology discovery!

Sailthru has created its own MongoDB topological discovery tool : DB Map

● Python Process● Automatically discovers nodes or config changes● Outputs all servers and information to a Mongo collection

Admin Tool - DB Map

Useful for many processes in our system

● Management scripts● Execute aggregation queries to pull specific systems● Keep Zabbix in sync using it as a source of truth● Exportable for Ansible inventory files or other

management software● Soon to be Open Sourced

Built By : Dr. Joshua Wickman

Backups

Many ways to skin a… cluster....?

● Volume snapshots (within our Datacenter)

● Snapshots of cloud secondaries (Hybrid Cloud)

● MMS Backups

Backups - Hybrid Cloud

SECONARY(HIDDEN)

SECONDARY

PRIMARY

DATACENTER

CLOUD

Sailthru had a hybrid cloud-physical topology.

Backups - Hybrid Cloud

● Disaster recovery is immediate● Backups can be taken care of by EC2

snapshotting

There are benefits to a hybrid setup

Backups - Hybrid Cloud

PRIMARY PRIMARY PRIMARY PRIMARY

SECONDARY SECONDARY SECONDARY SECONDARY

SECONDARY(hidden)

SECONDARY(hidden)

SECONDARY(hidden)

SECONDARY(hidden)

DC

Cloud

Backups - Hybrid Cloud

PRIMARY PRIMARY

SECONDARY SECONDARY

SECONDARY(hidden)

SECONDARY(hidden)

● Are these secondaries on hardware provisioned equally to the others?

● Is there enough bandwidth?● Can the disks keep up with

bursts of write activity?● Are the oplogs on these

secondaries long enough?● Is the connection to the

cloud secure and stable?

Backups - Hybrid Cloud

DO YOU HAVE THE TIME AND RESOURCES TO DO ALL OF THAT WORK??

We all just want backups that are fire-and-forget it!

Backups - MMS

● Save on your team’s time● Save on the provisioned hardware● Much cheaper than hybrid cloud solution

Sailthru has saved almost 1 million dollars year over year

Backups - MMS

● UI is easy to use and great for small/individual sets

● Need automation in order to bring up a cluster of any reasonable size○ Automation tools not yet

available out of the box● Pulls your data across the

internet - make sure you allocate this time!

The Power is Turning Off...

During 2014 Sailthru was forced to move Data Centers

Additionally we made the infrastructure decision to move from 17+ separate replica sets to a sharded cluster.

Data Migrations

DC1 DC2

CLOUD

With limited bandwidth and servers this becomes some interview’s brain teaser

Data Migrations - Dumps

DC1 DC2

MongodumpNetcat

Write to file then Mongorestore

● Lots of combinations, none ended up being fast enough.● Hampered by disk writes and reads.● If you touch disk you lose! The floor is lava!

Data Migrations - Mongopipe

Custom multiprocessing python process to insert without hitting disk

● Using python, multiprocessing, ZMQ, and some custom C objects

● Got around 2.4 bulk insert issue by sorting on shard key● Never touches disk, all processing is done in memory● Directly insert into many local mongos instances● Open source coming soon!

Data Migrations - Mongopipe

Cursor

Cursor

Cursor

Writer

Writer

Writer

Mongos

Mongos

Mongos

Target Cluster

ZMQ Batch Inserts

Sort on Shard Key

DC1 DC2

Data Migrations - Mongopipe

insert query update delete getmore command

64982 25 *0 *0 0 45|0

62484 23 *0 *0 0 50|0

37490 15 *0 *0 0 25|0

-1073585030 -4978381 *0 *0 -163 -5042014|0

197448 70 *0 *0 0 144|0

227440 105 *0 *0 0 181|0

49986 45 *0 *0 0 59|0

Data Migrations - Mongo Connector

● Mongoconnector is a way to mirror mongodb operations, creating almost a virtual secondary without adding it to a replica set

● Great for data migrations without downtime

https://github.com/10gen-labs/mongo-connector

Data Migrations - Mongo Connector

MONGO

OP LOG1….2….3….

TARGETDATASTORE

Elasticsearch..Solr...

Mongodb...

MONGO CONNECTOR

OPLOGMNGR.

DOC MNGR.

DOC MNGR.

DOC MNGR.

Access Patterns - Keystore

● What if I want to do a lot of findOnes on a cluster?● On many unique fields?● Am I doomed to many scatter gathers?

SHARD SHARD SHARD SHARD

MONGOSAssume sharded on _id: hashed

findOne({“ssn”: X}) findOne({“cell_phone”: X}) findOne({“_id”: X})

Created by : Ian White

Access Patterns - Keystore

Find by SSN

SHARDED COLLSharded on:{_id: hashed}

Doc:{ _id: SSN sid: ObjectId()}

Query on _id (shard key)Return an ObjectId

Main Sharded CollectionSharded on :{_id: hashed}

Use sid that was found to query the _id in the main collection

Access Patterns - Keystore

2 queries rather than n where n is your number of shards** Not useful unless you are sharded out very far **

● Time averaged by keystore : ~30 seconds● Time averaged by direct lookup: ~170 seconds

** tests done on a 32 shard cluster

Other Tools - Mongoexup

● Cron jobs are unreliable● Any ‘prototype’ inevitably becomes production● Constructed a python scheduler daemon to execute

these tasks● Looking to open source in the future

Business need to regularly execute mongoexport and uploads

Built By : Chandrakant Gopalan

Other Tools - Mongoexup

Mongo MongoExUp S3Greenlets Greenlets

Job Status Information

What are we doing next?

● Open source even more of our tools● Ansible Automation● Building API layers around all our DBs

○ Tornado - ASYNC RULES● MongoDB + Other Data Stores

○ Enhancing the Keystore concept● Upgrading

○ WT○ RocksDB