Embed Size (px)
Citation preview
Effective MySQL MonitoringBaron Schwartz
March 2012
www.percona.com
Who Am I?
www.percona.com
Who Am I?
www.percona.com
Who Am I?
● Maatkit● Innotop● Aspersa● JavaScript Libraries
● Percona Toolkit● Monitoring Plugins● Online Tools
www.percona.com
● Consulting
● Support
● Training
● Conferences
● Engineering
● Percona Server
● Percona XtraBackup
● Percona XtraDB Cluster
● Percona Toolkit
● Many More
www.percona.com
Agenda
● What Kind of Monitoring?● Historical Data, Graphing● Fault Detection and Alerting
● What to Monitor● Tools● Overview of Nagios Plugins● Demo of Cacti Templates
www.percona.com
Two Kinds of Monitoring
● Capturing metrics● Detecting faults
www.percona.com
Capturing Metrics
● Useful for troubleshooting, diagnosis, and “noticing things”
● Very handy for graphs/charts● Typical problem: you're missing the one thing
you want● What to monitor: everything?
www.percona.com
Fault Detection
● Monitor system health● Alert when there are problems
www.percona.com
Fault Detection
● Monitor system health● Alert when there are problems● Typical problem: spammy alerts
● Non-actionable● False positives● /dev/null email filters
www.percona.com
Fault Detection
● Monitor system health● Alert when there are problems● Typical problem: spammy alerts
● Non-actionable● False positives● /dev/null email filters
● What to monitor: only as much as needed?
www.percona.com
Why not do both in one tool?
● Caching● Consistency● Staleness
www.percona.com
Monitoring Tools
www.percona.com
Why So Many?
www.percona.com
Why So Many?
Thus is bornYet Another Tool
www.percona.com
Popular Tools
● Nagios● Cacti● Zabbix● MySQL Enterprise● New Relic● Circonus
● MRTG● Munin● Monit● MonYOG● Zenoss● … and many more
www.percona.com
Percona Monitoring Plugins
● The philosophy:● Don't create Yet Another Tool● Make existing tools better and easier to use● Nagios and Cacti are good enough for most
● The plugins are 100% free and open source● See http://www.percona.com/software
www.percona.com
Monitoring with Nagios
www.percona.com
Using Nagios
● Nagios is primarily for fault detection
www.percona.com
Using Nagios
● Nagios is primarily for fault detection● Works via plugins
● Executables that follow specific conventions● Exit code● Text to STDOUT
www.percona.com
Using Nagios
● Nagios is primarily for fault detection● Works via plugins
● Executables that follow specific conventions● Exit code● Text to STDOUT
● Nagios plugins are pretty universal● Portable to many/most similar systems● Zabbix, etc
www.percona.com
Nagios Plugins for MySQL
● There are too many plugins, too!● My suggestion:
● The official plugins (there are only a couple)● Plus Percona's (there is a decent selection)
www.percona.com
Why Percona's Plugins?
● Targeted towards avoiding spammy alerts● High quality● Good documentation● Free
www.percona.com
Why Percona's Plugins?
● Integrates with other Percona tools● Easy to extend and customize
● Highly structured● Designed for testability● Written in Bourne shell
www.percona.com
Why Percona's Plugins?
www.percona.com
Why Percona's Plugins?
This makes the pluginsunit-testable for quality.
www.percona.com
Overview of Percona Plugins
● Operating Environment:● LVM snapshots● Deleted files● Privileges● PID files● System memory
www.percona.com
Overview of Percona Plugins
● MySQL:● Table checksums● Replication● Processes● InnoDB status● Deadlocks● Arbitrary status counters
www.percona.com
What to Monitor
● Everything appropriate from the previous slides● Carefully selected status counters
● Uptime. If too small, server has restarted● Threads_connected approaches max_connections
www.percona.com
What NOT to Monitor
● In general, be cautious with...● Cache hit ratios● $variable-per-second● Threshold-based alerts
www.percona.com
What NOT to Monitor
● Why?● A single “right” threshold is hard to find● These are usually not reliable indicators● They are also usually not actionable
● They generate spam alerts!
www.percona.com
Writing Custom Checks
● Use the pmp-check-mysql-status plugin● You can easily monitor counters, ratios...● Example:
● -x Threads_running -w 20 -c 40● See the documentation for more examples
www.percona.com
Monitoring with Cacti
www.percona.com
Cacti
● Cacti is for metrics collection and graphing● Good: easy to install, easy to get started● Bad: tedious to template-ize
www.percona.com
Cacti
● Cacti is for metrics collection and graphing● Good: easy to install, easy to get started● Bad: tedious to template-ize
● Percona Monitoring Plugins fixes this● Ready-made templates● Template-generation utilities● Easy to extend and customize
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
www.percona.com
Take-away Points
● Record metrics eagerly● Alert cautiously● Try not to reinvent the wheel● Nothing's perfect; usually 2 systems are best
● One for graphing/trending/metrics● One for fault detection
www.percona.com
Monitoring Resources
● Blog posts about our plugins: http://goo.gl/PceN3 and http://goo.gl/i2XNA
● White papers about preventing downtime: percona.com/about-us/mysql-white-papers
www.percona.com
www.percona.com
● Public and on-premises courses● Upcoming dates:
● Dallas – May 14● London – May 21● Raleigh – June 18● Chicago – July 9
● See http://www.percona.com/training/
