Monitoring a Galera Cluster · High Availability 6 Slave Client Writes Client Reads Slave Client...

Monitoring a Galera ClusterCodership Training

Monitoring a Galera Cluster

IntroductionGalera & Monitoring OverviewGalera Status Variables

Server LogsNotification CommandCustomized Scripts

Third-Party SoftwareConclusion

Introduction

library@galeracluster.com

Monitoring a Galera ClusterSlide

Introductions

Codership OyCreators & Developers of

Galera Cluster

Employees in Multiple Countries

Galera ClusterReleased Initially in May 2007Over 1.5 Million Downloads

Russell Dyer, PresenterKB Editor, Documentation,

Instructor (MySQL, MariaDB)

Writer (O'Reilly Books)

Tutorial Outline

Galera & Monitoring OverviewGalera Status VariablesServer Logs

Notification CommandCustomized ScriptsThird-Party Software

Galera & Monitoring Overview

Basic Replication ConceptsNodes — Physical & Virtual ServersDatabases — MySQL, MariaDB, XtraDB

ReplicationLoad BalancingHigh Availability

Client Writes

Client Reads

Master

Galera Cluster ConceptsVirtual Synchronous ReplicationTrue Multi-Master SolutionConflict Detection & Resolution on

CommitMultiple, Odd Number of Nodes

Easy MaintenanceAutomatic Provisioning

Node Isolation

Rolling Upgrades

Node 1

Node 2

Node 3

Galera Cluster

Client Writes

Client Writes Client Reads

Monitoring a ClusterWhat to Monitor

Nodes Down

Partitioned ClustersLagging Nodes

Resources AvailableStatus Variables

Log FilesScripts & Software

Galera Status Variables

Galera Status VariablesCluster Integrity

Communication among Nodes

Node StatusNetworking & Basic Requirements of

Individual Nodes

Replication HealthPerformance and Dependability of Schema

and Data Changes

SHOW GLOBAL STATUS LIKE 'wsrep_%';

+------------------------+-------------------+ | Variable_name | Value | +------------------------+-------------------+ | wsrep_local_state_uuid | bd5fe1c3-7d80-... | | wsrep_protocol_version | 10 | | ... | ... | | wsrep_thread_count | 6 | +------------------------+-------------------+

Cluster IntegrityCompare UUIDs

(wsrep_cluster_state_uuid)

Count the Nodes (wsrep_cluster_size)Do an Audit (wsrep_cluster_conf_id)

Ensure Primary Status (wsrep_cluster_status)

SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';

+---------------------------+--------+ | Variable_name | Value | +---------------------------+--------+ | wsrep_local_state_comment | Synced | +---------------------------+--------+

Node StatusReady & Connected (wsrep_ready,

wsrep_connected)

Node State Comment

ERROR 1047 (08501) Unknown Command

SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';

+---------------------------+--------+ | Variable_name | Value | +---------------------------+--------+ | wsrep_local_state_comment | Synced | +---------------------------+--------+

SHOW STATUS LIKE 'wsrep_local_send%';+----------------------------+----------+| Variable_name | Value |+----------------------------+----------+| wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 1 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0.000000 |+----------------------------+----------+

Replication HealthBunching of Writes

(wsrep_local_recv_queue_avg)

Flow Control Paused (wsrep_flow_control_paused)

Sequentially in Parallel (wsrep_cert_deps_distance)

SHOW STATUS LIKE 'wsrep_local_recv%'; +----------------------------+----------+ | Variable_name | Value | +----------------------------+----------+ | wsrep_local_recv_queue | 0 | | wsrep_local_recv_queue_max | 1 | | wsrep_local_recv_queue_min | 0 | | wsrep_local_recv_queue_avg | 0.145000 | +----------------------------+----------+

Documentation on Replication Health: https://galeracluster.com/library/documentation/monitoring-cluster.html#check-replication-health

Server Logs

MySQL Server LogsError Log — Errors, Start-Up & Shutdown

Binary Log — Standard Replication and Point-in-Time Recovery

Slow Query LogSQL Error Log — Govt. Compliance

log-error=‘/var/log/mysqld.log'

log-warnings = 1

Relevant Entries from my.cnf configuration file

[mysqld] slow_query_log = ON slow_query_log_file=‘/var/log slow_query.log’ long_query_time = 0.5 log_queries_not_using_indexes log_slow_admin_statements

INSTALL PLUGIN sql_error_log SONAME 'sql_errlog.so';

Requires INSERT Privilege on mysql.plugin Table

SLOW QUERYLOG ENTRIES

log-bin binlog-expire-logs-seconds=302400

SQL ERROR LOG

BINARY LOG ENTRIES(PURGE AFTER 7 DAYS)

ERROR LOGENTRIES

Recording More in Server LogsLog Conflicts (wsrep_log_conflicts)

Certification Failures (cert.log_conflicts)

Debugging (wsrep_debug)

log-error wsrep_log_conflicts=ON wsrep_provider_options="cert.log_conflicts=ON" wsrep_debug=ON

Relevant Entries from my.cnf configuration file

Log Conflicts

Notification Command

NotifyingNotification Script on Each NodeNode Status Changes (e.g., Disconnects)Notification Command Triggered

Node 1

Node 3

Node 2

Galera Cluster

DISCONNECTED

Trigger Notification Command

Node Status StringStarting Node

Undefined — Not Part of PC

Joiner — Receiving SST

Existing NodesDonor — Sending SST

Joined — Processing Updates

Synced — Synchronized

Error — Problems Occurred

Documentation on Node Status Values: https://galeracluster.com/library/documentation/notification-cmd.html#node-status

Members ListNode UUIDNode Name (wsrep_node_name)

Incoming Address (wsrep_node_incoming_address)

09fd7887-a84d-11e9-bdaa-1718db36144e/galera2/AUTO, 0a1e9504-a871-11e9-944c-13bde588c4e3/galera3/AUTO, 38807f55-a7d2-11e9-aa9b-7674d700b955/galera1/AUTO

A MEMBERS LIST

Line Breaks Added

Documentation on Members List: https://galeracluster.com/library/documentation/notification-cmd.html#member-list-format

Customized Scripts

Simple Notification Script

Setting a Notification CommandLocate Script in Protected DirectoryTouch Log FileChange Permissions & OwnershipSet wsrep_notify_cmd in

Configuration File

cd /var/log touch galera-node-monitor.log chown mysql:mysql galera-node-monitor.log chmod 755 galera-node-monitor.log

wsrep_notify_cmd='/commands/note-cmd.bash'

Executed from Command-Line

PREPARINGEMPTY LOG FILE

TELLING GALERA ABOUTNOTIFICATION COMMAND

Excerpt from MySQL Configuration File (e.g., my.cnf)

Testing a Notification Script

Notifications Logged in MySQLLogging to MySQL will Replicate to

All NodesGenerating Reports is Easier

Patterns may Emerge

Detect Problems Early

MySQL Database Log

Notification ScriptInterfacing with MySQL

Creating an AlertConditional Statements for Alert

WorthinessEmail Alertmail

sendmail

SMS Telephone MessageSubscription to Text Messaging Service

Determine DBA On-Call

Assemble Alert Message

curl -X POST https://textbelt.com/text --data-urlencode phone='$admin_nbr' --data-urlencode message='$alert' -d key=textbelt

Third-Party Software

Monitoring & Trending SoftwareMonitors

Cluster StatusNode States

Cluster Size

Flow Control

Send & Receive QueueSlow Nodes

TrendsRecord Stats for Reports & Graphs

Host Metrics — Repetition & PatternsSpot Physical Server & Network Problems

InnoDB Status (e.g., Spikes)

Popular SoftwareMonitoring

Nagios Plugin

Galera Monitor (MariaDB MaxScale)Load Balancing, Designating Master & Slaves

MonyogIndividual Nodes

TrendingSCUMM (Several Nines)

Percona Monitoring & Management

SCUMM: https://severalnines.com/database-blog/scumm-agent-based-database-monitoring-infrastructure-clustercontrol

Nagios Plugin: https://www.fromdual.com/galera-cluster-nagios-plugin-en

Percona Monitoring & Management: SCUMM: https://severalnines.com/database-blog/scumm-agent-based-database-monitoring-infrastructure-clustercontrol

Conclusion

A Monitoring & Reaction PlanInventory of Nodes & Cluster

Notes on Location, Logins, Etc.

Minimum Nodes Needed for TrafficSpare Nodes — Notes on Deploying Them

Monitoring PlansList of Relevant Status Variables

Log Files — Path and Names

Method or Software for Monitoring

Practice Removing a Node for Maintenance

Adding a Node for Increased TrafficRestarting a Cluster

Additional ResourcesCodership Library (galeracluster.com/

library)Documentation (/library/documentation)

Knowledge Base (/library/kb)

FAQ (/library/faq)

Training (/library/training)

Videos (/library/training/videos)

Tutorials (/library/training/tutorials)

Tutorial Article on Monitoring Galera: https://galeracluster.com/library/training/tutorials/galera-monitoring.html

Thanks for Listening

Monitoring a Galera Cluster · High Availability 6 Slave Client Writes Client Reads Slave Client...

Documents

VME slave core Specifications No.Int-VME-SLAVE-0021 › resources › product_briefs › VME-Slave... · 2019-01-26 · VME slave core Specifications No.Int-VME-SLAVE-0021 Intelop

EtherCAT Slave - hilscher.com · The data below applies to EtherCAT Slave firmware and stack version 4.3.x. Supported Protocols SDO client and server side protocol (CoE component)

Catalogue 2014 - Schneider Electric Smart Panel/Modicon M241 PLC... · Modbus TCP (client & server), Modbus TCP slave, dynamic DHCP Client Configuration, Programming, download monitoring

Clean Reads

Consistent Reads Using ProxySQL and GTID...7 Master - Slave Replication Pain Points • Asynchronous replication • Replication lag is the major challenge• Semi-synchronous replication

248 I/O •6 HSC - w5.siemens.com · PLC Master Master Master Master Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave ... Programmazione diagnostica

Advanced Metering Infrastructure (AMI) Implementation · 3 36% Outside Meters 69,440 Remote Reads 104,145 Manual Reads 64% Inside Meters 303,000 Remote Reads 20,395 Customer Reads

MySQL Replication - code.openark.orgcode.openark.org/blog/wp-content/uploads/2009/07/replication_solut… · Master Slave Slave connects to master, Reads binary log Relay logs Slave

CA-NFS: A Congestion-Aware Network File System...Client Operations & Implicit Priorities synchronous versus asynchronous ops synchronous operations: – reads, metadata – must be

Good Reads

Quick Reads

PLC - منبع تغذیه مینول - MEAN WELL - منبع تغذیه سوئیچینگ learning farsi.pdfModbus THINGETPLC PLCModbus PLC Slave Slave Master Slave Master Slave ModbusThingetPLC

MCP7940N Low-Cost I2C Real-Time Clock/Calendar with …anarduino.com/docs/MCP7940N.pdf · reads, a master must signal an end of data to the slave by NOT generating an Acknowledge

The Basics. Network Management Automatic node detection Client-Master-Slave configuration Node pool management Active/Inactive node assignments No configuration

MySQL Multi-Site/Multi-Master Done Rightcontinuent-videos.s3.amazonaws.com/Continuent-Webinar-MSMM-20170608.pdf · MySQL Master MySQL Slave MySQL Slave Reads & Writes Reads & Writes

Echo server The client reads a line of text from its standard input and writes the line to the server The server reads the line from its network input

hoplongtech.com M2xx Logic... · Modbus TCP (client & server), Modbus TCP slave,dynamic DHCP Client Configuration, Programming, download rmnitoring_ Updated firmware, data exchange

A Slave is a Slave - H. Beam Piper

Power and Performance in I/O for Scientiﬁc Applicationscucis.ece.northwestern.edu/publications/pdf/ColCho05A.pdf · reads and writes. Another optimization made possible by client-sidecachingis

Groundbreaking reads