-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
1/35
MySQL High Availability SolutionsMySQL High Availability
Solutions
Lenz GrimmerLenz Grimmer
2009-02-082009-02-08FOSDEM 2009, Brussels, BelgiumFOSDEM 2009,
Brussels, Belgium
mailto:[email protected]:[email protected]
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
2/35
MySQL DevRoom in AW1.126 MySQL Cluster session in the
DevRoom
NOW (sorry) Updates to the MySQL DevRoom Schedule 13:15: Q&A
with Kaj Arn
http://forge.mysql.com/wiki/FOSDEM_2009
http://forge.mysql.com/wiki/FOSDEM_2009http://forge.mysql.com/wiki/FOSDEM_2009
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
3/35
Agenda High Availability: Concepts &
Considerations MySQL Replication
DRBD / Heartbeat MySQL Cluster Other HA tools/applications
Links
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
4/35
Why High Availability Matters Something can break and usually
will Maintenance requirements Downtime is expensive You miss $$$
Your Boss complains New Site visitors won't come back Adding HA to
an existing system is
tricky
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
5/35
What Is HA Clustering?
Redundancy One service goes down others take over IP address
takeover, service takeover Failover vs. failback vs. switchover Not
designed for high-performance
Not designed for high troughput (loadbalancing)
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
6/35
High Availability Levels
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
7/35
Eliminating the SPOF Identify what will fail
Disks Find out what can fail
Network cables OOM Power supplies
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
8/35
HA Components Heartbeat
Checks that services that are being failed over,are alive.
Can check individual servers, software services,networking
etc.
HA Monitor Configuration of the services Ensures proper shutdown
and startup
Allows manual control Shared storage / Replication
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
9/35
Replication vs. Shared Storage Shared storage resource can
become SPOF Split brain situations can lead to mayhem
(e.g. mounting file systems concurrently) SAN/NAS read I/O
overhead Consistency of replicated data Synchronous vs.
asynchronous replication
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
10/35
Split-Brain Communications failures can lead to
separated partitions of the cluster If those partitions each try
and take control of
the cluster, then it's called a split-brain
condition If this happens, then bad things will happen
http://linux-ha.org/BadThingsWillHappen Use Fencing or
Moderation/Arbitration to
avoid it
http://linux-ha.org/BadThingsWillHappenhttp://linux-ha.org/BadThingsWillHappen
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
11/35
Rules of High Availability Prepare for failure Aim to ensure
that no important data is lost Keep it simple, stupid (KISS)
Complexity is the enemy of reliability Automate it Test your setup
frequently!
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
12/35
MySQL Replication One-way, statement- or row-based One Master,
many Slaves Asynchronous Slaves can lag Master maintains binary
logs & index Easy to use and set up Built into MySQL
Replication is single-threaded No automated fail-over No heartbeat,
no monitoring
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
13/35
MySQL Replication Overview
Web/AppServer
Web/AppServer
Read & Write
MySQL Master
I/OThread
SQLThread
Write
Relay
Log
MySQL Slave
mysqld
Index &Binlogs
mysqld
DataBinlogReplication
Data
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
14/35
Statement-based replication Pro
Proven (around since MySQL 3.23) Smaller log files Auditing of
actual SQL statements No primary key requirement for replicated
tables Con
Non-deterministic functions and UDFs LOAD_FILE(), UUID(),
USER(),
FOUND_ROWS()(but RAND() and NOW() work)
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
15/35
Row-based replication Pro
All changes can be replicated Similar technology used by other
RDBMSes Fewer locks required for some INSERT,
UPDATE or DELETE statements Con
More data to be logged Log file size increases
(backup/restore
implications) Replicated tables require explicit primary keys
Possible different result sets on bulk INSERTs
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
16/35
Replication Topologies
Master > Slave
Masters > Slave (Multi-Source)
Master < > Master (Multi-Master)
Master > Slaves
Ring (Multi-Master)
Master > Slave > Slaves
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
17/35
Master-Master Replication Useful for easier failover Not
suitable for load-balancing
Writes still end up on both machines Neither machine has the
authorative data
Don't write to both masters! Use Sharding or Partitioning
instead (e.g.
MySQL Proxy)
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
18/35
MySQL Replication as an HA Solution
What happens if the Master fails? Nothing really, except the
application will not
work and the Slave will not have anything toreplicate from
What happens if the Slave fails? Nothing really (except data
will no longer bereplicated)
This doesnt sound like High Availability? Correct MySQL
Replication is a component of
an HA setup, but it doesnt implement all parts ofan HA
Solution!
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
19/35
Replication & HA Combined with Heartbeat Virtual IP takeover
Slave gets promoted to Master Side benefits: load balancing &
backup Tricky to fail back No automatic conflict resolution Proper
failover needs to be scripted
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
20/35
Linux-HA / Hearbeat Supports 2 or more nodes Resource monitoring
Active fencing mechanism: STONITH Policy-based resource
management,
dependencies & constraints Time-based rules Includes support
for many applications GUI support Low dependencies / requirements
Subsecond node failure detection
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
21/35
Replication
Applications
Virtual IPMaster HA Slave
Replication
Scale-out Slave
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
22/35
DRBD Distributed Replicated Block Device RAID-1 over network
Synchronous/asynchronous block replication Automatic resync on
recovery Application-agnostic Can mask local I/O errors
Active/passive configuration by default Dual-primary mode (requires
a cluster
filesystem like GFS or OCFS2) http://drbd.org/
http://drbd.org/http://drbd.org/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
23/35
DRBD in detail DRBD Replicates data between two disk
partitions DRBD integrates nicely with Linux-HA and
other HA Solutions MySQL runs on the
Active node as usual MySQL is dormant on
the Passive node DRBD is Linux only
Applications
Virtual IPActive Node Passive Node
DRBD
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
24/35
MySQL Cluster Shared-nothing architecture Automatic partitioning
Distributed Fragments Synchronous replication Fast automatic
fail-over of data nodes Automatic resynchronization Transparent to
Application Supports Transactions
l
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
25/35
MySQL Cluster In-memory indexes Not suitable for all query
patterns (complex
JOINs, range scans) No support for foreign key constraints Not
suitable for large datasets/transactions Latency matters Can be
combined with MySQL Replication
(RBR)
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
26/35
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
27/35
Requir em e nt s MySQL Re plicat ion
A v a i l a b i l i t y
No Yes Yes No
No No Yes Yes
Var ies Var ies < 30s < 3s
Aut o r esync of dat a No No Yes Yes
Yes Yes MySQL Replicat ion
S c a l a b i l i t y
MySQL Replicat ion MySQL Replicat ion MySQL Replicat ion
YesRead-int ensive Yes Yes MySQL Replicat ion YesWr it e-in t
ensive No No Possible Yes# Nodes/Clust er Mast er /Slave(s) Mast er
/Slave(s) Act ive/Passive 255
MySQL Replication &Heartbeat
MySQL, Heart beat &DRBD
MySQLCluster
Autom ated IPFailover Autom ated DBFailover Typical Failover t i
m e
Geographicredundancy
MySQLReplication
Built-in loadbalancing
l
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
28/35
MMM MySQL Master-Master Replication Manager
Collection of scripts to performmonitoring/failover and
management
Master-Master replication configurations (one writable node)
Failover by moving virtual IP
http://code.google.com/p/mysql-master-master/
Fli
http://code.google.com/p/mysql-master-master/http://code.google.com/p/mysql-master-master/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
29/35
Flipper Perl Script that manages pairs of MySQL servers
replicating to each other. Automatic switchover, triggered
manually Enables one half of the pair to be taken offline for
maintenance work while the other half carries ondealing with
queries from clients
Moves IP addresses based on a role ("read-only","writable")
between two nodes in the master pair, toensure that each role is
available
http://provenscaling.com/software/flipper/
R d H t Cl t S it
http://provenscaling.com/software/flipper/http://provenscaling.com/software/flipper/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
30/35
Red Hat Cluster Suite HA and load balancing Supports up to 128
nodes Shared storage Monitors services, file systems and
network
status Fencing (STONITH) or distributed lock
manager Configuration synchronization
http://www.redhat.com/cluster_suite/
S l i Cl t / O HA Cl t
http://www.redhat.com/cluster_suite/http://www.redhat.com/cluster_suite/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
31/35
Solaris Cluster / OpenHA Cluster Provides failover and
scalability services Solaris / OpenSolaris Kernel-level components
plus userland Agents monitor applications Geo Edition to provide
Disaster Recovery
using Replication Open Source since June 2007
http://www.opensolaris.org/os/community/ha-clusters/
Related tools / Links
http://www.opensolaris.org/os/community/ha-clusters/http://www.opensolaris.org/os/community/ha-clusters/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
32/35
Related tools / Links Linux Heartbeat
http://linux-ha.org/ Linux Cluster Information Center
http://www.lcic.org/ha.html
Red Hat Cluster Suitehttp://www.redhat.com/cluster_suite/ Sun
Open High Availability Cluster
http://opensolaris.org/os/project/ha-mysql/
Tools/Links
http://linux-ha.org/http://www.lcic.org/ha.htmlhttp://www.redhat.com/cluster_suite/http://opensolaris.org/os/project/ha-mysql/http://opensolaris.org/os/project/ha-mysql/http://www.redhat.com/cluster_suite/http://www.lcic.org/ha.htmlhttp://linux-ha.org/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
33/35
Tools/Links MySQL Multi-Master Replication Manager
http://code.google.com/p/mysql-master-master/ Maatkit
http://maatkit.sourceforge.net/
Mon scheduler and alert
managementhttp://www.kernel.org/software/mon/ Continuent Tungsten
Replicator
https://community.continuent.com/community/tungsten-replicator
Q & A
http://code.google.com/p/mysql-master-master/http://maatkit.sourceforge.net/http://www.kernel.org/software/mon/https://community.continuent.com/community/tungsten-replicatorhttps://community.continuent.com/community/tungsten-replicatorhttp://www.kernel.org/software/mon/http://maatkit.sourceforge.net/http://code.google.com/p/mysql-master-master/
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
34/35
Q & A
Questions, Comments?
-
8/8/2019 FOSDEM-MySQL-HA-2009-02-08
35/35
Thank you!Lenz Grimmer < [email protected] >
mailto:[email protected]:[email protected]
