MySQL Documentation _ Fail Over MySQL With HAProxy

8/12/2019 MySQL Documentation _ Fail Over MySQL With HAProxy

1/8

11/11/13 MySQL Documentation : Fail Over MySQL with HAProxy

mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy 1/8

Documentation

Documentation Home

Log in

Fail Over MySQL with HAProxy

Background

Once you've established replication, you still need to ensure the applications that use this database can always find a healthy replica.

This procedure takes an existing Master-Master Replicationand sets up fail over between the two servers, using the excellent open source HAProxyload balancer.

This lesson focuses on getting MySQL ready, and building up a simple HAProxy configuration. In production it would be wise to avoid a single point of failure by using

two HAProxies, with keepaliveddirecting traffic between them. It may also be wise to implement more detailed health checks between HAProxy and MySQL. This blog

postintroduces several shell scripts that can check the presence of vital databases, the status of replication, and even allow a server to voluntarily fail health check for

maintenance.

There is no substitute for experience.

You replicate data because it's important. Treat your important data right, and get concrete experience with replication in our hands-on course.

What's in the course?

Steps

1. Check that the Primary and Secondary Databases are Healthy

Primaryand Secondaryshould already be replicating, as at the end of the Establish Master-Master Replicationprocedure.

SSH to the Primary: primary.example.com

Connect to MySQL and insert a new row:

primary ~ $ mysql -u root

mysql> insert into important.stuff set details = 'Replicating Primary to Secondary';

Query OK, 1 row affected (0.02 sec)

SSH to the Secondary: secondary.example.com

Connect to MySQL and verify the row you inserted on the Primaryhas replicated:

secondary ~ $ mysql -u root

mysql> select * from important.stuff order by id desc limit 1;

+-----+----------------------------------+---------------------+

| id | details | happened |

+-----+----------------------------------+---------------------+

| 296 | Replicating Primary to Secondary | 2013-04-12 14:09:26 |

+-----+----------------------------------+---------------------+

1 row in set (0.00 sec)

mysql>

Still on the Secondary, insert another row:

mysql> insert into important.stuff set details = 'Replicating Secondary to Primary';


Returning to the Primary, verify the row has replicated:


+-----+----------------------------------+---------------------+


+-----+----------------------------------+---------------------+

| 297 | Replicating Secondary to Primary | 2013-04-12 14:10:26 |

+-----+----------------------------------+---------------------+


mysql>

After this step is completed:

1. On the primary, INSERT a new record in the table important.stuff

2. Within 1 second, that record is replicated to the secondary

3. On the secondary, INSERT a new record in the table important.stuff

4. Within 1 second, that record is replicated to the primary

2. Test the Application can Connect from the Proxy

The database has a user configured named applicationwith the privileges a web app might require on the table important.stuff .
http://mysql.wingtiplabs.com/courses/replication.phphttps://secure.wingtiplabs.com/login.php?after_auth=http://mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxyhttp://mysql.wingtiplabs.com/documentation/3gescaeg/establish-master-master-replicationhttp://mysql.wingtiplabs.com/courses/replication.phphttp://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/http://andyleonard.com/2011/02/01/haproxy-and-keepalived-example-configuration/http://haproxy.1wt.eu/http://mysql.wingtiplabs.com/documentation/3gescaeg/establish-master-master-replicationhttps://secure.wingtiplabs.com/login.php?after_auth=http://mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxyhttp://mysql.wingtiplabs.com/documentationhttp://www.wingtiplabs.com/


2/8



SSH to the Proxy: proxy.example.com

Make sure the applicationuser can log in from the Proxyserver.

proxy ~ $ mysql -h primary.example.com -u application -pMomToolAceHeap

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 23

Server version: 5.5.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit

proxy ~ $ mysql -h secondary.example.com -u application -pMomToolAceHeap







owners.


mysql> exit


1. From the proxy, the user applicationcan connect to MySQL on the primary.

2. From the proxy, the user applicationcan connect to MySQL on the secondary.

3. Install and Configure HAProxy

Install the HAProxy package on the Proxyserver:

proxy ~ $ sudo yum install haproxy

Loaded plugins: priorities, security, update-motd, upgrade-helper

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package haproxy.x86_64 0:1.4.22-3.2.amzn1 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================

Package Arch Version Repository Size

==============================================================================

Installing:

haproxy x86_64 1.4.22-3.2.amzn1 amzn-main 511 k

Transaction Summary

==============================================================================

Install 1 Package(s)

Total download size: 511 k

Installed size: 1.4 M

Is this ok [y/N]: y

Downloading Packages:

haproxy-1.4.22-3.2.amzn1.x86_64.rpm | 511 kB 00:00

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Installing : haproxy-1.4.22-3.2.amzn1.x86_64 1/1

Verifying : haproxy-1.4.22-3.2.amzn1.x86_64 1/1

Installed:

haproxy.x86_64 0:1.4.22-3.2.amzn1

Complete!

proxy ~ $

Now edit the HAProxy config file, /etc/haproxy/haproxy.cfg

proxy ~ $ sudoedit /etc/haproxy/haproxy.cfg

First, delete everything below the globalblock. The defaults that ship with HAProxy are tuned for web servers, not database servers.

/etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------

# Example configuration for a possible web application. See the

# full configuration options online.

#

# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#

#---------------------------------------------------------------------


3/8



#---------------------------------------------------------------------

# Global settings

#---------------------------------------------------------------------

global

# to have these messages end up in /var/log/haproxy.log you will

# need to:

#

# 1) configure syslog to accept network log events. This is done

# by adding the '-r' option to the SYSLOGD_OPTIONS in

# /etc/sysconfig/syslog

#

# 2) configure local2 events to go to the /var/log/haproxy.log

# file. A line like the following can be added to

# /etc/sysconfig/syslog #

# local2.* /var/log/haproxy.log

#

log 127.0.0.1 local2

chroot /var/lib/haproxy

pidfile /var/run/haproxy.pid

maxconn 4000

user haproxy

group haproxy

daemon

# turn on stats unix socket

stats socket /var/lib/haproxy/stats

...Delete all original content below this point...

Then insert this new config at the bottom of the file:


......some content not shown

group haproxy

daemon


stats socket /var/lib/haproxy/stats mode 777

listen MySQL 0.0.0.0:3306

timeout connect 10s

timeout client 1m

timeout server 1m

mode tcp

server Primary primary.example.com:3306

server Secondary secondary.example.com:3306

Restart the HAProxy service to apply the new config.

proxy ~ $ sudo service haproxy restart

Stopping haproxy: [FAILED]

Starting haproxy: [ OK ]


1. HA Proxy is running.

2. The stats socket /var/lib/haproxy/stats has permissions srwxrwxrwx

3. HA Proxy has a listenerconfigured, named MySQL

4. The listenercontains the server primary.example.com

5. The listenercontains the server secondary.example.com

4. Test Round Robin Load Balancing through HA Proxy

The HAProxy is listening on port 3306 (the default MySQL port) on the Proxyserver. Connections to MySQL through the Proxy's hostname, use HAProxy to

redirect to either Primaryor Secondary.

You can see whichserver HAProxy has connected you to by looking up the server_id, which has to be unique on every server participating in replication.

The Primaryis configured with server_id10.

The Secondaryis configured with server_id20.

Use the MySQL client's -eflag to get the results of one command, and disconnect.

Repeatedly checking the server_id, shows that the HAProxy connects to a different server each time:

proxy ~ $ mysql -h proxy.example.com -u application -pMomToolAceHeap -e'show variables like "server_id";'

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| server_id | 10 |

+---------------+-------+


+---------------+-------+


+---------------+-------+

| server_id | 20 |

+---------------+-------+

proxy ~ $


4/8




1. Repeated connections to MySQL on proxy.example.com alternate between server_id10 and 20.

5. Set up a Health Check

On the Primary, connect to MySQL and configure a user HAProxy can use to check the health of the two servers. This user doesn't need specific privileges, it will

just need to connect and pass authentication.

mysql> GRANT USAGE ON *.* TO 'haproxy'@'%';

Query OK, 0 rows affected (0.00 sec)

mysql>

This new MySQL user will replicate to the Secondaryautomatically.

Test that the Proxyserver can connect as the haproxyuser to both the Primaryand the Secondary.

proxy ~ $ mysql -h primary.example.com -u haproxy







owners.


mysql> exit

Byeproxy ~ $ mysql -h secondary.example.com -u haproxy







owners.


mysql> exit

Bye

Now, edit /etc/haproxy/haproxy.cnf to use the MySQL health check on both servers.





timeout connect 10s

timeout client 1m

timeout server 1m

mode tcp

option mysql-check user haproxy

server Primary primary.example.com:3306 check

server Secondary secondary.example.com:3306 check

Then restart HAProxy.


Stopping haproxy: [ OK ]Starting haproxy: [ OK ]

Now we can ask HAProxy for the status of our servers. HAProxy established a socket at /var/lib/haproxy/stats to take commands and show status. (It

could also host a web interface.)

proxy ~ $ echo "show stat" | socat stdio /var/lib/haproxy/stats

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,c

MySQL,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,

MySQL,Primary,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,42,0,,1,1,1,,0,,2,0,,0,L7OK,0,1,,,,,,,0,,,,0,0,

MySQL,Secondary,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,42,0,,1,1,2,,0,,2,0,,0,L7OK,0,1,,,,,,,0,,,,0,0,

MySQL,BACKEND,0,0,0,0,2000,0,0,0,0,0,,0,0,0,0,UP,2,2,0,,0,42,0,,1,1,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,

show statreturns a lot of information in comma-separated value format. Use the cutcommand to show just the first, second, and 18th column.

proxy ~ $ echo "show stat" | socat stdio /var/lib/haproxy/stats | cut -d, -f1,2,18

# pxname,svname,status

MySQL,FRONTEND,OPEN

MySQL,Primary,UPMySQL,Secondary,UP

MySQL,BACKEND,UP

The first column, MySQLis the name of our proxy, set in the listen MySQLline of our config file.

The rows MySQL,FRONTENDand MySQL,BACKENDare generated automatically because we used the simplified listenconfiguration. More complicated

configurations define the frontend and backend separately, see the HAProxy documentation.
http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#4http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#stats%20enable


5/8



MySQL,Primaryand MySQL,Secondaryare the servers, and both report that they are UP


1. HAProxy reports the Primaryis UP

2. HAProxy reports the Secondaryis UP

6. Configure HAProxy to be Active-Passive

It is not a good idea to commit writes to both masters at the same time. The Repair Replicationprocedure illustrates a primary key collision, but there are many

ways asynchronous replication can cause inconsistencies.

Instead, as long as the Primaryis available, HAProxy should always direct traffic to it.

Edit /etc/haproxy/haproxy.cnf to set the Secondaryto be a backup.





timeout connect 10s

timeout client 1m

timeout server 1m

mode tcp

option mysql-check user haproxy

server Primary primary.example.com:3306 check

server Secondary secondary.example.com:3306 check backup

Then restart HAProxy.


Stopping haproxy: [ OK ]


Ask HAProxy for the status of the servers again. This time, add column 20 and 21 to the cutcommand.

proxy ~ $ echo "show stat" | socat stdio /var/lib/haproxy/stats | cut -d, -f1,2,18,20,21

# pxname,svname,status,act,bck

MySQL,FRONTEND,OPEN,,

MySQL,Primary,UP,1,0

MySQL,Secondary,UP,0,1

MySQL,BACKEND,UP,1,1

Adding the backupoption in the config file has separated the servers into two groups, active and backup. HAProxy won't direct anytraffic to anybackup server

untilallthe active servers are down.

On the BACKENDrow, the actand bckcolumns count the number of healthy servers in each state: there is one active server, and one backup server.

On the server rows, the actand bckcolumns tell us which type a server is. The Primaryis active (1,0) and the Secondaryis backup (0,1)

Now, connecting repeatedly through HAProxy should always return server_id10:


+---------------+-------+


+---------------+-------+

| server_id | 10 |

+---------------+-------+


+---------------+-------+


+---------------+-------+

| server_id | 10 |

+---------------+-------+

proxy ~ $


1. HAProxy shows the Primaryis UPand active.

2. HAProxy shows the Secondaryis UPand backup.

3. Repeated connections to MySQL through the Proxyalways connect to the Primary

7. Test Failover by taking the Primary Down

Stop the MySQL service on the Primary.

mysql> exit

Bye

primary ~ $ sudo service mysqld stop

Stopping mysqld: [ OK ]

Now, on the Proxy, verify that HAProxy's health check has failed:




MySQL,Primary,DOWN,1,0


http://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication


6/8



There are two highlighted changes here. First, the Primaryhas been detected as DOWN. Second, there are now 0 active servers in the MySQL,BACKENDrow, so

traffic will be directed to the 1 live backup server.

Connect to MySQL through the proxy:

proxy ~ $ mysql -h proxy.example.com -u application -pMomToolAceHeap







owners.


mysql>

Validate you've been connected to the Secondary:

mysql> show variables like 'server_id';

+---------------+-------+


+---------------+-------+

| server_id | 20 |

+---------------+-------+


While the Primaryis down, the Secondarycan keep handling read and write traffic from the application. INSERT a new record in important.stuff .

mysql> insert into important.stuff set details='Inserted with primary down';



1. On the Primary, MySQL service is stopped.

2. HAProxy reports the Primaryis DOWN

3. Connections to MySQL through the Proxyalways connect to the Secondary

8. Restore the Primary, Verify Replication Brings it Up to Date

Restart the MySQL process on the Primary:

primary ~ $ sudo service mysqld start

Starting mysqld: [ OK ]

Back on the Proxyserver, close your MySQL connection through the HAProxy. Verify that HAProxy detects that the Primaryis back up.

mysql> exit

Bye







Connect to MySQL through the Proxyagain, and verify that you've reached the Primary:

proxy ~ $ mysql -h proxy.example.com -u application -pMomToolAceHeap

mysql> show variables like 'server_id';

+---------------+-------+


+---------------+-------+

| server_id | 10 |

+---------------+-------+


Check that the Primaryhas replicated the changes made on the Secondarywhile the Primarywas out of service.


+-----+----------------------------+---------------------+


+-----+----------------------------+---------------------+

| 289 | Inserted with primary down | 2013-05-02 17:22:38 |

+-----+----------------------------+---------------------+


mysql> exit

Bye


1. On the Primary, MySQL service is running.


3. Connections to MySQL through the Proxyalways connect to the Primary

9. Take the Primary Offline for Maintenance

It will occasionally be necessary to choose to failover to the Secondary, for example to perform maintenance or upgrades on the Primary. In that case, we'll need


7/8



to tell HAProxy to ignore the health check and never send traffic to the Primary.

First, we need to configure the HAProxy socket to take changes. Edit /etc/haproxy/haproxy.cfg


And update the stats socketto enable admin access.



group haproxy

daemon


stats socket /var/lib/haproxy/stats mode 777 level admin


timeout connect 10s

timeout client 1m


Restart the HAProxy service to apply the new config.


Stopping haproxy: [FAILED]


Now tell HAProxy to disable the Primaryserver.

proxy ~ $ echo "disable server MySQL/Primary" | socat stdio /var/lib/haproxy/stats

proxy ~ $ echo "show stat" | socat stdio /var/lib/haproxy/stats | cut -d, -f1,2,18,20,21# pxname,svname,status,act,bck


MySQL,Primary,MAINT,1,0



The Primary server reports its status as MAINTand does not receive any health check traffic.


1. HAProxy reports the Primaryis in MAINTmode.

2. Connections to MySQL through the Proxyalways connect to the Secondary

10. Restore Primary from Maintenance

To restore service, re-enable the Primaryserver through HAProxy:

proxy ~ $ echo "enable server MySQL/Primary" | socat stdio /var/lib/haproxy/stats









2. Connections to MySQL through the Proxyalways connect to the Primary

11. Set HAProxy to Start Automatically

By default the HAProxy service doesn't start automatically after a reboot.

You can verify this by running:

proxy ~ $ chkconfig --list haproxy

haproxy 0:off 1:off 2:off 3:off 4:off 5:off 6:off

We want HAProxy to start automatically for any appropriate runlevel. This command starts HAProxy in all the default runlevels, excluding0 (used to turn the

system off), 1 (used to recover from severe issues), and 6 (used to reboot).

proxy ~ $ sudo chkconfig haproxy on

proxy ~ $

Now check this server's current (normal) runlevel, and verify that HAProxy will start automatically the next time the server starts.

proxy ~ $ runlevel

N 3

proxy ~ $ chkconfig --list haproxy

haproxy 0:off 1:off 2:on 3:on 4:on 5:on 6:off


1. On the Proxy, chkconfigreports that HAProxy will start automatically in runlevel 3.

"Don't take the author's word for anything; prove it to yourself. Do the exercises and invent your


8/8



own." James Hague

We're happy to provide this how-to for anyone who needs it. If you're in the thick of a problem, we'd like to help where we can and stay out of your way.

But if you're preparing for a project, or you'd like some experience with advanced replication features, you should check out our hands-on online course.

What's in the course?

Wingtip Labs 2013

Something broken or confusing? [email protected]
mailto:[email protected]://mysql.wingtiplabs.com/courses/replication.phphttp://prog21.dadgum.com/169.html

Documents

MySQL Documentation _ Fail Over MySQL With HAProxy