MySQL Fabric in a USB

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

MySQL in a USB Fabric: Master Slave & Switching over safely Keith Hollman Principal Sales Consultant EMEA MySQL [email protected]


Objective

• Have a fully functioning MySQL instance running as a Master from a USB pen drive, that’s ejectable.

• Locally on the same server, a MySQL Slave instance, that can be the master if needed.

• Be able to promote the slave, eject the master, then synchronize the master with the slave once plugged back in again. And promote the master back to it’s original master role.

2


Agenda

1

2

3

Component Installation

Installing MySQL

Fabric Configuration

Safe switchover / USB eject & switchback.

3

4



4


Component Installation Basically

• A MySQL master slave replication scenario has to be configured.

• In order to do the switchover & switchback transparently, it’s best done via MySQL Fabric.

• This means that we’ll require:

– MySQL Python Connector

– MySQL Utilities (MySQL Fabric)

– 1x MySQL binary (5.7.7 RC)

• There is an installation of MySQL Server in the server as well as the python connector & the Utilities (Fabric).

• The USB contains the Masters data and config file.

5


Installing the Python Connector

• Download the binaries from http://dev.mysql.com/downloads/connector/python/.

• http://dev.mysql.com/doc/connector-python/en/connector-python-installation-source.html

“Installing Connector/Python from Source on Unix and Unix-Like Systems”

shell> tar xzf mysql-connector-python-1.2.2.tar.gz

shell> cd mysql-connector-python-1.2.2

shell> sudo python setup.py install

• http://dev.mysql.com/doc/connector-python/en/connector-python-verification.html

shell> ls –l /usr/lib/python2.7/dist-packages/mysql*

6

http://dev.mysql.com/downloads/connector/python/

http://dev.mysql.com/doc/connector-python/en/connector-python-installation-source.html










http://dev.mysql.com/doc/connector-python/en/connector-python-verification.html









Installing MySQL Utilities

• Download the source code from http://dev.mysql.com/downloads/tools/utilities/1.6.html

• http://dev.mysql.com/doc/mysql-utilities/1.6/en/mysql-utils-install-source.html

“1.2. Source Code”

• As they’re python scripts, we can compile from source: shell> tar zxvf mysql-utilities-1.6.1.tar.gz

shell> cd /home/khollman/ofi/_MySQL/sw/utils/mysql-utilities-1.6.1

shell> python ./setup.py build

shell> sudo python ./setup.py install

shell> mysqluc –version

MySQL Utilities mysqluc version 1.6.1

License type: GPLv2

7

http://dev.mysql.com/downloads/tools/utilities/1.6.html

http://dev.mysql.com/downloads/tools/utilities/1.6.html

http://dev.mysql.com/doc/mysql-utilities/1.6/en/mysql-utils-install-source.html












Confirming MySQL Fabric availbility

• We can test that Fabric is installed and executbale via the following: shell> mysqlfabric –version

mysqlfabric 1.6.1

• Also check that the following file exists: shell> ls -lt /etc/mysql/fabric.cfg

8


Agenda

1

2

3


Installing MySQL



9

4


Installing MySQL

10


Installing MySQL Requirements

• The MySQL instances will have the datadirs y configuration files:

– Installed locally on the server: MySQL Fabric ‘state store’ repository.

– Installed on the USB: MySQL instance for the ‘unpluggable’ Master.

– Installed locally on the server: MySQL Slave instance.

• In this example, the binaries to be used wil lbe installed locally on the server, not on the USB. On the USB there’ll only be data and config file.

• Requirements depend on the environments needs, free space, usage of separate installs, versions, application flexibility, security, etc.

11


Installing MySQL Common Tasks

• First, let’s prepare the environment and local installation:

shell> groupadd mysql

shell> useradd -r -g mysql mysql

shell> cd /usr/local

shell> tar zxvf mysql-5.7.7-rc-linux-glibc2.5-x86_64.tar.gz

shell> ln -s /usr/local/mysql-5.7.7-rc-linux-glibc2.5-x86_64 /usr/local/mysql

shell> cd mysql

shell> mkdir mysql-files

shell> chmod 770 mysql-files

shell> chown -R mysql .

shell> chgrp -R mysql .

12


Installing MySQL Fabric specific tasks

• Create a my.cnf config file for the Fabric repository:

• Eg. vi /usr/local/mysql/my.cnf

• Make the repository uniquely identifiable:

– port, server-id, and the path of the datadir “/opt/mysql/fabric/63301/data”, etc.

– See Annex “Fabric my.cnf”

• We’ll be using 5.7, so the install process will NOT use mysql_install_db: shell> mysqld --defaults-file=/usr/local/mysql/my.cnf --initialize --user=mysql

13


Installing MySQL Fabric specific tasks

• And in 5.7 security has been improved, so we’ll have to find the root password and change it:

shell> tail /opt/mysql/fabric/63301/data/mysqld.err

shell> mysqladmin --defaults-file=/usr/local/mysql/my.cnf --ssl=0 -uroot –S

/opt/mysql/fabric/63301/mysql.sock –p’JHjuyTnq3=Sk’ password ‘password‘

shell> mysql_ssl_rsa_setup --defaults-file=/usr/local/mysql/my.cnf

shell> chown -R root .

shell> chown -R mysql mysql-files (data está en /opt/mysql/fabric/63301)

• Start the instance so Fabric can work: shell> mysqld_safe --defaults-file=/usr/local/mysql/my.cnf --user=mysql &

14


Installing MySQL Preparing the Master

• Now let’s prepare the USB that will hold the Masters’ data and my.cnf.

• Find the UUID of the USB:

shell> ls -lrt /dev/disk/by-UUID

• Get the UUID and enter it in /etc/fstab:

UUID=0012-D687 /media/usb_externo vfat

users,rw,noauto,exec,nofail,mand,suid 0 0

• And mount it as the o.s. user ‘mysql’:

shell> mount /media/usb_externo

15


Installing MySQL Preparing the Master I

shell> cd /media/external_usb/

shell> mkdir mysql

shell> cp /usr/local/mysql/my.cnf /media/external_usb/mysql/my3306.cnf

shell> ln -s /media/usb_externo/mysql/my3306.cnf

/usr/local/mysql/my3306.cnf

• Make sure the paths in the config file reflect the USB’s location: shell> vi my3306.cnf

:1,$ s!/opt/mysql/fabric/63301!/media/external_usb/mysql!g

• And that the socket file is located on the local server storage, and not on the USB pen drive.

– See Annex “my3306.cnf”

16


Installing MySQL Preparing the Master II

• We initialize the Master environment: shell> mysqld --defaults-file=/usr/local/mysql/my3306.cnf --initialize --

user=mysql

shell> tail /media/usb_externo/mysql/data/mysqld.err

shell> mysqladmin -S /tmp/mysql3306.sock -uroot –p’FjfLxwNfh0_t’ password

‘password‘

• Complete the secure installation for the Master:

shell> mysql_ssl_rsa_setup --defaults-file=/usr/local/mysql/my3306.cnf

• Start the Master instance:

shell> mysqld_safe --defaults-file=/usr/local/mysql/my3306.cnf --

user=mysql &

17


Installing MySQL Preparing the Slave

• Now we create the Slave environment: shell> cd /usr/local/mysql

shell> cp my.cnf my3307.cnf

• Make sure again that the config file reflects the paths we want for the slave:

shell> vi my3307.cnf (Annex my3307.cnf) :1,$ s!/opt/mysql/fabric/63301!/opt/mysql/fabric/3307!g

• Initialize the installation: shell> mysqld --defaults-file=/usr/local/mysql/my3307.cnf --initialize --user=mysql

tail /opt/mysql/fabric/3307/data/mysqld.err

shell> mysql_ssl_rsa_setup --defaults-file=/usr/local/mysql/my3307.cnf

shell> mysqld_safe --defaults-file=/usr/local/mysql/my3307.cnf --user=mysql &

shell> mysqladmin -S /tmp/mysql3306.sock -uroot -p'orL&qYnT26A!' password

‘password'

18


Installing MySQL Just a reminder

• Startup: mysqld_safe --defaults-file=/usr/local/mysql/my.cnf --user=mysql &

mysqld_safe --defaults-file=/usr/local/mysql/my3306.cnf --user=mysql &


• Shutdown: mysqladmin -S /tmp/mysql3306.sock -uroot -p shutdown

mysqladmin -S /opt/mysql/fabric/3307/mysql.sock -uroot -p shutdown


• mysql CLI mysql –S /tmp/mysql3306.sock –uroot

mysql –S /opt/mysql/fabric/3307/mysql.sock –uroot

mysql –S /opt/mysql/fabric/63301/mysql.sock -uroot

19


Agenda

1

2

3


Installing MySQL



20

4

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 21



Fabric Configuration Users

• We’ll need to give permission to root to be able to connect from the other instances, to be executed on all 3 (for this example):

mysql> GRANT ALL ON *.* TO root@'%' identified by ‘password‘;

– Ignore warnings of the type "Warning (Code 1287): Using GRANT for creating new user is deprecated and will be removed in future release. Create new user with CREATE USER

statement."

• Now to create the user that the Fabric processes will need, again, on all 3 instances, and for this example, keeping it easy yet unsecure:

mysql> GRANT ALL ON *.* TO fabric@'%' identified by 'fabric‘;

Recommended permissions: http://dev.mysql.com/doc/mysql-fabric/1.6/en/fabric-create-user.html

22

http://dev.mysql.com/doc/mysql-fabric/1.6/en/fabric-create-user.html








Fabric Configuration Users I

• Make sure we can access: shell> mysql -S /opt/mysql/fabric/63301/mysql.sock -ufabric –pfabric

• Quick change in fabric.cfg and we adjust the port (63301) and passwords: shell> vi /etc/mysql/fabric.cfg

• Now to start the logical configuration from within Fabric: shell> mysqlfabric --config=/etc/mysql/fabric.cfg manage setup

[INFO] 1437133916.793842 - MainThread - Initializing persister: user (fabric), server (localhost:63301), database

(fabric).

Finishing initial setup

=======================

Password for admin user is not yet set.

Password for admin/xmlrpc:

23


Fabric Configuration Users II

• You have to enter the password twice for the ‘admin’ user, for the xmlrpc protocol.

• Afterwards, you can configure the config file /etc/mysql/fabric.cfg to make admin tasks easier. If not, Fabric will constantly ask us for the admin password each time.

• Up until this point, all that has been done is to create the repository, in which the ‘fabric’ schema holds the tables that contain the configuration of the ‘backing store’.

shell> mysql -S /opt/mysql/fabric/63301/mysql.sock -ufabric -pfabric fabric

mysql> show tables;

24


Fabric Configuration Logical topology

• Let’s start the Fabric process in ‘daemon’ mode: shell> mysqlfabric manage start –daemonize

• Creating the logic. “mygroup” is the name of the group: shell> mysqlfabric group create mygroup

Password for admin:

Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e

Time-To-Live: 1

uuid finished success result

------------------------------------ -------- ------- ------

a46e3609-baa2-4ef4-a0d3-7d364b2a102f 1 1 1

state success when description

----- ------- ------------- ------------------------------------------------------------------

3 2 1437134826.51 Triggered by <mysql.fabric.events.Event object at 0x7f7d970ac710>.

4 2 1437134826.51 Executing action (_create_group).

5 2 1437134826.52 Executed action (_create_group).

25


Fabric Configuration Logical topology I

• Check the group has been created ok: shell> mysqlfabric group lookup_groups

• >Add the 2 servers to the group: shell> mysqlfabric group add mygroup 127.0.0.1:3306

shell> mysqlfabric group add mygroup 127.0.0.1:3307

• Checking: shell> mysqlfabric group lookup_servers mygroup

• Make note of the “server_uuid”. As that’s what we’ll need to promote the Master:

mysqlfabric group promote mygroup --slave_id 247608f4-2be8-11e5-b6a5-7c7a91bc3176

26


Fabric Configuration Logical topology II

• Checking again: shell> mysqlfabric group lookup_servers mygroup

server_uuid address status mode weight

------------------------------------ -------------- --------- ---------- ------

247608f4-2be8-11e5-b6a5-7c7a91bc3176 127.0.0.1:3306 PRIMARY READ_WRITE 1.0

fdd78d44-2bfb-11e5-b8d1-fcab64f39719 127.0.0.1:3307 SECONDARY READ_ONLY 1.0

• Now we have the configuration complete, we just need to activate the Fabric monitoring::

shell> mysqlfabric group activate mygroup

27


Fabric Configuration Logical topology III

• Lets make sure replication is working. Running this on the Master: mysql -uroot -p -P3306 -h127.0.0.1

create database ant; create table master_info (id int(5), descr varchar(30));

insert into master_info values (1,'algo de descripcion'),(2,'otra descripcion');

select * from master_info;

• And we observe the slave: shell> mysql -uroot -p -P3307 -h127.0.0.1 -e "use ant; select * from

master_info;"

• As we’ve configured the 2 instances and promoted one, replication has been activated internally.

• Now let’s see what happens when we eject the USB; shutdown the master and promote the slave.

28


Agenda

1

2

3


Installing MySQL



29

4


Safe switchover / USB eject & switchback

30


Safe Switchover Ejecting the USB

• As the key data is in the Master, we want to eject the USB with the data one way or another.

• In this example, we’ll do a controlled shutdown, where if we did an uncontrolled shutdown, Fabric would act in the same way.

• As we know we’re going to eject the USB, we first promote the Slave: shell> mysqlfabric group promote mygroup --slave_id=fdd78d44-2bfb-11e5-b8d1-

fcab64f39719

server_uuid address status mode weight

------------------------------------ -------------- --------- ---------- ------

247608f4-2be8-11e5-b6a5-7c7a91bc3176 127.0.0.1:3306 SECONDARY READ_ONLY 1.0

fdd78d44-2bfb-11e5-b8d1-fcab64f39719 127.0.0.1:3307 PRIMARY READ_WRITE 1.0

31


Safe Switchover Ejecting the USB I

• Now we can shut down the instance on the USB: shell> mysqladmin -S /tmp/mysql3306.sock -uroot -p shutdown

• The status is: shell> mysqlfabric group health mygroup

uuid is_alive status is_not_running is_not_configured io_not_running

sql_not_running io_error sql_error

------------------------------------ -------- ------- -------------- ----------------- -------------- --

------------- -------- ---------

247608f4-2be8-11e5-b6a5-7c7a91bc3176 0 FAULTY 0 0 0

0 False False

fdd78d44-2bfb-11e5-b8d1-fcab64f39719 1 PRIMARY 0 0 0

0 False False

• Now we can eject the USB with the Master datadir and my3306.cnf without any problems.

32


Safe Switchover Switchback – promoting the old Master

• Before bringing the old Master back online, let’s insert some data into the new master (-P3307 / UUID...9719):

mysql> insert into ant.master_info values (3,'some more info');

• Check: mysql -uroot -p -P3307 -h127.0.0.1 -e "use ant; select * from master_info;"

• Start the old master: mysqld_safe --defaults-file=/usr/local/mysql/my3306.cnf --user=mysql &

• Even though Fabric considers the old Master “FAULTY”, as it was a slave before we unplugged, and as soon as it starts, replication continues and keeps synchronizing:

mysql -uroot -p -P3306 -h127.0.0.1 -e “select * from ant.master_info;"

33


Safe Switchover Switchback – promoting the old Master I

• Changing the old masters’ status in Fabric: mysqlfabric server set_status 247608f4-2be8-11e5-b6a5-7c7a91bc3176 spare

• Fabric includes the instance as SPARE as it detects everything is ok with the environment. Now we can change it to SECONDARY:

mysqlfabric server set_status 247608f4-2be8-11e5-b6a5-7c7a91bc3176 secondary

• Make sure replication continues working fine: mysql -uroot -p -P3307 -h127.0.0.1 -e "insert into ant.master_info values

(4,'additional');“

mysql -uroot -p -P3306 -h127.0.0.1 -e “select * from ant.master_info;"

• And now we promote the old Master back again: mysqlfabric group promote mygroup --slave_id=247608f4-2be8-11e5-b6a5-7c7a91bc3176

34


Safe Switchover Switchback – promoting the old Master II

• Let’s check the overall status again: mysqlfabric group health mygroup

uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error

sql_error

------------------------------------ -------- --------- -------------- ----------------- --------------

--------------- -------- ---------

247608f4-2be8-11e5-b6a5-7c7a91bc3176 1 PRIMARY 0 0 0

0 False False

fdd78d44-2bfb-11e5-b8d1-fcab64f39719 1 SECONDARY 0 0 0

0 False False

35


Starting and stopping a pre-configured environment

36


Starting the pre-configured environment

• Just in case there’s any doubt, before we start, let’s make sure everything stopped correctly:

– Check the log for Fabric: /var/log/fabric.log

– Check the logs for each of the MySQL instances: • Master: /media/usb_externo/mysql/mysqld_safe.log

• Slave: /opt/mysql/fabric/3307/mysqld_safe.log

• First start the Fabric instance: mysqld_safe --defaults-file=/usr/local/mysql/my.cnf --user=mysql &

• Start Fabric monitoring: mysqlfabric manage start --daemonize

37


Starting the pre-configured environment

• Then startup the Master & Slave instances: mysqld_safe --defaults-file=/usr/local/mysql/my3306.cnf --user=mysql &


• Check the logs:

vi + /var/log/fabric.log

• Check the replication: mysqlfabric group health mygroup uuid is_alive status is_not_running is_not_configured io_not_running sql_not_running io_error

sql_error

------------------------------------ -------- --------- -------------- ----------------- --------------

--------------- -------- ---------

247608f4-2be8-11e5-b6a5-7c7a91bc3176 1 PRIMARY 0 0 0

0 False False

fdd78d44-2bfb-11e5-b8d1-fcab64f39719 1 SECONDARY 0 0 0

0 False False

38


Controlled shutdown

• Depending on how we stop the environment, and for what reasons, we can shutdown in various different ways. If we stop Fabric first, the shutdown of the 2 instances is not registered within Fabric. If we stop the instances first, upon the following startup, Fabric will require us to change the status of each before monitoring can begin.

• Stop both Master and Slave instances as usual:

mysqladmin -S /tmp/mysql3306.sock -uroot -p shutdown


39


Controlled shutdown

• Fabric: mysqlfabric manage ping


Time-To-Live: 1

mysqlfabric manage stop


Time-To-Live: 1

mysqlfabric manage ping

<urlopen error [Errno 111] Connection refused>

• Once the Fabric process is halted, we can stop the repository instance: mysqladmin -S /opt/mysql/fabric/63301/mysql.sock -uroot -p shutdown

40


Annex

41


Annex: Fabric my.cnf 63301 [mysqld_safe]

log-error =/opt/mysql/fabric/63301/mysqld_safe.log

lc-messages-dir =/usr/local/mysql/share/english/errmsg.sys

[mysqld]

performance-schema-instrument ='%=ON'

server-id =63301

port =63301

pid-file =/opt/mysql/fabric/63301/khollman-es_63301.pid

socket =/opt/mysql/fabric/63301/mysql.sock

datadir =/opt/mysql/fabric/63301/data

log-error =/opt/mysql/fabric/63301/data/mysqld.err

general-log =TRUE

innodb_buffer_pool_size =40M

innodb_file_per_table =1

innodb_log_buffer_size =8M

innodb_log_file_size =12M

innodb_flush_log_at_trx_commit =2

innodb_data_file_path =ibdata1:12M;ibdata2:12M:autoextend

innodb_open_files =150

table_open_cache =80

open_files_limit =300

query_cache_size =0

max_connections =100

user =mysql

read_buffer_size =2M

sort_buffer_size =2M

42


Annex: Fabric my3306.cnf 3306 [mysqld_safe]

log-error =/media/usb_externo/mysql/mysqld_safe.log


[mysqld]


server-id =3306

port =3306

pid-file =/media/usb_externo/mysql/khollman-es_3306.pid

socket =/tmp/mysql3306.sock

datadir =/media/usb_externo/mysql/data

log-error =/media/usb_externo/mysql/data/mysqld.err

general-log =TRUE










query_cache_size =0


user =mysql



# Replication

log-bin =fabric3306

binlog-row-image =minimal

binlog-rows-query-log-events =1

log-slave-updates =TRUE

43

gtid-mode =ON

enforce-gtid-consistency =TRUE

master-info-repository =TABLE

relay-log-info-repository =TABLE

sync_binlog =1

sync_master_info =1

slave-parallel-workers =2

slave_transaction_retries =0

binlog-checksum =CRC32

master-verify-checksum =1

slave-sql-verify-checksum =1


binlog_format =ROW

report-host =khollman-es

report-port =3306


Annex: Fabric my3307.cnf 3307 [mysqld_safe]

log-error =/opt/mysql/fabric/3307/mysqld_safe.log


[mysqld]


server-id =3307

port =3307

pid-file =/opt/mysql/fabric/3307/khollman-es_3307.pid

socket =/opt/mysql/fabric/3307/mysql.sock

datadir =/opt/mysql/fabric/3307/data

log-error =/opt/mysql/fabric/3307/data/mysqld.err

general-log =TRUE










query_cache_size =0


user =mysql



# Replication

log-bin =fabric3307

binlog-row-image =minimal


log-slave-updates =TRUE

44

gtid-mode =ON

enforce-gtid-consistency =TRUE

master-info-repository =TABLE

relay-log-info-repository =TABLE

sync_binlog =1

sync_master_info =1

slave-parallel-workers =2

slave_transaction_retries =0

binlog-checksum =CRC32

master-verify-checksum =1

slave-sql-verify-checksum =1


binlog_format =ROW

report-host =khollman-es

report-port =3307


Questions?


Next Steps

46

Learn more:

• http://www.mysql.com/products/enterprise/fabric.html

• http://dev.mysql.com/doc/mysql-utilities/1.6/en/fabric.html

Try me:

• http://dev.mysql.com/downloads/utilities/

Let us know what you think:

• http://forums.mysql.com/list.php?144

• [email protected]


Additional Resources

• MySQL High-Availability

– http://mysqlhighavailability.com

• Mats Kindahl

– http://mysqlmusings.blogspot.com

• Alfranio Correia – http://alfranio-distributed.blogspot.com

• Narayanan Venkateswaran

– http://vnwrites.blogspot.com

47

Technology

MySQL Fabric in a USB