Embed Size (px)
Citation preview
Introduction To MySQL ReplicationKenny Gryp <[email protected]>
Percona Live Washington DC / 2012-01-11
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
2
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
3
www.percona.com
What is Replication?
Replication enables data from one MySQL database server (the master) to be replicated to one or more MySQL
database servers (the slaves)(Source: http://dev.mysql.com/doc/refman/5.5/en/replication.html)
4
MASTER SLAVE
...
MyISAM
INNODB
...
MyISAM
INNODB
BINLOGS RELAY LOGSIO Thread
SQL Thread
APPLICATION
Queries
www.percona.com
Replication Diagram
5
MASTER SLAVE
...
MyISAM
INNODB
...
MyISAM
INNODB
BINLOGS RELAY LOGSIO Thread
SQL Thread
APPLICATION
Queries
BINLOGS
Queries
www.percona.com
Replication Diagram
6
www.percona.com
Replication
Happens at MySQL level, not Storage Engine Level (*NDB)Asynchronous! (Semi-sync available in 5.5)A server can have only 1 masterIO Thread: Fetches from masterSQL Thread: Executes on slaveSingle Threaded Execution (Expect enhancements in 5.6)Different schema’s are possible between master and slave (Watch out!!):
different indexesstorage enginesdata typescolumns
7
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
8
BINLOG.000001BINLOG.000002
BINLOG.000003BINLOG….
BINLOG.index
BINLOG.000001BINLOG.000002BINLOG.000003
www.percona.com
Binary Logs
9
www.percona.com
Binary Logs
Set of filesContains all writes and schema changes!= REDO/Transaction logRotated when full (Set max_binlog_size)Incrementing numbers (000001,000002,000003,...)Relay Logs are also binary logs2 Formats:
Statement Based (SBR)Row Based (RBR, since MySQL 5.1)
Binary file
10
www.percona.com
Binary Logs - Info Files
master.infoContains IO Thread information & connection information
relay.infoContains SQL Thread information
11
www.percona.com
Binary Log - Formats (binlog_format)
Statement Based Replication (SBR):Writes statements to binary logs, slave executes the stmts (An update stmt that changes 1 row but reads many will also do the same on the slave)Some functions are non-deterministic and cause inconsistencies:
UUID(), SYSDATE(), FOUND_ROWS(), UPDATE .. LIMIT without ORDER BY...Works for NOW(): timestamp included in binary log
More complete list of issues: http://dev.mysql.com/doc/refman/5.5/en/replication-features.html
12
www.percona.com
Binary Log - Formats (binlog_format)
Row Based Replication (RBR, since 5.1):Write row changes (larger binlogs)Check Binlog_cache_disk_use, possibly increase binlog_cache_sizeDoes not need to parse/execute queries, just make the changes necessaryMuch less/different issues compared to SBR: http://dev.mysql.com/doc/refman/5.5/en/replication-rbr-usage.html
Mixed: Combination of both: defaults to SBR, use RBR when necessary
13
www.percona.com
Looking at Binary Log Contents
mysqlbinlogSHOW BINLOG EVENTS
14
www.percona.com
Example SBR> SHOW GLOBAL VARIABLES LIKE 'binlog_format';+---------------+-----------+| Variable_name | Value |+---------------+-----------+| binlog_format | STATEMENT |+---------------+-----------+1 row in set (0.00 sec)
> CREATE DATABASE replication;Query OK, 1 row affected (0.14 sec)
> use replicationDatabase changed
> CREATE TABLE repl (a int) ENGINE=innodb;Query OK, 0 rows affected (0.25 sec)
> INSERT INTO repl VALUES (1); Query OK, 1 row affected (0.14 sec)
15
www.percona.com
Example SBR - mysqlbinlog# mysqlbinlog mysql-bin.000193...# at 106#120106 15:19:13 server id 9999 end_log_pos 203 ! Query! thread_id=11CREATE DATABASE replication/*!*/;# at 203#120106 15:19:32 server id 9999 end_log_pos 312 ! Query! thread_id=11! exec_time=1! error_code=0use replication/*!*/;SET TIMESTAMP=1325859572/*!*/;CREATE TABLE repl (a INT) ENGINE=innodb/*!*/;# at 312#120106 15:19:55 server id 9999 end_log_pos 387 ! Query! thread_id=11! exec_time=0! error_code=0SET TIMESTAMP=1325859595/*!*/;BEGIN/*!*/;# at 387#120106 15:19:55 server id 9999 end_log_pos 484 ! Query! thread_id=11! exec_time=0! error_code=0SET TIMESTAMP=1325859595/*!*/;INSERT INTO repl VALUES (1)/*!*/;# at 484#120106 15:19:55 server id 9999 end_log_pos 511 ! Xid = 14COMMIT/*!*/;
16
www.percona.com
Example SBR - SHOW BINLOG EVENTS> SHOW BINLOG EVENTS FROM 106\G*************************** 1. row *************************** Log_name: mysql-bin.000193 Pos: 106 Event_type: Query Server_id: 1End_log_pos: 203 Info: CREATE DATABASE replication*************************** 2. row *************************** Log_name: mysql-bin.000193 Pos: 203 Event_type: Query Server_id: 1End_log_pos: 312 Info: use `replication`; CREATE TABLE repl (a INT) ENGINE=innodb*************************** 3. row *************************** Log_name: mysql-bin.000193 Pos: 312 Event_type: Query Server_id: 1End_log_pos: 387 Info: BEGIN*************************** 4. row *************************** Log_name: mysql-bin.000193 Pos: 387 Event_type: Query Server_id: 1End_log_pos: 484 Info: use `replication`; INSERT INTO repl VALUES (1)*************************** 5. row *************************** Log_name: mysql-bin.000193 Pos: 484 Event_type: Xid Server_id: 1End_log_pos: 511 Info: COMMIT /* xid=14 */5 rows in set (0.00 sec) 17
www.percona.com
Example RBR> SHOW GLOBAL VARIABLES LIKE 'binlog_format';+---------------+-------+| Variable_name | Value |+---------------+-------+| binlog_format | ROW |+---------------+-------+1 row in set (0.00 sec)
> CREATE DATABASE replication;Query OK, 1 row affected (0.14 sec)
> use replicationDatabase changed
> CREATE TABLE repl (a int) ENGINE=innodb;Query OK, 0 rows affected (0.25 sec)
> INSERT INTO repl VALUES (1); Query OK, 1 row affected (0.14 sec)
18
www.percona.com
Example RBR - mysqlbinlog# mysqlbinlog mysql-bin.000193 --start-position=606...# at 606#120106 15:54:54 server id 1 end_log_pos 703 ! Query! thread_id=11! exec_time=0! error_code=0CREATE DATABASE replication/*!*/;# at 703#120106 15:55:02 server id 1 end_log_pos 812 ! Query! thread_id=11! exec_time=0! error_code=0use replication/*!*/;SET TIMESTAMP=1325861702/*!*/;CREATE TABLE repl (a int) ENGINE=innodb/*!*/;# at 812...# at 937#120106 15:55:06 server id 1 end_log_pos 937 ! Table_map: `replication`.`repl` mapped to number 17#120106 15:55:06 server id 1 end_log_pos 971 ! Write_rows: table id 17 flags: STMT_END_F
BINLOG 'SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAAE=SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA=='/*!*/;# at 971#120106 15:55:06 server id 1 end_log_pos 998 ! Xid = 34COMMIT/*!*/;
19
www.percona.com
Example RBR - mysqlbinlog --verbose
# mysqlbinlog mysql-bin.000193 --verbose --verbose...# at 937#120106 15:55:06 server id 1 end_log_pos 937 ! Table_map: `replication`.`repl` mapped to number 17#120106 15:55:06 server id 1 end_log_pos 971 ! Write_rows: table id 17 flags: STMT_END_F
BINLOG 'SgsHTxMBAAAAMgAAAKkDAAAAABEAAAAAAAEAC3JlcGxpY2F0aW9uAARyZXBsAAEDAAE=SgsHTxcBAAAAIgAAAMsDAAAAABEAAAAAAAEAAf/+AQAAAA=='/*!*/;### INSERT INTO replication.repl### SET### @1=1 /* INT meta=0 nullable=1 is_null=0 */# at 971
20
www.percona.com
Example RBR - SHOW BINLOG EVENTS> SHOW BINLOG EVENTS FROM 606\G*************************** 1. row *************************** Log_name: mysql-bin.000193 Pos: 606 Event_type: Query Server_id: 1End_log_pos: 703 Info: CREATE DATABASE replication*************************** 2. row *************************** Log_name: mysql-bin.000193 Pos: 703 Event_type: Query Server_id: 1End_log_pos: 812 Info: use `replication`; CREATE TABLE repl (a int) ENGINE=innodb...*************************** 4. row *************************** Log_name: mysql-bin.000193 Pos: 887 Event_type: Table_map Server_id: 1End_log_pos: 937 Info: table_id: 17 (replication.repl)*************************** 5. row *************************** Log_name: mysql-bin.000193 Pos: 937 Event_type: Write_rows Server_id: 1End_log_pos: 971 Info: table_id: 17 flags: STMT_END_F*************************** 6. row *************************** Log_name: mysql-bin.000193 Pos: 971 Event_type: Xid Server_id: 1End_log_pos: 998 Info: COMMIT /* xid=34 */6 rows in set (0.00 sec)
21
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
22
www.percona.com
Setting up Replication
PrerequisitesChange master/slave MySQL configurationConfigure ReplicationStart Replication/Check Status
23
www.percona.com
Prerequisites
No way to easily create slave with 1 commandIt’s required to Create/Restore consistent backup using your favorite backup tool
with mysqldump, use --master-dataBinlog file and position from backup should be recorded
File: mysql-bin.000008Pos: 106
24
www.percona.com
Change master/slave Configuration
On the master:Enable Binary Logging: log-bin=log-binSet A Server ID: server-id=1
On the slave:Set A Server ID: server-id=2
25
www.percona.com
Why server-id ?
26
MASTER SLAVE
...
MyISAM
INNODB
...
MyISAM
INNODB
BINLOGS RELAY LOGSIO Thread
SQL Thread
APPLICATION
Queries
BINLOGS
Queries
server-id=1 server-id=2
events get server-id=1
server-id != 2
MASTER SLAVE
...
MyISAM
INNODB
...
MyISAM
INNODB
BINLOGS RELAY LOGSIO Thread
SQL Thread
APPLICATION
Queries
BINLOGS
Queries
server-id=1 server-id=2
events get server-id=1
server-id != 2
IO ThreadSQL Thread RELAY LOGS
events get server-id=2server-id != 1
log_slave_updateslog_slave_updates
www.percona.com
Why server-id ?
27
www.percona.com
Why server-id ?
Avoid events to be written more than oncereplicate_same_server_id does what it says
28
www.percona.com
Configure Replication
On Master, add permissions:> GRANT REPLICATION SLAVE ON *.* TO ‘repl’@‘slave’ IDENTIFIED BY ‘pass’;On Slave, configure replication:> CHANGE MASTER TO MASTER_HOST=‘master’, MASTER_USER=‘repl’, MASTER_PASSWORD=‘pass’, MASTER_LOG_FILE=‘mysql-bin.000008’, MASTER_LOG_POS=106;
29
www.percona.com
Start Replicationslave> START SLAVE;slave> SHOW SLAVE STATUS\G Slave_IO_State: Waiting for master to send event Master_Host: master Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000008 Read_Master_Log_Pos: 254 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 399 Relay_Master_Log_File: mysql-bin.000008 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 254 Relay_Log_Space: 566 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error:
Threads Running?
IO Thread Read Up To
SQL Thread Read Up To
How Far Is Slave Behind?
IO Thread State
30
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
31
www.percona.com
Commands
SQL CommandsAdminstrative Commands Diagnostics Commands
Shell Commandsmysqlbinlog
32
www.percona.com
Administrative Commands
Rotate binary log: FLUSH BINARY LOGSRotate relay log: FLUSH RELAY LOGSSTART/STOP SLAVE IO_THREAD/SQL_THREADRemove binary logs: PURGE MASTER LOGS TO ‘mysql-bin.000005’;PURGE MASTER LOGS BEFORE ‘2012-01-01 00:00:00’;Remove all binary logs: RESET MASTERRemove slave configuration and files: RESET SLAVE
33
www.percona.com
Diagnostics Commands
On MasterSHOW MASTER STATUSSHOW PROCESSLISTSHOW SLAVE HOSTSSHOW BINLOG EVENTS
On SlaveSHOW SLAVE STATUSSHOW PROCESSLIST
34
www.percona.com
On Master:SHOW MASTER STATUS
Current binary log file and position:master> SHOW MASTER STATUS\G*************************** 1. row *************************** File: mysql-bin.000008 Position: 254 Binlog_Do_DB: Binlog_Ignore_DB: 1 row in set (0.00 sec)
35
www.percona.com
On Master: SHOW PROCESSLIST
Find Connected Slaves using SHOW PROCESSLIST:master> SHOW PROCESSLIST\G...*************************** 2. row *************************** Id: 4 User: repl Host: localhost:43537 db: NULLCommand: Binlog Dump Time: 1264 State: Has sent all binlog to slave; waiting for binlog to be updated Info: NULL...
36
www.percona.com
On Master: SHOW SLAVE HOSTS
Shows connected slaves with their server-id:master> SHOW SLAVE HOSTS;+-----------+-----------+------+-----------+| Server_id | Host | Port | Master_id |+-----------+-----------+------+-----------+| 2 | slave | 3306 | 1 |+-----------+-----------+------+-----------+1 row in set (0.00 sec)
Only works when slaves have configured:report-host, report-user, report-password, report-port
37
www.percona.com
On Slave: SHOW PROCESSLIST
Slave Thread Status:SLAVE> SHOW PROCESSLIST\G*************************** 1. row *************************** Id: 5 User: system user Host: db: NULLCommand: Connect Time: 88611 State: Waiting for master to send event Info: NULL*************************** 2. row *************************** Id: 8 User: system user Host: db: NULLCommand: Connect Time: 83 State: Has read all relay log; waiting for the slave I/O thread to update it Info: NULL
38
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
39
www.percona.com
Other Common Configuration Options
Filtering: binlog-%, replicate-%Don’t start replication at start: skip_slave_startPut relay log events in it’s own binary log: log_slave_updatesDisallow writing on slaves: read_onlyAutomatically remove binlogs: expire_logs_daysChange binlog format: binlog_format
40
www.percona.com
Filtering on Master
binlog-do-db=..., binlog-ignore-db=...Warning! Different behavior between SBR and RBR:
SBR: Log all statements executed if the ‘default database’ (use database) should be binloggedRBR: Log all row changes of the databases that should be binlogged
41
www.percona.com
Filtering on Slave
replicate-do-db= & replicate-ignore-db=Warning! Different behavior between SBR and RBRSimilar to binlog-do/ignore-dbreplicate-do/ignore-table: Filter on table levelreplicate-wild-do/ignore-table: Use wildcards (%,_) to filter on table levelreplicate-rewrite-db=db1->db2
42
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
43
www.percona.com
Replication Architectures
Master-SlaveMaster-MasterMulti Tiered ReplicationCircular Replication
44
Master
Slave
www.percona.com
Master-Slave
45
Master
Slave
www.percona.com
Master-Slave
Application
45
Master
Slave
www.percona.com
Master-Slave
Application
45
Master
Slave
www.percona.com
Master-Slave
ApplicationWrites/Reads
45
Master
Slave
www.percona.com
Master-Slave
ApplicationWrites/Reads
45
Master
Slave
www.percona.com
Master-Slave
Application
Reads
Writes/Reads
45
www.percona.com
Master-Slave
Master
Slave SlaveSlave
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application Writes/Reads
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application Writes/Reads
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application
Reads
Writes/Reads
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application
Reads
Writes/Reads
46
www.percona.com
Master-Slave
Master
Slave SlaveSlave
Application
Reads
Writes/Reads
46
www.percona.com
Master-Slave
Scaling Reads:Slaves can take reads (remember asynchronous!)Reporting/Search queries can go to separate slave
Does not take resources on master ‘live-site’-slave. (‘live site’: lower response time requirements than reports)Can have other indexes, even other tables (summary tables,...)
Not much benefit when too much writesMaintenance (schema changes...)
Upgrade slaves first:Take slave offlineSET SQL_LOG_BIN=0; <-don’t put this session queries inALTER TABLE ...; the binary logSET SQL_LOG_BIN=1;
47
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application Writes/Reads
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application Writes/Reads
Reads
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application Writes/Reads
Reads
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
Application Writes/Reads
Reads
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
ApplicationReporting
Writes/Reads
Reads
48
www.percona.com
Master-Slave: Reporting Queries
Master
Slave SlaveSlave
ApplicationReporting
Writes/Reads
Reads
48
www.percona.com
Master-Slave: Write Intensive
Capacity left for READS
Capacity taken by WRITES
SlavesMaster
Capacity left for READS
Capacity taken by WRITES
SlavesMaster
49
Master Master
www.percona.com
Master-Master
50
www.percona.com
Master-Master
Used for:High AvailabilityMaintenance Tasks
Write to 1 master, failover to the passive master when necessaryPassive master is a real slave, can be used for readsScale Writes? NO!
Every write still has to be performed on both mastersWriting to both masters at the same time? NO*!
Can cause inconsistencieseven with auto_increment_increment/auto_increment_offset problems can happen
51
Master Master
www.percona.com
Master-Master: Reads & Writes
Application
Reads
Writes/Reads
52
Master Master
www.percona.com
Master-Master: Maintenance
Application
Writes/Reads
53
Master Master
www.percona.com
Master-Master: Maintenance
Application
SET SQL_LOG_BIN=0;ALTER TABLE ...;...SET SQL_LOG_BIN=1;
Writes/Reads
53
Master Master
www.percona.com
Master-Master: Maintenance
Application
Writes/Reads
53
Master Master
www.percona.com
Master-Master: Maintenance
Application
Writes/Reads
SET SQL_LOG_BIN=0;ALTER TABLE ...;...SET SQL_LOG_BIN=1;
53
Master Master
www.percona.com
Master-Master: Write to Both?Inconsistencies
Application
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
54
Master Master
www.percona.com
Master-Master: Write to Both?Inconsistencies
ApplicationUPDATE users SET
email=‘[email protected]’ where userid=1;
UPDATE users SET email=‘[email protected]’ where userid=1;
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
54
Master Master
www.percona.com
Master-Master: Write to Both?Inconsistencies
ApplicationUPDATE users SET
email=‘[email protected]’ where userid=1;
UPDATE users SET email=‘[email protected]’ where userid=1;
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+------------------+| userid | email |+--------+------------------+| 1 | [email protected] |+--------+------------------+
+--------+-------------------+| userid | email |+--------+-------------------+| 1 | [email protected] |+--------+-------------------+
54
Master Master
www.percona.com
Master-Master: Write to Both?Inconsistencies
Application
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
UPDATE users SET email=‘[email protected]’ where userid=1;
UPDATE users SET email=‘[email protected]’ where userid=1;
+--------+------------------+| userid | email |+--------+------------------+| 1 | [email protected] |+--------+------------------+
+--------+-------------------+| userid | email |+--------+-------------------+| 1 | [email protected] |+--------+-------------------+
54
Master Master
www.percona.com
Master-Master: Write to Both?Inconsistencies
Application
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
UPDATE users SET email=‘[email protected]’ where userid=1;
UPDATE users SET email=‘[email protected]’ where userid=1;
+--------+------------------+| userid | email |+--------+------------------+| 1 | [email protected] |+--------+------------------+
+--------+-------------------+| userid | email |+--------+-------------------+| 1 | [email protected] |+--------+-------------------+
54
Master Master
www.percona.com
Master-Master: Write to Both?Replication Breaks
Application
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
55
Master Master
www.percona.com
Master-Master: Write to Both?Replication Breaks
ApplicationINSERT INTO USERS
(email) VALUES (‘[email protected]’)
UPDATE USERS SET email=‘[email protected]’ WHERE userid=1;
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
+--------+----------------+| userid | email |+--------+----------------+| 1 | [email protected] |+--------+----------------+
55
Master Master
www.percona.com
Master-Master: Write to Both?Replication Breaks
ApplicationINSERT INTO USERS
(email) VALUES (‘[email protected]’)
UPDATE USERS SET email=‘[email protected]’ WHERE userid=1;
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
+--------+--------------------+| userid | email |+--------+--------------------+| 1 | [email protected] || 2 | [email protected] |+--------+--------------------+
+--------+--------------------+| userid | email |+--------+--------------------+| 1 | [email protected] |+--------+--------------------+
55
Master Master
www.percona.com
Master-Master: Write to Both?Replication Breaks
Application
CREATE TABLE users (userid INT AUTO_INCREMENT,email VARCHAR(128) NOT NULL,PRIMARY KEY (userid),UNIQUE KEY idx_email (email));
INSERT INTO USERS (email) VALUES (‘[email protected]’)
UPDATE USERS SET email=‘[email protected]’ WHERE userid=1;
+--------+--------------------+| userid | email |+--------+--------------------+| 1 | [email protected] || 2 | [email protected] |+--------+--------------------+
+--------+--------------------+| userid | email |+--------+--------------------+| 1 | [email protected] |+--------+--------------------+
Error 'Duplicate entry '[email protected]' for key 'idx_email'' on query. Default database: 'test'. Query: 'insert into users(email) values ('[email protected]')'
55
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication
56
www.percona.com
Multi Tiered Replication
2 Second query takes 6 seconds to reach the 3rd levelTop level slaves could be a master for some dbs/tables too
Replication main data (users...) to top level slavesTop level slaves are sort of functionally partitioned
57
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication: Delay
58
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication: Delay
2 Seconds
58
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication: Delay
2 Seconds2 Seconds
58
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication: Delay
2 Seconds2 Seconds
2 Seconds
58
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication: Top Level Slaves
MASTER
LOGSEARCH
SEARCHSEARCH
MASTER
59
Master
Slave SlaveSlave
Slave Slave
www.percona.com
Multi Tiered Replication
No Updates Anymore
60
Master
Master
MasterMaster
www.percona.com
Circular Replication
61
Master
Master
MasterMaster
www.percona.com
Circular Replication
62
Master
Master
MasterMaster
www.percona.com
Circular Replication
63
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
64
www.percona.com
Common Issues
Replication LagReplication BreakingCrash Safe ReplicationConsistency
65
www.percona.com
Replication Lag - Causes
Single threaded execution on slave!!!Long running queries with SBR: will take ~same time on slaveMany row changes with RBRMyISAM & table level locking: Long running SELECT on slave might block replicationSchema changes:
20min on master,20min on slave <-- 20 min replication lag
66
www.percona.com
Replication Lag - Tips 1
Optimize DML StatementsAvoid DDL Statements in Replication (SQL_LOG_BIN=0)Hardware specifications usually need to be equal or better on slaveMySQL 5.6: http://d2-systems.blogspot.com/2011/04/mysql-56x-feature-preview-multi.html
Ensure BBU&Write Back Cache is used or lower durability: innodb_flush_log_at_trx_commit=2Change architecture: functional/horizontal partitioning
67
www.percona.com
Replication Lag - Tips 2
Try to avoid SQL Thread to need to do disk I/O:Replication Booster (Yoshinori Matsunobu)
prefetches relay logs and converts queries to select and runs them, causing data to be cached before SQL thread reaches
innodb_fake_changeshttp://www.percona.com/doc/percona-server/5.5/management/innodb_fake_changes.htmlhttp://dom.as/2011/12/03/replication-prefetching/
68
www.percona.com
Replication Booster
http://yoshinorimatsunobu.blogspot.com/2011/10/making-slave-pre-fetching-work-better.html69
www.percona.com
Replication Breaking - ExamplesLast_Errno: 1146Last_Error: Error 'Table 'db.table' doesn't exist' on query. Default database: 'userdb'. Query: '...'Last_Errno: 1062Last_Error: Error 'Duplicate entry '1748099' for key 1' on query. Default database: 'db'. Query: 'INSERT INTO table (url)VALUES(‘http://www.google.com')'Last_Errno: 1053Last_Error: Query partially completed on the master (error on master: 1053) and was aborted. There is a chance that your master is inconsistent at this point. If you are sure that your master is ok, run this query manually on the slave and then restart the slave with SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE; . Query: ALTER TABLE ‘users’ ADD ‘name’ VARCHAR(128) NULL DEFAULT;
70
www.percona.com
Replication Breaking - Causes
Writes happening on SlavesSBR with bad statementsCrashes: Relay Log CorruptionDifferent schemasTemporary tablesMixing MyISAM with InnoDBDifferent behavior between MySQL versions (minor/major)Killing queries that change MyISAM tables will be partially executed, and break replication...
71
www.percona.com
Replication Breaking - Fixing
Questions to ask yourself: Why? Where? Impact?Quick ‘fix’: SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
Causes 1 event to be skippedCauses inconsistencies!!!
Look at:Check error & Investigate data on master/slaveMySQL errorlogsSHOW SLAVE STATUS\Gmysqlbinlog on relaylogs/binarylogs and investigatept-slave-restart
More On: http://www.percona.com/files/presentations/percona-live/london-2011/PLUK2011-diagnosing-and-fixing-mysql-replication.pdf
72
www.percona.com
Crash Safe Replication
On master: sync_binlog=1 to fsync at every binlog writePerformance impact, mainly on:
Ext3 filesystem http://www.mysqlperformanceblog.com/2009/01/21/beware-ext3-and-sync-binlog-do-not-play-well-together/
High write-response time disksubsystem (no WriteBackCache& BBU)On slave:
relay-log.info/master.info: not durable/consistent!When OS crash, old information might be in the fileIf InnoDB only: use innodb_overwrite_relay_log_infoin Percona Server: http://www.percona.com/doc/percona-server/5.5/reliability/innodb_recovery_update_relay_log.html
73
www.percona.com
Consistency
Replication can cause inconsistencies:Statement Based Replication
Certain functions like UUID()...UPDATE/DELETE with LIMIT without ORDER BY unique key
Writing to multiple mastersWriting to slaves (by accident? use read_only)Failover done wrongMyISAM with killing queries and/ord MySQL CrashesWrongly restored slavesReplication broke and used SQL_SLAVE_SKIP_COUNTER...
How to know if my data is consistent?
74
www.percona.com
Consistency
Checking Consistency: pt-table-checksumChecksums chunks of data on master and slavesMonitors replication lagLow checksum response time to lower impact on productionUses Replication to ensure consistent view of data
Fixing Consistency: pt-table-syncCan checksum and use pt-table-checksum resultsFixes inconsistencies
75
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
76
www.percona.com
Replication Use Cases
Scale ReadsHigh AvailabilityBackups
77
www.percona.com
High Availability
Failover:Virtual IPsChange IP in application servers
Warning: Split-Brain possible!!!Ensure no writes happen on old master before movingMake sure new master is in sync
Scripts available to automate failover:MySQL-MHA http://code.google.com/p/mysql-master-ha/ MMM (not-recommended) http://mysql-mmm.org/
Percona-PRMhttp://www.mysqlperformanceblog.com/2011/11/29/percona-replication-manager-a-solution-for-mysql-high-availability-with-replication-using-pacemaker/
78
www.percona.com
High Availability
Due to asynchronous’ness, data might be lost:Semi-Synchronous Replication:
ensure at least one slave’s IO thread has received the event before the transaction is committed.http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
79
www.percona.com
Backups
Point In Time Recovery, using binary logsDelayed slaveBackup Slaves
80
www.percona.com
Incremental / Point In Time Recovery
Restore Full Backup from your favorite backup toolReplay binary logs with mysqldump from the last position of the backup# mysqlbinlog --start-position=15115 1sbinarylog | mysql# mysqlbinlog next-binary-logs | mysql
If PITR, stop at the desired position/time:# mysqlbinlog --stop-position=102151 binarylog | mysql# mysqlbinlog --stop-datetime=“2012-01-01 23:00:00” \ lastbinarylog | mysql
81
www.percona.com
Delayed Slave
Deliberately lag a slave for a predefined amount of timeDecrease recovery time in case of bad changes caused by:
Humans (DROP DATABASE)Application (DELETE FROM table;)
How to configure:pt-slave-delay --delay 60m(http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-delay.html)
5.6 Built-in feature:> CHANGE MASTER TO MASTER_DELAY=3600;
When a problem happens stop replication just before the malicious statement.START SLAVE UNTIL MASTER_LOG_FILE='log_name', MASTER_LOG_POS = pos;
82
www.percona.com
Backup using Slaves
Online Backups might not be possible on active master:Too much impact to make online:
InnoDB backup (using MySQL enterprise backup or XtraBackup)LVM Snapshots MyISAM tables
Offsite Backup
83
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
84
www.percona.com
Other Replication Implementations
Tungsten ReplicatorUses binary logParallel replicationEnhanced filteringMulti-masterReplication between different databases
Galera ClusterDoes not use binary logSynchronous replicationActive-active masterParallel replicationPercona XtraDB Cluster! http://www.mysqlperformanceblog.com/2012/01/09/announcement-of-percona-xtradb-cluster-alpha-release/
85
www.percona.com
Tungsten Replicator
http://continuent.com/http://code.google.com/p/tungsten-replicator/
86
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
88
www.percona.com
Tools & More Info
Percona Toolkit http://www.percona.com/software/percona-toolkit/pt-slave-find, pt-table-checksum, pt-table-sync, pt-slave-delay, pt-slave-restart, pt-heartbeat
Percona Server http://www.percona.com/software/percona-server/
OpenArk kit http://code.openark.org/forge/openark-kit
Replication Booster https://github.com/yoshinorim/replication-booster-for-mysql
High AvailabilityMMM http://mysql-mmm.org/Percona-PRMhttp://www.mysqlperformanceblog.com/2011/11/29/percona-replication-manager-a-solution-for-mysql-high-availability-with-replication-using-pacemaker/MySQL-MHA http://code.google.com/p/mysql-master-ha/
High Performance MySQL, 2nd Edition: http://shop.oreilly.com/product/9780596101718.do
89
www.percona.com
MySQL Replication
Replication OverviewBinary LogsSetting Up ReplicationCommandsOther Common Configuration SettingsReplication ArchitecturesCommon IssuesReplication Use CasesOther Replication ImplementationsTools
90
