Download pdf - Repair Replication Mysql Duplication Error

8/9/2019 Repair Replication Mysql Duplication Error

1/15

Repair Replication

BackgroundMySQL Replication is asynch ronous and one-way. (Even master-master r eplication is 2 one-wayrelationships, not a single two-way rel ationship.)

The slave server copies e vents f rom the master's b inary log into its o wn relay log, then appliesevents f rom its r elay log to its o wn live database. There is n o feedback m echanism from the slaveto the master if applying events t o the slave's d atabase causes a n unrecoverable error; the slavestops r eplicating and waits f or help from an administrator.

This procedure will solve a data corruption problem that halts a master-master replication.

Specically, we'll handle a primary key collision. The table important.stuff has atypical AUTO_INCREMENTprimary ke y. Both servers i n this m aster-master replication acceptINSERTs t o the s ame t able. Because MySQL replication is a synchronous, it is p ossible for bothmasters t o write a new row that uses t he same p rimary key. When these new rows r eplicate, thereceiving master will have a duplicate key on a column that must be unique, and replication willstop in both directions.

There is no substitute for

experience.You replicate data because it's i mportant. Treat your important data right, and get concreteexperience with replication in our hands-on course.

What's in the c ourse?

Steps

1. Test Replication from A to B

At the beginning of this lesson, replication should be working correctly.

Log in to a.example.com

From A, insert a new record in the important.stuff table.

a ~ $ mysql -u root

mysql> IN ERT INTO important.stuff ET !etails"#$ift from A to %#&
http://mysql.wingtiplabs.com/documentation/3gescaeg/establishing-master-master-replicationhttp://mysql.wingtiplabs.com/documentation/3gescaeg/establishing-master-master-replicationhttp://mysql.wingtiplabs.com/courses/replication.phphttp://mysql.wingtiplabs.com/courses/replication.phphttp://mysql.wingtiplabs.com/documentation/3gescaeg/establishing-master-master-replication


2/15

Query OK, 1 row affected (0.03 sec)

Log in to '.example.com

Now on B, check that the record has replicated.

b ~ $ mysql -u rootmysql> E(ECT ) *ROM important.stuff OR+ER %, i! +E C(IMIT $

*************************** 1. row***************************

d! "

deta ls! # ft from to %

&a''e ed! 013 03 " 0+!0 !+0

1 rows set (0.00 sec)

After this step is completed:

1. INSERT a new record on A in the table important.stuff

2. Within 1 sec ond, that record is replicated to B

2. Test Replication from the B to A

Now test that changes t o B a lso replicate to A.

Still on B, insert a new record in the important.stuff t able.

mysql> IN ERT INTO important.stuff ET !etails"#$ift from% to A#&


mysql>

Now on A, check that the record has replicated.

mysql> E(ECT ) *ROM important.stuff OR+ER %, i! +E C(IMIT $

*************************** 1. row***************************

d!

deta ls! # ft from % to &a''e ed! 013 03 " 0+!0 !+0


3/15

1 rows set (0.00 sec)


1. INSERT a new record on B in the table important.stuff

2. Within 1 sec ond, that record is replicated to A

3. Pause Replication

In this step, we're going to force a collision by slowing down replication.

MySQL replication uses t wo threads on the slave machine. The I/O Thread c opies binary logdata from the master to the relay log on the slave. Then the SQL Thread applies changes

from the relay log to the slave database.

Stopping the s lave I/O thread is n o d ifferent than h aving a s hort network o utage between thetwo servers. Both servers a re healthy, but they're unable to tell each other about changes.Because MySQL replication is a synchronous, neither server stops a ccepting client traffic.

On both se rvers, stop the I/O thread.

mysql> TO/ (A0E IO_T1REA+&

Query OK, 0 rows affected (0.01 sec)

mysql> s2o3 sla4e status $

*************************** 1. row***************************

-la e/ O/-tate!

aster/2ost! a.e am'le.com

aster/4ser! re'l cator

aster/5ort! 3306

7o ect/8etry! 60

aster/9o:/; le! mysql b .000001

8ead/ aster/9o:/5os! 3+1


5/15


1. 1O5 (A0E TATU on A shows la4e_IO_Runnin6 is No

2. 1O5 (A0E TATU on B shows la4e_IO_Runnin6 is No

4. Causing a Primary Key Collision

Insert a n ew row on A

mysql> IN ERT INTO important.stuff ET !etails"#Unique!ata from A#&


Insert something complicated on B

mysql> IN ERT INTO important.stuff ET !etails"#Unique!ata from %#&


Restart the replication I/O Thread on both se rvers.

mysql> TART (A0E&


Now check r eplication status o n both se rvers. The INSERTs h ave caused replication to halt.


*************************** 1. row***************************

-la e/ O/-tate! Ba t : for master to se de e t

aster/2ost! b.e am'le.com

aster/4ser! re'l cator aster/5ort! 3306

7o ect/8etry! 60


8ead/ aster/9o:/5os! +1 "

8elay/9o:/; le! mysql relay b .000003

8elay/9o:/5os!


6/15

-la e/ O/8u :! ?es

-la e/-Q9/8u :! o

8e'l cate/@o/@%!

8e'l cate/ : ore/@%! 8e'l cate/@o/Aable!

8e'l cate/ : ore/Aable!

8e'l cate/B ld/@o/Aable!

8e'l cate/B ld/ : ore/Aable!

9ast/Crr o! 106

9ast/Crror! Crror F@u'l cate e try F11F

for Dey F58 8?FF o query. @efault database! FF. Query!F -C8A AO m'orta t.stuff -CA deta lsGF4 que data from%FF

-D '/7ou ter! 0

C ec/ aster/9o:/5os! 3==+

8elay/9o:/-'ace! 1


7/15

8e'l cate/ : ore/-er er/ ds!

aster/-er er/ d! 0

1 row set (0.00 sec)

mysql>


1. 1O5 (A0E TATU on A shows la4e_IO_Runnin6 is ,es

2. 1O5 (A0E TATU on A shows la4e_ 7(_Runnin6 is No

3. 1O5 (A0E TATU on A shows (ast_Errno is 89:

4. 1O5 (A0E TATU on B shows la4e_IO_Runnin6 is ,es

5. 1O5 (A0E TATU on B shows la4e_ 7(_Runnin6 is No

6. 1O5 (A0E TATU on B shows (ast_Errno is 89:

5. Update Auto Increment for Master-Master Replication

MySQL has two s ettings t hat can h elp m aster-master pairs avo id d uplicate a uto-incrementvalues.

auto_increment_increment tells MySQL how much to increase t henext AUTO_INCREMENT va lue. The default increment is 1 : the next value is t he last value+1. Changing the increment to 2 will make M ySQL use every-other value ( e.g., 1,3,5,7instead of 1,2,3,4).

auto_increment_offset tells MySQL where to start numbering AUTO_INCREMENTvalues. The default offset is 1 .

The formula for what AUTO_INCREMENT value to assi gn the Nth rowis auto_increment_offset ;


8/15

>etc>my.cnf

HmysqldI

auto_increment_increment " :

auto_increment_offset "

relay/lo: G mysql relay b

lo:/sla e/u'dates G 1

lo:/b G mysql b

ser er/ d G 10

......some co te t ot s&ow

Now restart MySQL:

a ~ $ su!o ser4ice mysql! restart

-to'' : mysqld!H OK I

-tart : mysqld!H OK I

On ser ver B, edit >etc>my.cnf

mysql> exit

b ~ $ su!oe!it >etc>my.cnf

Add these two lines t o directly u nder ?mysql!@

>etc>my.cnf

HmysqldI

auto_increment_increment " :auto_increment_offset " :

relay/lo: G mysql relay b

lo:/sla e/u'dates G 1

lo:/b G mysql b

ser er/ d G 0


Now restart MySQL:


9/15

b ~ $ su!o ser4ice mysql! restart

-to'' : mysqld!H OK I

-tart : mysqld!H OK I


1. On A >etc>my.cnf se ts auto_increment_increment to :

2. On A >etc>my.cnf se ts auto_increment_offset to

3. On A the MySQL variable auto_increment_increment has value :

4. On A the MySQL variable auto_increment_offset has value

5. On B >etc>my.cnf se ts auto_increment_increment to :

6. On B >etc>my.cnf se ts auto_increment_offset to :

7. On B the MySQL variable auto_increment_increment has value :

8. On B the MySQL variable auto_increment_offset has value

6. Clean Up Conicts on B

The new AUTO_INCREMENT settings will help prevent this pr oblem with future INSERTs,but it won't x the rows that are blocking replication right now.

Look at the (ast_Error in the slave status t o see the id of the row causing the problem:

b ~ $ mysql -u root


*************************** 1. row***************************




8e'l cate/B ld/ : ore/Aable!

9ast/Crr o! 106


10/15

9ast/Crror! Crror F@u'l cate e try F11Ffor Dey F58 8?FF o query. @efault database! FF. Query!F -C8A AO m'orta t.stuff -CA deta lsGF4 que data from%FF

-D '/7ou ter! 0 C ec/ aster/9o:/5os! 3==


Note the value of the duplicate entry. (It's 1 1 in this e xample.) Look up that record:

mysql> select ) from important.stuff 32ere i!" &

J J J J

d deta ls &a''e ed

J J J J

11 4 que data from % 013 0+ 06 1 !33!++

J J J J


mysql>

Now delete the row with that ID, only from B. It's important that the act of deleting the row

never replicate to A, or it will delete the unique data with the same id. By t urning off binarylogging, this c hange will apply t o the local database, but won't be seen by the I/O threadon A.

mysql> ET sql_lo6_'in" O** &


mysql> s2o3 4aria'les liBe sql_lo6_'in &

J J J Ear able/ ame Ealue

J J J

sql/lo:/b O;;

J J J


mysql> !elete from important.stuff 32ere i!" &


11/15


Now turn binary logging back o n, then re-INSERT the row. We want this ve rsion of theINSERT to replicate to A (and of course everything after this.) Providing a NULL value for theid will let us m ake sure t he new auto-increment settings w ork.

mysql> ET sql_lo6_'in" ON &


mysql> insert into important.stuff 0A(UE select ) from important.stuff or!er 'y i! +E C

(IMIT &J J J J

d deta ls &a''e ed

J J J J

1 4 que data from % 013 0+ 06 1 !33!++

J J J J


mysql>

Note the new id. It should be higher than the old value and, because it was i nserted on B, itshould be eve n.

Start replication on B.

mysql> sla4e start&



*************************** 1. row***************************



aster/4ser! re'l cator

aster/5ort! 3306


12/15

7o ect/8etry! 60


8ead/ aster/9o:/5os! 10"

8elay/9o:/; le! mysql relay b .000006 8elay/9o:/5os!


13/15

9ast/-Q9/Crr o! 0

9ast/-Q9/Crror!

8e'l cate/ : ore/-er er/ ds!

aster/-er er/ d! 101 row set (0.00 sec)

The formerly c onicting row from A no w happily co exists w ith the new row on B.

mysql> select ) from important.stuff 32ere i! H" &

J J J J

d deta ls &a''e ed

J J J J

11 4 que data from 013 0+ 06 1 !33!++

1 4 que data from % 013 0+ 06 1 !33!++

J J J J

rows set (0.00 sec)

mysql>


1. 1O5 (A0E TATU on B shows la4e_IO_Runnin6 is ,es

2. 1O5 (A0E TATU on B shows la4e_ 7(_Runnin6 is ,es

3. 1O5 (A0E TATU on B shows (ast_Errno is 8

7. Clean Up Conicts on A

on A, we still need to skip the insert from B with the bad index.

a ~ $ mysql -u root

mysql> ET $(O%A( sql_sla4e_sBip_counter " &


mysql>

Now start replication o n A.

mysql> sla4e start&


14/15


mysql>

You sh ould se e the row from B with the new id.mysql> select ) from important.stuff 32ere i! H" &

J J J J

d deta ls &a''e ed

J J J J

11 4 que data from 013 0+ 06 1 !33!++

1 4 que data from % 013 0+ 06 1 !33!++

J J J J

rows set (0.00 sec)

mysql>


1. 1O5 (A0E TATU on A shows la4e_IO_Runnin6 is ,es

2. 1O5 (A0E TATU on A shows la4e_ 7(_Runnin6 is ,es

3. 1O5 (A0E TATU on A shows (ast_Errno is 8

8. Test that Normal Replication has R esumed

Re-test normal replication, following the instructions f rom step 1 and step 2 .



2. Within 1 sec ond, that record is replicated to B


4. Within 1 sec ond, that record is replicated to A

9. Test that Delayed Replication Can't Cause Duplicate Keys
http://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#test-replication-from-a-to-bhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#test-replication-from-b-to-ahttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#test-replication-from-b-to-ahttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#test-replication-from-a-to-bhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#test-replication-from-b-to-a


15/15

Re-test paused replication, following the instructions f rom step 3 and step 4 .

This t ime, the changes w e made in step 5 will prevent a duplicate key, and replicationwill not h alt.


1. On A, stop the replication I/O Thread.

2. On B, stop the replication I/O Thread.



5. On A, start the replication I/O Thread.

6. On B, start the replication I/O Thread.

7. New record replicates from A to B

8. New record replicates from B to A

9. On A, 1O5 (A0E TATU shows (ast_Errno is 8

10. On B, 1O5 (A0E TATU shows (ast_Errno is 8
http://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#pause-replicationhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#pause-replicationhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#causing-a-primary-key-collisionhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#causing-a-primary-key-collisionhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#update-auto-increment-for-master-master-replicationhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#update-auto-increment-for-master-master-replicationhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#pause-replicationhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#causing-a-primary-key-collisionhttp://mysql.wingtiplabs.com/documentation/reprv7zv/repair-replication#update-auto-increment-for-master-master-replication