Download pdf - Flexible Replication

© Continuent 2010

Linas Virbalas Continuent, Inc.

© Continuent 2010

/  Definition & Motivation /  Three Stories /  How? /  Questions and Answers

© Continuent 2010

© Continuent 2010

Heterogeneous Replication

Replication between different types of DBMS

© Continuent 2010

1.  Real-time integration of data between different DBMS types

2.  Seamless migration out of one DBMS type to another 3.  Data warehousing (real-time) from different DBMS

types

© Continuent 2010

/  Name: Linas Virbalas /  Country: Lithuania /  Implementing for Tungsten:

•  MySQL -> PostgreSQL •  MySQL -> Greenplum •  MySQL -> Oracle •  PostgreSQL WAL •  PostgreSQL Streaming Replication •  PostgreSQL Logical Replication

via Slony logs

/  Blog: http://flyingclusters.blogspot.com

© Continuent 2010

© Continuent 2010

1.  Story of MySQL -> … •  How to replicate from MySQL to PostgreSQL/Greenplum?

Oracle?

2.  Story of PostgreSQL -> PostgreSQL •  Advanced Logical Replication

3.  Story of PostgreSQL -> … •  For now: how to replicate from PostgreSQL to MySQL

© Continuent 2010

Why?

Because It Can! :)

With Tungsten Replicator

© Continuent 2010

/  Interfaces to implement new: •  Extractors •  Filters •  Appliers

/  Multiple replication services per one process /  JAVA /  Open Source GPL v2

© Continuent 2010

Technology: Replication Pipelines

© Continuent 2010

© Continuent 2010

/  Statement Based Replication

/  Row Based Replication

© Continuent 2010

© Continuent 2010

Tungsten Replicator

Tungsten Replicator

© Continuent 2010

Master Replicator

MySQL Extractor

Transaction History Log

Slave Replicator

PostgreSQL Applier


Filters Filters

© Continuent 2010

/  Provisioning /  Data Type Differences /  Database vs. Schema /  Default (Implicitly Defined) Schema Selection /  SQL Dialect Differences

•  Statement Replication vs. Row Replication

/  Character Sets and Binary Data /  Old Versions of MySQL

© Continuent 2010

Provisioning

/  Harder way: Dump data explicitly

/  Easier way: Replicate a mysqldump backup

Replicator

© Continuent 2010

MySQL PostgreSQL ! TINYINT SMALLINT

SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT

! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP

! TEXT (diff. sizes) TEXT ! BLOB BYTEA

…

/  Note the type differences between MySQL and PG

© Continuent 2010

Database vs. Schema

/  In MySQL these are the same: ! !CREATE DATABASE foo!

! !CREATE SCHEMA foo!

/  In PostgreSQL these are very different: CREATE DATABASE foo!! !CREATE SCHEMA foo!

/  Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas

© Continuent 2010

MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s;

! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1);

/  MySQL: Trivial to use ÙSE` /  MySQL: Going without ÙSE` generates different

events

/  PG: Extract the default schema from the event /  PG: Set it before applying

MySQL PostgreSQL USE s; > SET search_path TO s, "$user”;

© Continuent 2010

MySQL PostgreSQL CREATE TABLE complex (id INTEGER AUTO_INCREMENT PRIMARY KEY, i INT);

CREATE TABLE complex (id SERIAL PRIMARY KEY, i INT);

CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); …

/  Differences between DDL and DML statement SQL dialects

/  Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle

/  Tungsten Replicator Filters come to the rescue! •  Simple to develop Java or JavaScript extensions •  Event structure IN -> Filter -> Event structure OUT

© Continuent 2010

MySQL PostgreSQL INSERT INTO embedded_blob (key, data) VALUES (1, ‘?\0Ês\0^\0\’’)

ARGH!!! (SQL statement fails)

create table xlate(id int, d1 varchar(25) character set latin1, d2 varchar(25) character set utf8);

ARGH!!! (no way to translate to common charset)

/  Statement replication: MySQL syntax is “permissive” /  Embedded binary / alternate charsets /  Different charsets for different clients

/  Row replication: database/table/column charsets may differ

/  Answer: Stick with one character set throughout; use row replication to move binary data

© Continuent 2010

© Continuent 2010

© Continuent 2010

Logical Physical MySQL Statement Based x

MySQL Row Based x MySQL Mixed x

PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + -

Different data/structure on slave possible

+ -

/  A transaction is not accessible to the replicator under physical replication

/  Tungsten Replicator manages WAL/Streaming Replication

© Continuent 2010

Logical Physical MySQL Statement Based x

MySQL Row Based x MySQL Mixed x

PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x

Tungsten Replicator w/ PostgreSQLSlonyExtractor

x

Filters (data transformation) possible + - Different data/structure on slave

possible + -

/  With PostgreSQLSlonyExtractor transaction goes through the Replicator pipeline

Slave Replicator


Master Replicator

PostgreSQL SlonyExtractor


Filters Filters

© Continuent 2010

© Continuent 2010

Slave Replicator

MySQLApplier


Master Replicator

PostgreSQL SlonyExtractor


Filters Filters

© Continuent 2010

© Continuent 2010

© Continuent 2010

© Continuent 2010

Open Source http://tungsten-replicator.org #tungsten @ irc.freenode.net

My Blog: http://flyingclusters.blogspot.com

Commercial [email protected]

Continuent Web Site: http://www.continuent.com