Embed Size (px)
DESCRIPTION
First, we'll harness Tungsten Replicator's flexibility to implement heterogeneous replication from MySQL to PostgreSQL. Then, we'll introduce advanced logical replication for PostgreSQL, which opens up new horizons in replication abilities. Finally, we'll take a look at how logical replication enables data replication from PostgreSQL to MySQL.
Citation preview
© Continuent 2010
Linas Virbalas Continuent, Inc.
© Continuent 2010
/ Definition & Motivation / Three Stories / How? / Questions and Answers
© Continuent 2010
© Continuent 2010
Heterogeneous Replication
Replication between different types of DBMS
© Continuent 2010
1. Real-time integration of data between different DBMS types
2. Seamless migration out of one DBMS type to another 3. Data warehousing (real-time) from different DBMS
types
© Continuent 2010
/ Name: Linas Virbalas / Country: Lithuania / Implementing for Tungsten:
• MySQL -> PostgreSQL • MySQL -> Greenplum • MySQL -> Oracle • PostgreSQL WAL • PostgreSQL Streaming Replication • PostgreSQL Logical Replication
via Slony logs
/ Blog: http://flyingclusters.blogspot.com
© Continuent 2010
© Continuent 2010
1. Story of MySQL -> … • How to replicate from MySQL to PostgreSQL/Greenplum?
Oracle?
2. Story of PostgreSQL -> PostgreSQL • Advanced Logical Replication
3. Story of PostgreSQL -> … • For now: how to replicate from PostgreSQL to MySQL
© Continuent 2010
Why?
Because It Can! :)
With Tungsten Replicator
© Continuent 2010
/ Interfaces to implement new: • Extractors • Filters • Appliers
/ Multiple replication services per one process / JAVA / Open Source GPL v2
© Continuent 2010
Technology: Replication Pipelines
© Continuent 2010
© Continuent 2010
/ Statement Based Replication
/ Row Based Replication
© Continuent 2010
© Continuent 2010
Tungsten Replicator
Tungsten Replicator
© Continuent 2010
Master Replicator
MySQL Extractor
Transaction History Log
Slave Replicator
PostgreSQL Applier
Transaction History Log
Filters Filters
© Continuent 2010
/ Provisioning / Data Type Differences / Database vs. Schema / Default (Implicitly Defined) Schema Selection / SQL Dialect Differences
• Statement Replication vs. Row Replication
/ Character Sets and Binary Data / Old Versions of MySQL
© Continuent 2010
Provisioning
/ Harder way: Dump data explicitly
/ Easier way: Replicate a mysqldump backup
Replicator
© Continuent 2010
MySQL PostgreSQL ! TINYINT SMALLINT
SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT
! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP
! TEXT (diff. sizes) TEXT ! BLOB BYTEA
…
/ Note the type differences between MySQL and PG
© Continuent 2010
Database vs. Schema
/ In MySQL these are the same: ! !CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ In PostgreSQL these are very different: CREATE DATABASE foo!! !CREATE SCHEMA foo!
/ Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas
© Continuent 2010
MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s;
! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1);
/ MySQL: Trivial to use `USE` / MySQL: Going without `USE` generates different
events
/ PG: Extract the default schema from the event / PG: Set it before applying
MySQL PostgreSQL USE s; > SET search_path TO s, "$user”;
© Continuent 2010
MySQL PostgreSQL CREATE TABLE complex (id INTEGER AUTO_INCREMENT PRIMARY KEY, i INT);
CREATE TABLE complex (id SERIAL PRIMARY KEY, i INT);
CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); …
/ Differences between DDL and DML statement SQL dialects
/ Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle
/ Tungsten Replicator Filters come to the rescue! • Simple to develop Java or JavaScript extensions • Event structure IN -> Filter -> Event structure OUT
© Continuent 2010
MySQL PostgreSQL INSERT INTO embedded_blob (key, data) VALUES (1, ‘?\0^Es\0^\0\’’)
ARGH!!! (SQL statement fails)
create table xlate(id int, d1 varchar(25) character set latin1, d2 varchar(25) character set utf8);
ARGH!!! (no way to translate to common charset)
/ Statement replication: MySQL syntax is “permissive” / Embedded binary / alternate charsets / Different charsets for different clients
/ Row replication: database/table/column charsets may differ
/ Answer: Stick with one character set throughout; use row replication to move binary data
© Continuent 2010
© Continuent 2010
© Continuent 2010
Logical Physical MySQL Statement Based x
MySQL Row Based x MySQL Mixed x
PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + -
Different data/structure on slave possible
+ -
/ A transaction is not accessible to the replicator under physical replication
/ Tungsten Replicator manages WAL/Streaming Replication
© Continuent 2010
Logical Physical MySQL Statement Based x
MySQL Row Based x MySQL Mixed x
PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x
Tungsten Replicator w/ PostgreSQLSlonyExtractor
x
Filters (data transformation) possible + - Different data/structure on slave
possible + -
/ With PostgreSQLSlonyExtractor transaction goes through the Replicator pipeline
Slave Replicator
Transaction History Log
Master Replicator
PostgreSQL SlonyExtractor
Transaction History Log
Filters Filters
© Continuent 2010
© Continuent 2010
Slave Replicator
MySQLApplier
Transaction History Log
Master Replicator
PostgreSQL SlonyExtractor
Transaction History Log
Filters Filters
© Continuent 2010
© Continuent 2010
© Continuent 2010
© Continuent 2010
Open Source http://tungsten-replicator.org #tungsten @ irc.freenode.net
My Blog: http://flyingclusters.blogspot.com
Commercial [email protected]
Continuent Web Site: http://www.continuent.com
