Migration From Oracle to MySQL _ an NPR Case Study Presentation

7/30/2019 Migration From Oracle to MySQL _ an NPR Case Study Presentation

1/24

npr.org

Migration From Oracle to MySQL

An NPR Case Study

By Joanne Garlow


2/24

Overview Background

Database Architecture

SQL Differences

Concurrency Issues

Useful MySQL Tools

Encoding Gotchas


3/24

Background NPR (National Public Radio)

Leading producer and distributor of radio programming

All Things Considered, Morning Edition, Fresh Air, Wait, Wait,

Dont Tell Me, etc.

Broadcasted on over 800 local radio stations nationwide

NPR Digital Media

Website (NPR.org) with audio content from radio programs

Web-Only content including blogs, slideshows, editorial columns About 250 produced podcasts, with over 600 in directory

Mobile apps and sites

Syndication


4/24

High-Level System Architecture


5/24

Limitations of the Oracle Architecture Reached capacity of single system to support our

load

Replication outside our budget

Databases crashes were becoming frequent


6/24

Database Architecture Goals Redundancy

Scalability

Load balancing

Separation of concerns

Better security


7/24

High-Level System Architecture


8/24

Updated by our Content Management System

Transaction Oriented

Resource Contention

Highly Normalized

Isolation from main website

Read-only by our webservers

Horizontally scalable

Database Architecture

Main

InnoDB

AMG

MyISAMPUBLIC

InnoDB

STATIONS

InnoDB

Main

RO slave

Main

RO slave

ContentMgmt System

Web Servers

Scripts Backup

RO slave

Read and updated only by our website

Low resource contention

Small tables or log tables Short Transactions

Updated by a nightly script

Read-only by our ContentManagement System

Need fast full text queries

(replacing Oracle Text)

Large tables

Updated by a quarterly script

Read-only from our website Some log type information written

Low resource contention

No transactions


9/24

Issues When Converting SQL MySQL is case sensitive

Oracle outer join syntax (+) -> OUTER JOIN clause

Oracle returns a zero to indicate zero rows updated

MySQL returns TRUE (1) to indicate it successfully

updated 0 rows

MySQL sorts null to the top, Oracle sorts null to the

bottom

Use order by colName desc for sorting asc with nulls at

bottom

M SQL has Limit clause YAY!


10/24

Replacing Oracle Sequences Initialize a table with a single row:

CREATE TABLE our_seq (

id INT NOT NULL

);

INSERT INTO our_seq (id) VALUES (120000000);

Do the following to get the next number in the sequence:

UPDATE our_seq SET id=LAST_INSERT_ID(id+1);

SELECT LAST_INSERT_ID();


11/24

Replacing Oracle Sequences For updating many rows at once, get the total number of unique IDs you need first:

SELECT @totalRows := COUNT(*) FROM...

Then update npr_seq by that many rows:UPDATE npr_seq SET id=LAST_INSERT_ID(id+@totalRows);

and store that ID into another variable:SELECT @lastSeqId := LAST_INSERT_ID();

Then use the whole rownum workaround described above to get a unique value foreach row:INSERT INTO my_table (my_primary_id . . . ) SELECT@lastSeqId - (@rownum:=@rownum+1), . . . FROM (SELECT@rownum:=-1) r, . . .


12/24

Converting Functions NVL() -> IFNULL() or COALESCE() DECODE() -> CASE() or IF() Concatenating strings || -> CONCAT()

test || null returns test in Oracle

CONCAT(test,null) returns null in MySQL LTRIM and RTRIM -> TRIM() INSTR() works differently.

Use LOCATE() for Oracles INSTR() with occurrences = 1. SUBSTRING_INDEX() and REVERSE() might also work.


13/24

Converting Dates sysdate -> now()

Adding or subtracting

In Oracle 1 subtracts a day

In MySQL - 1 subtracts a milisecond must use

interval

TRUNC() -> DATE() TO_DATE and TO_CHAR -> STR_TO_DATE and

DATE_FORMAT


14/24

Update Differences You can't update a table that is used in the WHERE

clause for the update (usually in an "EXISTS" or asubselect) in mysql.UPDATE tableA SET tableA.col1 = NULLWHERE tableA.col2 IN

(SELECT tableA.col2FROM tableA A2, tableBWHERE tableB.col3 = A2.col3 AND

tableB.col4 = 123456);

You can join tables in an update like this (Mucheasier!):UPDATE tableAINNER JOIN tableB ON tableB.col3 = tableA.col3SET tableA.col1 = NULLWHERE tableB.col4 = 123456;


15/24

RANK() and DENSE_RANK() We really found no good MySQL equivalent for these

functions

We used GROUP_CONCAT() with an ORDER BY

and GROUP BY to get a list in a single column over a

window of data


16/24

Collation You can set collation at the server, database, table or

column level.

Changing the collation at a higher level (say on the

database) wont change the collation for preexisting

tables or column.

Backups will use the original collation unless you

specify all the way down to column level.


17/24

Concurrency Issues In our first round of concurrency testing, our system

ground to a halt!

Deadlocks

Slow Queries

MySQL configuration

sync_binlog = 1 // sync to disk, slow but safe

innodb_flush_log_at_trx_commit = 1 // write each

commit

transaction_isolation = READ-COMMITTED


18/24

Useful MySQL Tools MySQL Enterprise Monitor

http://www.mysql.com/products/enterprise/

MySQL GUI Tools Bundle:

http://dev.mysql.com/downloads/gui-tools/5.0.html

MySQL Query Browser similar to Oracles SQL

Developer

MySQL Administrator
http://www.mysql.com/products/enterprise/http://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://dev.mysql.com/downloads/gui-tools/5.0.htmlhttp://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/http://www.mysql.com/products/enterprise/


19/24

Innotop and innoDB Status innotop

http://code.google.com/p/innotop

Helped us identify deadlocks and slow queries (dont

forget the slow query log!)

In mysql, use

show engine innodb status\G;

Useful for contention and locking issues
http://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotophttp://code.google.com/p/innotop


20/24

Query Profiling Try the Query Profiler with Explain Plan when

debugging slow queries

http://dev.mysql.com/tech-resources/articles/using-new-

query-profiler.html
http://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.htmlhttp://dev.mysql.com/tech-resources/articles/using-new-query-profiler.html


21/24

Concurrency Solution Tuning our SQL and our server configuration helped

Turns out that the RAID card we were using had no

write cache at all. Fixing that allowed us to go live.


22/24

Encoding Gotchas Switched from ISO-8859-1 to UTF-8

Migration Tool

Issues with characters that actually were not ISO-8859-

1 in our Oracle database

Lack of documentation for the LUA script produced by

the migration GUI

Update encoding end to end

JSPs, scripts (Perl), PHP, tomcat (Java)


23/24

Continuing Issues Bugs with innodb locking specific records (as

opposed to gaps before records)

Uncommitted but timed out transactions

Use innotop or show engine innodb status\G; and

look for threads waiting for a lock but no locks blocking

them

Requires MySQL reboot


24/24

Questions?

Joanne Garlow

[email protected]

http://www.npr.org/blogs/inside
mailto:[email protected]://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidehttp://www.npr.org/blogs/insidemailto:[email protected]:[email protected]:[email protected]

Documents

Migration From Oracle to MySQL _ an NPR Case Study Presentation