MySQL 5.6 & 5.7 & 5.7+ Unlimited scaling

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Slide 1/45

Carsten Thalheimer Technical Sales Consultant ([email protected])

Linux - Virtualization - MySQL (LVM GBU - EMEA)

MySQL 5.6 & 5.7 & 5.7+

Unlimited scaling ???


The following is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and should not be relied

upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.

Die folgenden Informationen sind zur Skizzierung der grundsätzlichen Produktausrichtung von

Oracle bestimmt. Sie sind nur für reine Informationszwecke gedacht und dürfen nicht in einen

Vertrag aufgenommen werden. Es besteht keine Verpflichtung Material, Code oder

Funktionalitäten zu liefern und die Informationen dürfen nicht als Grundlage für

Einkaufsentscheidungen herangezogen werden.

Die Entwicklung, Produktplanung und die zeitliche Koordinierung von Produkteigenschaften

oder Funktionalitäten bleibt in der alleinigen Verantwortung von Oracle.

Safe Harbor Statement


• MySQL is Open Source (GPL 2)

– http://dev.mysql.com/downloads/mysql

– http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

– as well as under a variety of propriety agreements

• MySQL Lab releases MySQL

– http://labs.mysql.com/

– Haadop Applier, MySQL Utilities 1.4 etc, MySQL Utilities

• MySQL Development Milestone Releases

http://dev.mysql.com/downloads/mysql

MySQL 5.7.2 Only for testing, not fit for production (5.7.3 ???)

• MySQL Community Edition / MySQL Commercial Editions

– http://www.mysql.com/downloads/mysql

– Community Edition Optional Oracle Support offering available

Commercial Edition Standard Edition, Enterprise Edition, Cluster CGE

MySQL – Transparent Development

• Fully Functional Release Candidate

Quality

• Two to Four per Year

• Early Community Testing, Use, &

Feedback

• Accelerates Rate of Enhancements

• Improves Quality


http://labs.mysql.com/


http://www.mysql.com/downloads/mysql/


Program Agenda

Replication basics

Replication changes in MySQL 5.6

MySQL 5.7 DMR

MySQL 5.7 Labs release

Q & A


Program Agenda

Replication basics


MySQL 5.7 DMR


Q & A


MySQL Enterprise Monitor 2.2

MySQL Cluster 7.1

MySQL Cluster Manager 1.0

MySQL Workbench 5.2

MySQL Database 5.5

MySQL Enterprise Backup 3.5


Driving MySQL Innovation

Red bullet points: Announced during MySQL Connect 2013

All GA! - 2010


Oracle VM Template for MySQL

Enterprise Edition

MySQL Enterprise Oracle

Certifications

MySQL Windows Installer

MySQL Enterprise Security

MySQL Enterprise Scalability


MySQL Cluster 7.2


MySQL Utilities 1.0

MySQL Migration Wizard

MySQL for Excel


MySQL Enterprise Audit

MySQL Enterprise HA

with DRBD

All GA! - 2011

MySQL Database 5.6

MySQL Utilities 1.3

MySQL Cluster 7.3

MySQL Workbench 6.0



MySQL yum Linux repository

All GA! - 2012

All GA! - 2013

Lab and DMR* 2013 *Development Milestone Release

MySQL Proxy 0.8.3

MySQL Applier for Hadoop

MySQL 5.7.2 DMR*

MySQL Utilities Fabric Lab*

MySQL Multi Master Rep. Lab*

Plus

Connector/ODBC, Connector/PHP, Connector/Net, Connector/J, Connector Python, Connector /C++, Connector/C, Ruby Driver



MySQL Cluster 7.1


MySQL Workbench 5.2

MySQL Database 5.5



Driving MySQL Innovation

Red bullet points: Announced during MySQL Connect 2013

All GA! - 2010


Oracle VM Template for MySQL

Enterprise Edition

MySQL Enterprise Oracle

Certifications

MySQL Windows Installer

MySQL Enterprise Security

MySQL Enterprise Scalability


MySQL Cluster 7.2


MySQL Utilities 1.0

MySQL Migration Wizard

MySQL for Excel


MySQL Enterprise Audit

MySQL Enterprise HA

with DRBD

All GA! - 2011

MySQL Database 5.6

MySQL Utilities 1.3

MySQL Cluster 7.3

MySQL Workbench 6.0



MySQL yum Linux repository

All GA! - 2012

All GA! - 2013

Lab and DMR* 2013 *Development Milestone Release

MySQL Proxy 0.8.3

MySQL Applier for Hadoop

MySQL 5.7.2 DMR*

MySQL Utilities Fabric Lab*

MySQL Multi Master Rep. Lab*

Plus

Connector/ODBC, Connector/PHP, Connector/Net, Connector/J, Connector Python, Connector /C++, Connector/C, Ruby Driver


Current Generation of MySQL Replication

Simple to configure

Heterogeneous

Low TCO – no shared storage

Configure over LAN and WAN

Fast failover

Asynchronous: risk of of data loss, Semi-Synchronous is configurable

Automated failover or switchover via Utilities


Why Replicate?

Duplicates database from a “master” to a “slave”

Redundant copies of the data provide foundation for High Availability

Scale out by distributing queries across the replication cluster

Master

Slaves

Web / App Servers

Writes & Reads Reads


Replication Replication Solutions


Complex Topologies Ratio Master to Slave ~ 1:10

• Relieve master server by creating relay slaves

• Just keep binary logs

• Row Based Replication best

• Do not store data in tables

• --log slave-updates

• Use Blackhole engine “Black hole” that accepts data

but throws it away and does not store it


MySQL Replication

Native to MySQL

Duplicates updates from a “master” to a “slave”

Common solution for HA

– Used to scale out as well

Failover is either scripted or provided by additional middleware

Supports Asynchronous and Semi-Synchronous replication


Async Replication


Semi-sync Replication


• Scales to 48 CPU Threads

• Up to 230% performance gain over MySQL 5.5

IMPROVED PERFORMANCE AND SCALABILITY

• Better transactional throughput and availability

IMPROVED INNODB

• Better query exec times and diagnostics for query tuning and debugging

IMPROVED OPTIMIZER

• Higher performance, availability and data integrity

IMPROVED REPLICATION

• Better Instrumentation, User/Application level statistics and monitoring

IMPROVED PERFORMANCE SCHEMA

• Fast, Key Value access with full ACID compliance, better developer agility

New! NoSQL ACCESS TO INNODB

MySQL 5.6: Best Release Ever!


Development Priorities

• Improve read consistency from slaves

• Reduce risk of data loss if master fails

• Minimize overhead

PERFORMANCE

• Self-Healing

• Improve availability

FAILOVER & RECOVERY

• Correct, consistent & accessible

DATA INTEGRITY

• Enhance responsiveness to business

• Reduce TCO

DEV/OPS AGILITY


MySQL 5.6: Transforming Replication

• Multi-Threaded Slaves

• Binary Log Group Commit

• Optimized Row-Based Replication

PERFORMANCE

• Global Transaction Identifiers

• Replication Failover & Admin Utilities

• Crash Safe Slaves & Binlogs

FAILOVER & RECOVERY

• Replication Event Checksums

DATA INTEGRITY

• Replication Utilities

• Time-Delayed Replication

• Remote Binlog Backup

• Informational Log Events

• Server UUIDs

DEV/OPS AGILITY


Multi-Threaded Slaves

Increases slave throughput, reducing lag

All transactions received into slave’s relay

log

Implements multiple SQL threads, based

on database

Applies events to different databases in

parallel

Great for systems which isolate application

data using databases – e.g. multi-tenant


5x Higher Performance with MySQL 5.6

0 5 10

QPS 58,11 144,4 282,53

0

50

100

150

200

250

300

Qu

eri

es p

er

Seco

nd

Worker Threads

Multi-Threaded Slave Performance


Optimized Row Based Replication

Increases replication throughput for master and slave

Reduces Binlog size, memory requirements & network bandwidth

Only replicates those elements of the Row image that have changed

Primary Key Changed Columns


Binlog Group Commit

Increases replication throughput by

increasing performance of the

master

Commits multiple transactions as a

group to Binlog on disk

Finer grained locking; reducing lock

wait times


Replication Binary Log Group Commit Performance

Up to 2.3x with binlog

configured versus 5.6.5 with

no binlog

Up to 3.5x faster with binlog

configured

Sysbench R/W

8 x Socket / 6-core Intel Xeon

7540, 2GHz

512GB RAM

SSD

Sync_binlog = 1


Master

GTID=123456

GTID=123456

GTID=123456 GTID=123456

Global Transaction Identifiers

Simple to track & compare replication across the farm

Unique identifier for each transaction written to the Binlog

Automatically identify the most up-to-date slave for failover

Deploy n-tier replication hierarchies


Crash-Safe Slaves & Binlog

Before:

– Transaction Data: in tables

– Replication Info: in files

MySQL 5.6

– Transaction Data: in tables

– Replication Info: in tables

Data

Position Info CRASH!

Time

Data

Position Info

Time

master.info & relay-log.info files

can be replaced with tables

Automatic recovery of a slave and Binlog

after a failure

Binlog and table data are transactionally

consistent

Resumes replication without intervention

Automatically rolling back replication to last

committed event

Eliminates risk of data loss or corruption

Atomic

Atomic


Replication Event Checksums

Ensures replicated data is correct,

consistent and accessible

Detects corrupt replication events before

they’re applied

Returns an error

Protects entire replication path

Memory

Disk

Network

Bugs


Replication Event Checksums

Implemented in the binary and relay logs

New mysqld options:

binlog-checksum=CRC32

Checksums written to the binary log

master-verify-checksum=1

Master validates checksum read from the binary log

slave-sql-verify-checksum=1

Slave validates checksum when reading from the relay log


Time Delayed Replication

Configure time period before replication

events applied to slave

Per-slave, via execution of SQL Thread

Second-level granularity, up to 68 years!

Protects against operational error

Dropping a table, etc.

Allows database to be inspected without

loading a back-up

Master :3306

Binary Logs

Relay Log

Relay Log

Slave 1 :3307

Slave2 :3308

10 Minute Delay


Time Delayed Replication - configuring

slave2> CHANGE MASTER TO

-> MASTER_HOST = 'localhost‘,

-> MASTER_PORT = 3306,

-> MASTER_USER = 'repl_user‘,

-> MASTER_PASSWORD = ‘sun123!‘,

-> MASTER_DELAY = 600;

slave2> START SLAVE;


Master

GTID=123456

GTID=123456

GTID=123456 GTID=123456

Global Transaction Identifiers

Simple to track & compare replication across the farm

Unique identifier for each transaction written to the Binlog

Automatically identify the most up-to-date slave for failover

Deploy n-tier replication hierarchies


Time Delayed Replication - configuring

slave2> CHANGE MASTER TO

-> MASTER_HOST = 'localhost‘,

-> MASTER_PORT = 3306,

-> MASTER_USER = 'repl_user‘,

-> MASTER_PASSWORD = ‘sun123!‘,

-> MASTER_DELAY = 600;

slave2> START SLAVE;


MySQL Utilities

New, standalone package: MySQL Utilities 1.3

Automate common Dev/Ops tasks

Replication: provisioning, testing, monitoring and failover (now as daemon)

Database comparisons: consistency checking

Database administration: users, connections, tables, etc

Auditing

Python scripts

Standalone or launched from MySQL Workbench

Available from dev.mysql.com

Extensible to include custom scripting; Python library for extensibility

Powerful DevOps Management tools for MySQL


MySQL Utilities

Automate common Dev/Ops tasks

Replication: provisioning, testing, monitoring and failover

Database comparisons: consistency checking

Database administration: users, connections, tables, etc

New utilities in development, i.e. log analysis

Implemented as Python scripts, plug-in for MySQL Workbench

Extensible to include custom scripting

Resources: Documentation & Community Forum

http://dev.mysql.com/doc/workbench/en/mysql-utils-man.html

http://forums.mysql.com/list.php?155











Utility Workflow for Replication

Replicate: Initiates Replication to the new slave

Check: Validates the replication configuration and operation

Show: Display Replication topology

Fail-Over & Admin: Detects and failovers (or switches) from master to

slave. Status monitoring

Check Show Fail-Over

& Admin Replicate


mysqlreplicate Utility

Starts replication process

User can provide login and

connection parameters for the master

Configurable replication start points:

current position / GTID, specific

event, entire binlog

Checks storage engine compatibility

Enables fast, simple scaling

New Slave

Login,

Connection &

Start-Point

Master

Replication configured,

binlog position

Repl


mysqlreplicate example

$ mysqlreplicate --master=root:billy@localhost:3306

--slave=root:billy@localhost:3307

# master on localhost: ... connected.

# slave on localhost: ... connected.

Repl


mysqlrplshow Utility

Discovers & displays replication

topology

Shows slaves attached to each

master

Labels each slave with hostname

and port number

Displays multiple topologies,

including hierarchical and circular

Displays results in a graph or list

Enables user to monitor topology

on-demand

$ mysqlrplshow --master=root@localhost:3311

# master on localhost: ... connected.

# Finding slaves for master: localhost:3311

# Replication Topology Graph::

localhost:3306 (MASTER)

|

+--- localhost:3307 - (SLAVE)

|

+--- localhost:3308 - (SLAVE + MASTER)

|

+--- localhost:3313 - (SLAVE)

Show


mysqlrplcheck Utility

Tests replication on master and slave to ensure proper configuration of

replication

Checks binlog is enabled and displays any configured exceptions

Checks slave access and privileges to master

Checks slave connection status

Checks for conflicts in master.info file (or mysql.slave_master_info table)

Checks storage engine compatibility

Tests executed sequentially

Reports status of each test

Simple verification of deployment & fast fault resolution

Check


mysqlrplcheck example

$ mysqlrplcheck --master=root@host1:3310

--slave=root@host2:3311

# master on host1: ... connected.

# slave on host2: ... connected. Test Description Status

-------------------------------------------------------------

Checking for binary logging on master [pass]

Are there binlog exceptions? [pass]

Replication user exists? [pass]

Checking server_id values [pass]

Is slave connected to master? [pass]

Check master information file [pass]

Checking InnoDB compatibility [pass]

Checking storage engines compatibility [pass]

Checking lower_case_table_names settings [pass]

Checking slave delay (seconds behind master) [pass]

# ...done.

Check


Replication Administration Utility

Perform switchover to eliminate

downtime during planned maintenance

Start and stop slaves

Slave discovery & monitoring

Slave status, thread status

Replication processing, including any lag

Configure slave promotion policies

Master

Slaves

Administration

Utility

Status &

Switchover

Admin


Planned switchover to new master with mysqlrpladmin

$ mysqlrpladmin --master=root@rod:3306

--slaves=root@jane:3306,root@jane:3307,root@freddy:3306

--new-master=root@jane:3306 --demote-master switchover

# Performing switchover from master at rod:3306 to slave at jane:3306.

# Checking candidate slave prerequisites.

# Waiting for slaves to catch up to old master.

# Stopping slaves.

# Performing STOP on all slaves.

# Demoting old master to be a slave to the new master.

# Switching slaves to new master.

# Starting all slaves.

# Performing START on all slaves.

# Checking slaves for errors.

# Switchover complete.

# ...done.

Switch-

Over


Enabling self-healing replication topologies

Automated failover & recovery mysqlfailover Utility

Switchover & administration

mysqlrpladmin Utility

Fully configurable slave promotion policies

Delivers HA within the core MySQL

distribution

HA Utilities Monitoring

Failed

Master

Slaves

Promoted

Master

Fail-Over

& Admin Replication HA Utilities


mysqlfailover – monitoring resumes MySQL Replication Monitor and Failover Utility

Failover Mode = auto Next Interval = Mon Mar 19 16:05:12 2012

Master Information

------------------

Binary Log File Position Binlog_Do_DB Binlog_Ignore_DB

mysql-bin.000001 1117

GTID Executed Set

A0F7E82D-3554-11E2-9950-080027685B56:1-5

UUIDs

+------------+-------+---------+--------+------------+---------+

| host | port | role | state | gtid_mode | health |

+------------+-------+---------+--------+------------+---------+

| jane | 3306 | MASTER | UP | ON | OK |

| jane | 3307 | SLAVE | UP | ON | OK |

| freddy | 3306 | SLAVE | UP | ON | OK |

+------------+-------+---------+--------+------------+---------+

Fail-Over


mysqlfailover – master failed Failover starting...

# Candidate slave jane:3306 will become the new master.

# Preparing candidate for failover.

# Creating replication user if it does not exist.

# Stopping slaves.

# Performing STOP on all slaves.

# Switching slaves to new master.

# Starting slaves.

# Performing START on all slaves.

# Checking slaves for errors.

# Failover complete.

# Discovering slaves for master at jane:3306

Failover console will restart in 5 seconds.

Fail-Over


Program Agenda

Replication basics


MySQL 5.7 DMR


Q & A


MySQL Connect 2013 Replication & Utilities Update


Some words about 5.7.2 DMR

MySQL 5.7 is the next big thing, it’ll be the new ‘Best release ever’

Demonstrates ongoing Oracle commitment

Some customers will use it for production

Lots of new features

– It’s faster, it scales higher

– Improved InnoDB Online Alter Table, Online Rename Index, Online Change Varchar

– Faster Replication

– Enhanced Performance Schema, Memory, Replication and Stored Program Instrumentation

It won’t be available in the short term (not in FY14, 5.6 GA - Feb. 5th 13)

(Micro view)


0

5000

10000

15000

20000

25000

30000

35000

40000

With Selects Without Select

Co

nn

ecti

on

s p

er

seco

nd

MySQL 5.7.2

MySQL 5.6.9

MySQL 5.7.2: Connections / second Faster processing of new connections

Built with input from Facebook

Offloaded THD initialization and network initialization to worker thread

• mysql-bench

• 25 concurrent client threads

• Executing connect/select/disconnect

• 100000 iterations each

+32% +64%


0

500

1000

1500

2000

2500

To

tal T

ime

in

Se

co

nd

s

MySQL 5.7 versus 5.6 2 - 11 X Improvement

5.6.13

5.7.2

MySQL 5.7.2: InnoDB Temporary Tables Improved Performance

11X faster

in CREATE/DROP

Eliminated all IO during the table

create/drop phase)

2-4X faster

in INSERT/DELETE/ UPDATE

Eliminated redo logging, locking and

change Buffering

* sql-bench, modified to use temp-tables

* Seed size of 10K for Create/Drop

* Seed size of 5M for Insert/Delete/Update

Total time to complete in seconds

11X 2X

4X 2X


MySQL 5.7.2 Sysbench Benchmarks

Sysbench Point Select

Intel(R) Xeon(R) CPU X7560 x86_64

4 sockets x 8 cores-HT (64 CPU threads)

2.27GHz, 256G RAM

Oracle Linux 6.2

95% Faster than MySQL 5.6


0

100000

200000

300000

400000

500000

600000

8 16 32 64 128 256 512 1024

Qu

eri

es p

er

Seco

nd

Concurrent User Sessions

MySQL-5.7

MySQL-5.6

MySQL-5.5

500,000 QPS


MySQL 5.7.2 Sysbench Benchmarks OLTP Read Only



2.27GHz, 256G RAM

Oracle Linux 6.2

0

5000

10000

15000

20000

25000

8 16 32 64 128 256 512 1024

Tra

nsacti

on

s p

er

Seco

nd


MySQL-5.7

MySQL-5.6

MySQL-5.5



sb_OLTP_RO_1M_8tab-ps Max-TPS


MySQL 5.7.3 DMR: Benchmark

InnoDB Memcached



2000Mhz, 256G RAM

Oracle Linux 6.2


1,150,000 QPS

0

200000

400000

600000

800000

1000000

1200000

1400000

8 16 32 64 128 256 512 1024

Qu

eri

es p

er

Seco

nd


5.7.3

5.6

Preview!


Slave Master

Lossless Semi-Synchronous Replication

Master waits until defined set of slaves have

received the transaction before:

Committing txn to Storage Engine

Making change visible to other clients

Acknowledging commit to application

Does not wait for slave to apply change

Latency minimized

Ensures that no other transaction acts on new

data until it’s safe on a slave

(rpl_semi_sync_master_wait_point = AFTER_SYNC)

App commits

Written to Binary log

Send to slave(s)

Commit to SE & externalized

Ack Commit

Write to relay log

Apply txn

MySQL 5.7.2 DMR


Multi-Threaded Inter-Transactional Replication

Multiple worker threads apply multiple transactions to the slave in parallel

Parallel transaction must act on disjoint data

Don’t read or write any overlapping rows

Including within the same database!

Consistency maintained

Increases the slave’s throughput

Work-in-progress

Refactoring work done, Tuning work underway

MTR, Higher slave throughput

MySQL 5.7.2 DMR


Replication Monitoring

Replication is a delicate module of MySQL

- people want to know its status frequently and in detail

Table to monitor via SQL replication status

– replication_connection_configuration,

– replication_connection_status,

– replication_execute_configuration and

– replication_execute_status

– replication_execute_status_by_coordinator

– replication_execute_status_by_worker

Now in Performance Schema

MySQL 5.7.2 DMR


Program Agenda

Replication basics


MySQL 5.7 DMR


Q & A


Multi-Source Replication

Consolidate updates from multiple

Masters into one Slave

Consolidated view of all shards

Centralized point for backups

Compatible with Semi-

Synchronous Replication &

enhanced MTS

Control and filters will be per-

source (filters global in labs)

Application should keep data sets

disjoint between sources

Binlog

Master 1

Binlog

Master 2

…

…

Binlog

Master N

IO 1

Relay 1

Coordinator

W1 W2 … WX

IO 2

Relay 2

Coordinator

W1 W2 … WX

…

…

Coordinator

W1 W2 … WX

IO N

Relay N

Coordinator

W1 W2 … WX

Slave

MySQL 5.7.2 Lab


MySQL Replication Current issue


MySQL Utilities - Fabric

Inherits all major MySQL Replication properties

Plus horizontal partitioning (sharding).

Technically, this version today is a set of Python scripts.

Fabric ships MySQL Replication HA out of the box

MySQL aims to automate management of nodes.

Utilizing Fabric in application, needs a Fabric aware connector

but should be available for existing 5.6 version as well

‘On the fly resources’ thinking

MySQL Replication learns sharding for read and write scale out

MySQL Utilities Lab


MySQL Utilities - Fabric High availibity and sharding

MySQL Fabric

Connector

Application

Read-slaves

SQL

Master group

Read-slaves

Master group

MySQL Utilities 1.4 - Lab



MySQL Fabric is daemon for managing farms of MySQL servers

– Easy to use, built-in high availability

– Easy to use, elastic, horizontal scaling for the cloud

– Attractive data models and drivers for developers

Farms consist of „groups“.

– Any number of individual MySQL servers

– A MySQL Replication cluster

– A group describing a replication cluster consists of a master and any number of slaves

MySQL Fabric can setup, administrate and monitor groups

Failure detectors (heartbeating) can be used for automatic failover

– Failover is based on MySQL 5.6+ GTID logic, use MySQL 5.6.10 or newer.

New

MySQL Utilities Lab



Scale-out through sharding

Connectors

Python

Java

PHP

Application provides shard key

Range or Hash

Tools for resharding

Global updates & tables

Available in MySQL Utilities 1.4.0

High availibity and sharding

MySQL Fabric

Connector

Application

Read-slaves

SQL

Master group

Read-slaves

Master group

MySQL Utilities 1.4 - Lab



Fabric supports range, hash or list

based partitioning of tables

A column reflects the shard key

Each partition is assigned to a logical

shard group

Sharding is not transparent to SQL users

Sharding

MySQL Utilities Lab


Next Steps

• MySQL Connect 2013 is Over, But You Can Keep Learning

• http://www.oracle.com/mysqlconnect/index.html#

• MySQL Replication

• http://www.mysql.com/products/enterprise/replication.html

• MySQL Lab releases MySQL

• http://labs.mysql.com/

MySQL Utilities

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

http://www.oracle.com/mysqlconnect/index.html

http://www.oracle.com/mysqlconnect/index.html

http://www.mysql.com/products/enterprise/replication.html












Q&A Werner Ressle Carsten Thalheimer ([email protected]) ([email protected])

Technical Sales Consultant

LVM GBU (EMEA)

You can meet us @DOAG 2013

Oracle booth

Documents

MySQL 5.6 & 5.7 & 5.7+ Unlimited scaling