Become a MySQL DBA - Webinars - Schema Changes for MySQL Replication & Galera Cluster (slides)

Copyright 2015 Severalnines AB

Schema Changes for MySQL Replication & Galera Cluster

August 25, 2015

Krzysztof Książek

Severalnines

[email protected]

1

mailto:[email protected]


! We want to help all non-DBA people who look after MySQL infrastructures

! Share tips and good practices

! Watch the replays of the previous webinars on our Slideshare page

! http://www.slideshare.net/Severalnines/videos

2

“Become a MySQL DBA” series

http://www.slideshare.net/Severalnines/videos


! Different methods to perform schema changes on MySQL Replication and Galera

! rolling schema change

! online alters

! external tools, e.g., pt-online-schema-change

! Differences between MySQL 5.5 and 5.6

! Differences between MySQL Replication vs Galera

! Example real-life scenarios with MySQL Replication and Galera setups

3

Agenda


Schema changes in MySQL

4


! Application development

! New features to existing modules

! New modules

! Data management and archiving (partitions)

! Schema optimization

! Performance optimization

5

Why schemas need changes?


! Point-in-time schema changes

! Everything happens at a given point of time, over the whole infrastructure

! Rolling schema changes

! Changes happen at different times across the infrastructure

! Requirement is that schema versions have to be compatible with each other

6

Types of schema changes


! Schema changes cause significant performance overhead as they are I/O-heavy operations

! Schema changes may take long time to finish on larger tables

! Direct schema changes don’t play well with MySQL replication due to the lag inducted

! Direct schema changes don’t play well with Galera

! Rolling schema changes may require significant amount of work

7

Schema changes - limitations


! Replication has it’s limitations, only some changes can be replicated. RBR is very strict here

! Remove a column at the end of the table - yes

! Remove a column in the middle - no go

! Add a column at the end - works

! Add a column in the middle - not really

! Not every conversion between column types works - refer to the documentation

8



! You can’t change the schema and the code at the exact time

! New schema have to be compatible with old code

! Depreciate columns, not remove

! Remove them two, three iterations later, once you confirm you _can_ remove

9



Ways to perform a schema change

10


! Online DDL

! Rolling schema upgrade

! Online schema change

! Rolling Schema Upgrade for Galera

! Total Order Isolation for Galera

11

Methods to perform a schema change


! DDL, historically, was a locking operation

! Some of them were fast, i.e. DROP (secondary) INDEX

! InnoDB plugin in 5.1 and 5.5 introduced Fast Index Creation that allowed reads to be executed while secondary index has been added

! MySQL 5.6 introduced online DDL’s for InnoDB

12

Online DDL


! Many operations are non-blocking for reads and writes

! Add an index

! Add a column

! Reorder columns

! Add a primary key

! and many others (check the documentation)

! https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html#innodb-online-ddl-summary-grid

13

Online DDL

https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html#innodb-online-ddl-summary-grid


! Busy tables may be an issue (innodb_online_alter_log_max_size)

! Lag will be an issue - online DDL is online on the master only

! Not suitable for large tables in replication setup

14

Online DDL


! Run DDL from the bottom to the top of replication chain

! SET SESSION sql_log_bin=0 to avoid errant transactions

! Once you reach the master, failover and then alter the old master

! Schema changes have to be compatible, you’ll be running DML’s on a mix of altered and not altered tables

! Schema changes have to be compatible, you’ll be reading from a mix of altered and not altered tables

15

Rolling Schema Upgrade


! Different tools (pt-online-schema-change, Online Schema Change from FB), same idea

! Create a new table with desired schema

! Setup triggers to copy data from old to the new table

! Copy the data from old to new table in batches

! Rename table once the data has been copied

16

Online Schema Change - external tools


Online Schema Change - external tools

17


! pt-online-schema-change - the most popular tool

! Point-in-time change, you can use for any DDL

! Replicaton-aware, it can monitor lag and throttle itself

! Introduces additional load, slow down your system

! It has issues with foreign keys

! Can’t work with tables which have triggers

! Won’t work with tables without PK and unique key

18

pt-online-schema-change


! Otherwise it’s a very reliable tool

! Schema change may take a long time but it’s throttling itself and can be paused at any time

! One of the most important tools in the DBA arsenal

! Sometimes it’s the only way to run a schema change without downtime

19

pt-online-schema-change


Schema changes in Galera Cluster

20


! All changes happen at the exact time in the cluster

! All changes are blocking (you won’t benefit from online DDL’s)

! Great for consistency

! You can run any change this way

! Not suitable for larger tables due to severe locking

21

Total Order Isolation


! Similar to the rolling schema upgrade we discussed earlier

! Process is partially automated - you need to set wsrep_OSU_method to RSU and proceed with DDL

! Node will be in Desync state for a duration of the DDL

! Once DDL finishes, node applies remaining writesets

! Writesets are stored in gcache, both in memory and on disk

22




23


Different scenarios for a schema change

24


! If a change is non-blocking - online DDL

! pt-online-schema-change may also work

! pt-osc should not take more than a couple of minutes

! For Galera - TOI if you can accept few seconds of stall

! Otherwise - pt-online-schema-change is the only option

25

Small tables (up to 10s of a direct DDL)


! pt-online-schema-change is the best option here for both MySQL and Galera

! pt-online-schema-change may take even couple of hours to finish

! It’s less cumbersome than rolling upgrade

! Rolling schema upgrade may be the option of choice if pt-osc can’t be used due to it’s limitations (no PK, triggers in altered table, etc.)

26

Medium tables (20 - 30m, up to 1h)


! pt-osc will work but it will take long time (36 - 48h) to finish

! You need to have enough disk space for binlogs and altered copy of the table

! Rolling schema upgrade may be a feasible option

! For Galera pt-osc can be also the best option

! RSU is another option but ensure you have large gcache (or a space on disk for gcache files)

27

Large tables (more than 1h, up to 12h)


! Are you sure that you need such a large table?

! Try to avoid them by splitting data across many tables or archiving old data

! Rolling schema upgrade is the way to go

! Use snapshots or xtrabackup to provision new nodes

! Galera may need large gcache (both in memory and on disk)

! Process of joining desynced node will take a long time

28

Very large tables (more than 12h)


! Schema change speed depends on your hardware and data structure

! Test the changes on development environment

! Is it worth using triggers or foreign keys and not have the option to use pt-osc?

! You _do_ want to have PK defined in all tables - do it before it grows too large to alter

! Schema changes will happen, get used to them

29

What to keep in mind?


! More blogs in “Become a MySQL DBA” series:

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

! Contact: [email protected]

30

Thank You!

http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

mailto:[email protected]

Technology

Become a MySQL DBA - Webinars - Schema Changes for MySQL Replication & Galera Cluster (slides)