Upload
sveta-smirnova
View
117
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Slides which I presented at MySQL Connect 2013 together with Matt Lord
Citation preview
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.2
Safe Harbour Statement
THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES O FUNCTIONALITY DESCRIBED FOR ORACLE’S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3
Insert Picture HereMaking Backups in Extreme SituationsSveta SmirnovaPrincipal Technical Support Engineer
Matt LordMySQL Product Manager
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4
Program Agenda
Introduction Agenda Overview Dealing With Space Constraints Dealing With Time Constraints Dealing with Resource Usage Constraints Minimizing Locking Impacts Handling Data Corruption Conclusion
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5
Insert Picture Here
Introduction
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6
Who We Are
Matt Lord
Former MySQL Support engineer
Worked with hundreds of backup issues
Now Product Manager
Sveta Smirnova
MySQL Support engineer
Support representative in MySQL Enterprise Backup (MEB) team
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7
Why Backups?
1. They are a critical insurance against: Hardware failure User error Program error
2. You can use them to deploy new instances
3. Setup replication clones
4. Data is your most valuable IP. You have to protect it from loss.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8
The Backup Conundrum
1. Backup tools guarantee consistency by: Locking tables Running long transactions Even using global locks
2. Backup jobs generate a lot of system overhead
3. Backups take up a lot of storage space
How do you backup your critical production systems then? Servers are performing critical business functions Downtime is not acceptable Negative performance impacts are not acceptable Storage space is critical
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9
So How Do I Perform Backups...
without using local storage?
with minimal local resource usage?
with minimal impact on the MySQL instance I'm backing up?
when I need the backup to complete in a small time window?
without locking tables and entire MySQL instances?
when the data or the backup itself has corrupt data?
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10
What We Won't Cover
Full backups
Partial backups
Incremental backups
Backup automation
General backup tuning
Instead, see:
MySQL Enterprise Backup BOF [BOF3558, Saturday, 5:30 PM]
MySQL Backup-and-Recovery [CON7659, Sunday, 11:30 AM]
Backing Up MySQL Databases [CON3278, Sunday, 1:00 PM]
MySQL Enterprise Edition – Hands-On Lab, Sunday, 10:00 AM
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11
Insert Picture Here
Dealing With Storage Constraints
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12
When There Is Little Storage Space
Very large data directory size
Small amount of local storage space left
But MEB process should be run on the same machine!
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13
Backup Options
Compressed backups: --compress, --compress-level=LEVEL
Stream backups over SSH
Backup directly to tape, using the SBT interface
Create partial backups: --include, --with-tts
Create incremental backups
Don’t backup empty and unused pages: --skip-unused-pages
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14
Backup Options: Compression Compression ratios of 80% or more are possible!
shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed \> backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed334M /opt/mysql/backups/uncompressed334M total
shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed \> --compress --compress-level=9 backup 2>/dev/null && du –csh \> /opt/mysql/backups/compressed81M /opt/mysql/backups/compressed81M total
shell# mysqlbackup --backup-dir=/opt/mysql/backups/uncompressed \> backup 2>/dev/null && du -csh /opt/mysql/backups/uncompressed334M /opt/mysql/backups/uncompressed334M total
shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed \> --compress --compress-level=9 backup 2>/dev/null && du –csh \> /opt/mysql/backups/compressed81M /opt/mysql/backups/compressed81M total
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15
Backup Options: Stream Backups
We can avoid local storage altogether
Simply specify the tmpdir as the backup location (for temporary data)
Should be big enough to keep all changes made during backup
Still no space requirements for data files
shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup \> --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo \> "cat > /tmp/backup.img“
shell# ssh matt@solo "ls -lh /tmp/backup.img"-rw-r--r-- 1 matt staff 333M Sep 10 15:54 /tmp/backup.img
shell# mysqlbackup --backup-image=- --backup-dir=/tmp/backup \> --disable-manifest backup-to-image 2>/dev/null | ssh matt@solo \> "cat > /tmp/backup.img“
shell# ssh matt@solo "ls -lh /tmp/backup.img"-rw-r--r-- 1 matt staff 333M Sep 10 15:54 /tmp/backup.img
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16
Backup Options: Using the SBT Interface We can avoid local storage altogether
Backup directly to your company’s MMS
– Oracle Secure Backup (fully supported)– IBM Tivoli Storage Manager– Symantec NetBackup
shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08 \> --backup-dir=/tmp/backup backup-to-image
shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup \> --sbt-lib-path=/usr/lib/libobk.so \> --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt” \> --backup-dir=/tmp/backup backup-to-image
shell-osb# mysqlbackup --backup-image=sbt:backup-mattprod-2013-09-08 \> --backup-dir=/tmp/backup backup-to-image
shell-tsm# mysqlbackup --backup-image=sbt:my-tsm–backup \> --sbt-lib-path=/usr/lib/libobk.so \> --sbt-environment=“TDPO_OPTFILE=/opt/ibm/tsm/tdpo.opt” \> --backup-dir=/tmp/backup backup-to-image
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17
Backup Options: Partial Backups
Only backup your important schemas and tables
MEB now supports InnoDB transportable tablespaces
shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related \> --backup-dir=/opt/mysql/backups backup
shell# mysqlbackup --use-tts=with-minimum-locking \> --include=employees.* --backup-dir=/opt/mysql/backups backup
shell# mysqlbackup --include=sakila.* --only-innodb-with-frm=related \> --backup-dir=/opt/mysql/backups backup
shell# mysqlbackup --use-tts=with-minimum-locking \> --include=employees.* --backup-dir=/opt/mysql/backups backup
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18
Backup Options: Incremental Backups Only backup what’s changed since your last full backup
shell# mysqlbackup --incremental \> --incremental-base=dir:/opt/mysql/backup/monday \> --incremental-backup-dir=/opt/mysql/backup/tuesday backup
shell# mysqlbackup --incremental \> --incremental-base=history:last_backup --with-timestamp \> --incremental-backup-dir=/opt/mysql/backup backup
shell# mysqlbackup --incremental \> --incremental-base=dir:/opt/mysql/backup/monday \> --incremental-backup-dir=/opt/mysql/backup/tuesday backup
shell# mysqlbackup --incremental \> --incremental-base=history:last_backup --with-timestamp \> --incremental-backup-dir=/opt/mysql/backup backup
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19
Backup Options: Skip Blank & Unused Pages Tell InnoDB not to backup blank or unused pages
Use this in combination with compression for optimal storage efficiency
shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed \> --compress --compress-level=9 --skip-unused-pages backup
shell# mysqlbackup --backup-dir=/opt/mysql/backups/compressed \> --compress --compress-level=9 --skip-unused-pages backup
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20
Restore Options
MEB 3.9 introduces single-step restores
– Even for compressed and single image file backups
This allows us to avoid unnecessary duplication of data
shell# ssh matt@solo "cat /opt/mysql/backups/backup.img" | mysqlbackup \> --backup-dir=/tmp/backup --uncompress --backup-image=- \> --datadir=/var/lib/mysql --innodb_log_group_home_dir=. \> --innodb_log_files_in_group=4 --innodb_log_file_size=2G \> --innodb_data_file_path="ibdata1:1G:autoextend"> copy-back-and-apply-log
shell# ssh matt@solo "cat /opt/mysql/backups/backup.img" | mysqlbackup \> --backup-dir=/tmp/backup --uncompress --backup-image=- \> --datadir=/var/lib/mysql --innodb_log_group_home_dir=. \> --innodb_log_files_in_group=4 --innodb_log_file_size=2G \> --innodb_data_file_path="ibdata1:1G:autoextend"> copy-back-and-apply-log
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21
Insert Picture Here
Dealing With Time Constraints
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22
I Need the Backup Now!
Scheduled maintenance windows may be 30 mins or less
The backup may be needed immediately for a new deployment
One or more schemas may need to be moved to another host ASAP
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23
Backups Methods
Direct FS backup
– Requires mysqld to be shutdown, or at least globally locked
Logical backups (mysqldump)
– VERY slow, consumes a lot of disk space, requires too many locks
MySQL Enterprise Backup (MEB)
– Provides online hot backups for transactional storage engine InnoDB
– Includes many performance and resource usage controls
– Supports partial backups and transportable tablespaces
– Supports incremental backups
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24
MySQL Enterprise Backup
49x faster than mysqldump
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25
MySQL Enterprise Backup
80x faster than mysqldump
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26
MEB Performance Options
Increase buffers for CPU intensive multi-threaded ops: --number-of-buffers
Increase read I/O threads: --read-threads
Increase write I/O threads: --write-threads
Increase CPU processing threads: --process-threads
Increase the maximum memory used for caching: --limit-memory
Don't backup unused pages in InnoDB tables: --skip-unused-pages
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27
MEB Example: Time Comparison 2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM
shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp backup 2>/tmp/log.txtreal 0m34.403suser 0m3.069ssys 0m2.043s
shell# time mysqlbackup --backup-dir=/opt/mysql/backups \> --process-threads=4 --limit-memory=1000 --skip-unused-pages \> --with-timestamp backup 2>/tmp/log.txtreal 0m29.883suser 0m3.121ssys 0m1.889s
shell# time mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp backup 2>/tmp/log.txtreal 0m34.403suser 0m3.069ssys 0m2.043s
shell# time mysqlbackup --backup-dir=/opt/mysql/backups \> --process-threads=4 --limit-memory=1000 --skip-unused-pages \> --with-timestamp backup 2>/tmp/log.txtreal 0m29.883suser 0m3.121ssys 0m1.889s
Cut backup times by 50%+ on enterprise class machines with large datasets
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28
Insert Picture Here
Dealing With Resource Usage Constraints
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29
Why is the Application Slow Right Now?
Backups can be CPU intensive
Backups can be I/O intensive, also causing CPU cycles in IO_Wait
Backups may use various locks within MySQL
So how to take backups without having a noticeable affect…
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30
Backup Options
Limit memory usage: --limit-memory
Limit CPU usage: --process-threads
Sleep X ms after each time we copy 1 block of data: --sleep
Avoid internal locks entirely: --no-locking, --innodb-only
Limit disk I/O: --read-threads, --write-threads
Don’t copy empty and unused pages: --skip-unused-pages
Process limits at OS level: cgroups, taskset, cpulimit, etc.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31
MEB Example: Resource Usage 2G datadir, 1 7.2K HDD, 4 vCPUs , 16G of RAM
shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' \> mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp \> backup 2>&1 | tail -1Exec Time: 0:23.11, CPU: 21%, MEM: 951648KB, IO: 4229360
shell# taskset -c 1 /usr/bin/time -f \> 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup \> --backup-dir=/opt/mysql/backups --limit-memory=50 \> --process-threads=1 --skip-unused-pages --no-locking --sleep=2000 \> --with-timestamp backup 2>&1 | tail -1Exec Time: 4:22.06, CPU: 1%, MEM: 222096KB, IO: 3384064
shell# /usr/bin/time -f 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' \> mysqlbackup --backup-dir=/opt/mysql/backups --with-timestamp \> backup 2>&1 | tail -1Exec Time: 0:23.11, CPU: 21%, MEM: 951648KB, IO: 4229360
shell# taskset -c 1 /usr/bin/time -f \> 'Exec Time: %E, CPU: %P, MEM: %MKB, IO: %O' mysqlbackup \> --backup-dir=/opt/mysql/backups --limit-memory=50 \> --process-threads=1 --skip-unused-pages --no-locking --sleep=2000 \> --with-timestamp backup 2>&1 | tail -1Exec Time: 4:22.06, CPU: 1%, MEM: 222096KB, IO: 3384064
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32
Insert Picture Here
Minimizing Locking Impacts
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33
MEB Locking Related Options
Skip ALL locking: --no-locking (not 100% safe for non-transactional data)
Only backup transactional data: --innodb-only
Incremental backups using the redo log: --incremental-with-redo-log-only
Partial backups with minimal locking: --use-tts=with-minimum-locking
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34
I Thought MEB Backups Were “Hot”?
FLUSH TABLES WITH READ LOCK is used at the end
– Necessary to get a snapshot of all non-transactional data
Can be avoided with various options:
– Don’t take any locks at all: --no-locking
– Only backup InnoDB tables: --only-innodb[-with-frm]
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35
What About the Non-Transactional Data?
mysqlhotcopy – lock and copy each non-transactional table
mysqldump – lock and logically export each table
FS snapshots – copy individual table files out of the snapshot
Manage the backups on an existing MySQL Replication Slave
mysqlbinlog –read-from-remote-server --raww
The amount of non-transactional data will continue to shrink in MySQL 5.7
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36
Recap: Backups With Minimal Impact
Limit shared resource usage: MEB: --limit-memory, --sleep, --[read|write|process]-threads Stream the backup to a remote FS
Backup only as much data as required: MEB: Partial backups with --use-tts=with-minimum-locking MEB: Skip unused pages with --skip-unused-pages (InnoDB only) MEB: Redo log only incremental backups (InnoDB only)
Use 100% non-locking backups: MEB: --only-innodb, --only-innodb-with-frm, --no-locking
– Not 100% safe for .frm files, and for non-transactional data
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37
Insert Picture Here
Handling Data Corruption
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38
Backups with Corruption
Sometimes tables get corrupted!
If production is corrupted, but the backup is fine: Do a full or partial (--with-tts) restore from the backup Apply changes from binary logs
If both the production data, and the backup itself are corrupted: InnoDB
– --innodb_force_recovery– SELECT INTO OUTFILE
MyISAM– myisamchk
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39
Backups with Corruption
What to do if: You have only full MEB backup, but need to restore single table You can not setup spare instance You can not use TTS
MyISAM Simply copy .frm, .MYI, and .MYD files
InnoDB Things get harder Each table needs to have the same table ID in the shared tablespace, in
both the backup and production server
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40
Backups with Corruption:Full Backup and InnoDB
If you can setup separate instance and restore full backup there
Do it! If you can use TTS
– Backup needed table using MEB with option --use-tts– Restore it
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41
Backups are Corrupted:Full Backup and InnoDB
If you can not!
The problem Table ID in the InnoDB shared tablespace must be same in the backup and
production server– They are if:
– this is a backup of the same production server– you originally cloned this server from this backup and– You use option –innodb-file-per-table
– Simply copy the .ibd file
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42
Backups are Corrupted:Full Backup and InnoDB
If table ID is not the same and you use option –innodb-file_per_table
Recover manually– Simulate the internal InnoDB table counter
– Start spare instance– Create, then drop table with the same structure as the original one
until counter increases to appropriate value– Import backed up tablespace into spare instance– Copy .ibd file to production server
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43
Backups are Corrupted:Full Backup and InnoDB
But how will you find what to type in CREATE TABLE statement? mysqlfrm, the FRM reader will help you
$ mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333# Starting the spawned server on port 3333 ... done.# Reading .frm files## Reading the city.frm file.## CREATE statement for city.frm:#
CREATE TABLE `test1`.`city` (…
$ mysqlfrm --basedir=/usr/local/bin/mysql test1:city.frm --port=3333# Starting the spawned server on port 3333 ... done.# Reading .frm files## Reading the city.frm file.## CREATE statement for city.frm:#
CREATE TABLE `test1`.`city` (…
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44
Insert Picture Here
Conclusion
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45
Conclusion
Backups are a must
Production constraints are a fact of life
Difficult problems can be solved with MySQL Enterprise Backup
MySQL Support is there to help with your specific problems
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46
References
Documentation:
http://dev.mysql.com/doc/mysql-enterprise-backup/3.9/en/index.html
http://dev.mysql.com/doc/workbench/en/mysqlfrm.html
https://blogs.oracle.com/mysqlenterprisebackup/
MySQL Enterprise Backup Trial Downloads:
https://edelivery.oracle.com/
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47
Graphic Section Divider
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48