Upload
nilnandan-joshi
View
346
Download
7
Tags:
Embed Size (px)
DESCRIPTION
Its related to Percona utilities.
Citation preview
Percona Xtrabackup
Nilnandan JoshiSupport EngineerPercona LLC 2014
www.percona.com
Introduction
Percona Xtrabackup provides a fully open-source, free, high-performance, non-blocking backup system for InnoDB or XtraDB tables
It can also backup MyISAM, Archive, Merge, and other SQL-level objects
It is a reliable, widely-used alternative to Oracle's MySQL Enterprise Backup
www.percona.com
Features
Full Binary Backups Incremental backups Open Source (GPL)
Online (non-blocking)* Parallel backups & compression Cost: Free
InnoDB & MyISAM Streaming backups MySQL compatibility
Compressed backups Compact backups MySQL Forks' support
Point-in-time recovery
support
Export individual tables; restore
to a different server
MySQL replication &
Galera support
Incremental backups LRU backups Community Help
Throttling OS buffer optimizations Commercial Support
Partial Backups Encrypted backups Easy migration
www.percona.com
Compatibility
Versions <= 2.0 are compatible with :
Oracle MySQL 5.0, 5.1, 5.5 and 5.6
Equivalent versions of Percona Server, MariaDB, and Drizzle
Version 2.1 is compatible with:
Oracle MySQL 5.1 (plugin version [1] ), 5.5, 5.6
Equivalent versions of Percona Server, MariaDB, and Drizzle
[1] The InnoDB Engine plugin 1.0 is distributed -but not enabled- by default. See:
http://dev.mysql.com/doc/innodb-plugin/1.0/en/innodb-plugin-installation.html
www.percona.com
xtrabackup vs. innobackupex
Percona XtraBackup includes two main executables
xtrabackup is the low level binary that handles the creation and consistency checking of InnoDB backups.
innobackupex is a higher-level Perl wrapper for xtrabackup and is also responsible for the rest of the operations, including the backup of other engines
You usually want to use innobackupex
www.percona.com
xtrabackup vs. innobackupex
www.percona.com
Installation
The most straightforward way is installing the Percona apt or yum repositories
Then install the package with:- yum install percona-xtrabackup- aptitude update && aptitude install percona-
xtrabackup
Other installation options can be found at http://www.percona.com/downloads/XtraBackup/
www.percona.com
Requirements and Limitations
Only fully supported on Linux machines
innobackupex is a Perl script Package should take care of all
dependencies (Perl interpreter, mysql driver, other libraries)
Local access to the MySQL datadir is required
READ, WRITE and EXECUTE filesystem privileges are needed
This makes xtrabackup incompatible with RDS or some web hosting providers
www.percona.com
How Percona XtraBackup Works: Full Backup
www.percona.com
How Percona XtraBackup Works: Full Restore
www.percona.com
Command Line Execution
Minimal options for a full backup:
# innobackupex /path/to/backup/
• Actual backup is stored in a timestamped subdirectory.
• It reads the datadir and innodb paths from my.cnf
• A more complete set of options:
# innobackupex --host=localhost --user=root --password=secret /tmp
www.percona.com
What it looks like
root@nilnandan-Dell-XPS:/var/lib/mysql# sudo innobackupex --user=root --password=root /home/nilnandan/backup
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oyand Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published underthe GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:http://www.percona.com/xb/p
140709 13:52:04 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).140709 13:52:04 innobackupex: Connected to MySQL server140709 13:52:04 innobackupex: Executing a version check against the server...140709 13:52:04 innobackupex: Done.
www.percona.com
Target Directory
root@nilnandan-Dell-XPS:~# ls -al /home/nilnandan/backup/2014-07-09_13-52-04total 77868drwxr-xr-x 6 root root 4096 Jul 9 14:28 .drwxrwxr-x 4 nilnandan nilnandan 4096 Jul 9 14:21 ..-rw-r--r-- 1 root root 356 Jul 9 13:52 backup-my.cnfdrwx------ 2 root root 4096 Jul 9 13:52 dbtest-rw-r----- 1 root root 79691776 Jul 9 13:52 ibdata1drwx------ 2 root root 4096 Jul 9 13:52 mysqldrwxr-xr-x 2 root root 4096 Jul 9 13:52 performance_schemadrwx------ 2 root root 4096 Jul 9 13:52 test-rw-r----- 1 root root 95 Jul 9 13:52 xtrabackup_checkpoints-rw-r--r-- 1 root root 528 Jul 9 13:52 xtrabackup_info-rw-r----- 1 root root 2560 Jul 9 13:52 xtrabackup_logfileroot@nilnandan-Dell-XPS:~#
www.percona.com
Files in Target Directory
Hierarchy of files and directories mirroring the original database structure.
backup-my.cnf: Not a backup of the server configuration. It only contains the minimal InnoDB settings at the time the backup to execute the --apply-log phase
xtrabackup_info: Contains all informations about xtrabackup binary, server version, start and end time of backup, binlog position etc.
xtrabackup_checkpoints: Metadata about the backup (type of backup, lsn, etc.)
xtrabackup_logfile: Data needed for the --apply-log phase
www.percona.com
Preparation Phase
Before the backup directory can be used, we must:
Make sure that the InnoDB tablespaces have consistent,non-corrupt data
Create new transaction logs
innobackupex executes xtrabackup twice:
First, to do a controlled crash recovery, applying the log entries with xtrabackup's embedded InnoDB. This redoes all written committed transactions and undoes uncommitted ones.
Second, to create the empty logs
www.percona.com
Preparation Phase (cont)
Only xtrabackup/innobackupex and the backup directory are needed
It can be done on different machine than the original server (we have the xtrabackup binary and the innodb configuration parameters)
Generally, you do it just before restore. If you only do full backups and want to minimize the MTTR (Mean Time To Recover), you can prepare immediately after backup
www.percona.com
Command Line Execution
The simplest form:
$ innobackupex --apply-log/media/backups/2017-07-08_13-22-12
You may specify a memory parameter:
$ innobackupex –apply-log --use-memory=4G/media/backups/2017-07-08_13-22-12
this increases the available buffer pool for the recovery phase (100M by default), to improve performance
www.percona.com
How It Looks Like
nilnandan@nilnandan-Dell-XPS:~/backup$ sudo innobackupex --apply-log 2014-07-09_13-52-04
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oyand Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published underthe GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:http://www.percona.com/xb/p
IMPORTANT: Please check that the apply-log run completes successfully. At the end of a successful apply-log run innobackupex prints "completed OK!".
140710 10:06:57 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/home/nilnandan/backup/2014-07-09_13-52-04/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/nilnandan/backup/2014-07-09_13-52-04 --tmpdir=/tmp
www.percona.com
Restore
After --apply-log, we have a consistent full raw backup
We can just move it back to the datadir with standard OS commands once the server is shut down
As a physical backup, restoration is as fast as you can copy the directory contents to the local filesystem
There are specific commands for restoration in XtraBackup
www.percona.com
Copy Back
The integrated command to restore a backup is:
# innobackupex --copy-back /media/backups/2017-07-08_13-22-12
It complains if the datadir, as read from the config file, is not empty [1] (it will not overwrite existing data)
Only partial restores can be done to a running server
xtrabackup_* files are not needed
[1] We can override this behavior with the option --force-non-empty-directories , but it will never overwrite existent files
www.percona.com
What It Looks Like
root@nilnandan-Dell-XPS:~# sudo innobackupex --copy-back /home/nilnandan/backup/2014-07-09_13-52-04/...innobackupex: Starting to copy files in '/home/nilnandan/backup/2014-07-09_13-52-04'innobackupex: back to original data directory '/var/lib/mysql'innobackupex: Copying '/home/nilnandan/backup/2014-07-09_13-52-04/xtrabackup_info' to '/var/lib/mysql/xtrabackup_info'innobackupex: Creating directory '/var/lib/mysql/mysql'innobackupex: Copying '/home/nilnandan/backup/2014-07-09_13-52-04/mysql/time_zone_leap_second.frm' to ...innobackupex: Starting to copy InnoDB log filesinnobackupex: in '/home/nilnandan/backup/2014-07-09_13-52-04'innobackupex: back to original InnoDB log directory '/var/lib/mysql'innobackupex: Copying '/home/nilnandan/backup/2014-07-09_13-52-04/ib_logfile1' to '/var/lib/mysql/ib_logfile1'innobackupex: Copying '/home/nilnandan/backup/2014-07-09_13-52-04/ib_logfile0' to '/var/lib/mysql/ib_logfile0'innobackupex: Finished copying back files.
140710 11:04:12 innobackupex: completed OK!root@nilnandan-Dell-XPS:~#
www.percona.com
Move Back
As an alternative to --copy-back, we can also do:
# innobackupex --move-back /media/backups/2017-07-08_13-22-12
This may be faster or more convenient if the backup is already on the same partition or don't have enough free space
www.percona.com
Restoring Permissions
After restore, as file permissions are preserved, youwill probably have to fix file ownership:
# chown -R mysql:mysql /var/lib/mysql
If MySQL cannot write to the data directory or some of it files, it will refuse to start
www.percona.com
Incremental Backup
Differential & Incremental Backups
Differential backup: saves only the difference in the data since the last full backup
Incremental backup: saves only the difference in the data since the last backup (full or another incremental)
Xtrabackup can perform both, which can lead to an important save in space (and potentially, time), specially if the backups are very frequent or the database is very static
www.percona.com
Incremental Backup(cont..)
Differential and incremental backups work thesame way with xtrabackup:
Every page stores the Log Sequence Number (LSN) when it was last changed
Only pages that have a higher LSN than the base backup (full or incremental) are copied
www.percona.com
Incremental Backup(cont..)
www.percona.com
Incremental Backup(cont..)
Restoration of Incremental Backups
On prepare time, the new pages have to be applied in order to the full backup before restore
This is the reason why you want to wait to fully apply the log just before restore time
Uncommitted transactions must not be undone, as they may have finished on a subsequent incremental backup
Intermediate backups have to apply changes with the option --redo-only
www.percona.com
Incremental Backup(cont..)
www.percona.com
Partial Backups
When using xtrabackup (or innobackupex), we can filter out some of the tables, effectively copying only some databases or tables
Two main restrictions apply for InnoDB tables:
Original database must use innodb_file_per_table, as xtrabackup works at filesystem level
Recovering individual tables is only possible for MySQL 5.6. Previous versions require Percona XtraDB extensions with tablespace import enabled
www.percona.com
Partial Backups (cont..)
Command Line Options for Segmented Backups:
--include : Matches the fully qualified name (database.table) against a regular expression
--databases : Matches a list of databases (or tables) separated by spaces.
--tables-file : Matches a list of tables, one per line, stored on the specified file
www.percona.com
Partial Backups (cont..)
Examples
Backup only the test database:
# innobackupex --include=^test[.] /data/backups
Backup all except the mysql database*:$ mysql -e "SELECT concat(table_schema, '.',table_name) from information_schema.tables WHEREtable_schema NOT IN ('mysql', 'performance_schema','information_schema')" > /tmp/tables
# innobackupex --tables-file=/tmp/tables /data/backups
www.percona.com
Streaming
Xtrabackup allows the streaming of the backup filesto the UNIX standard output
This allows to post-process them or send them to aremote location while the backup is ongoing
www.percona.com
Streaming (cont..)
The stream option accepts two parameters:
--stream=tarIt sends all files to the standard output in the tararchive format (single file without compression)
--stream=xbstreamThe standard output will using the new includedxbstream format, which allows simultaneouscompression, encryption and parallelization
A temporal file path is still required
www.percona.com
Streaming (cont..)
Using tar:
# innobackupex --stream=tar /tmp > backup.tar# innobackupex --stream=tar /tmp | \bzip2 - > backup.tar.bz2
To extract, the tar -i option (ignore zeros) must beused:
# tar -xivf backup.tar# tar -jxif backup.tar.bz2
www.percona.com
Streaming (cont..)
Using xbstream
# innobackupex --stream=xbstream /tmp > \backup.xbstream
To extract, use the xbstream binary provided withPercona Xtrabackup:
# xbstream -x < backup.xbstream -C /root
www.percona.com
Compression
We can use --compress to produce quicklz-compressed files:
# innobackupex --compress /media/backups
To decompress, we can use --decompress orqpress* to decompress the individual .qp filesmanually:
# innobackupex --decompress \/media/backups/2017-07-08_13-22-12
# find . -name "*.qp" -execdir qpress -d \{\} . \;
www.percona.com
Compression and xbstream
The --compress flag can be combined withxbstream (not with tar):
# innobackupex --compress --stream=xbstream /tmp \> /media/backups/backup.xbstream
Before preparation, we must use first xbstream,then decompress:
# xbstream -x -C /media/backups < backup.xbstream# innobackupex --decompress /media/backups
www.percona.com
Parallel Copy and Compression
If using xbstream, we can use --parallel to archiveseveral files at the same time:
# innobackupex --parallel=4 --stream=xbstream \/tmp > /media/backups/backup.tar
If we are using compression, we can parallelize theprocess for each file with –compress-threads
# innobackupex --compress --compress-threads=8 \--stream=xbstream /tmp > /media/backups/backup.tar
www.percona.com
Remote Backups
xtrabackup can send automatically the backup filesto a remote host by redirecting the output toapplications like ssh or netcat:
# innobackupex --stream=tar /tmp \| ssh user@host "cat - > \/media/backups/backup.tar"
# ssh user@host "( nc -l 9999 > \/media/backups/backup.tar & )" \&& innobackupex --stream=tar /tmp \| nc host 9999
www.percona.com
Throttling Remote Backups
pv (pipe viewer) utility can be useful for bothmonitoring and limiting the bandwidth of the copyprocess
The following limits the transmission to 10MB/s:# innobackupex --stream=tar ./ \| pv -q -L10m \| ssh user@desthost \"cat - > /media/backups/backup.tar"
www.percona.com
Compact Backups
In order to reduce backup size, we can skip the copyof secondary index pages:
# innobackupex --compact /media/backups
The space saved will depend on the size of thesecondary indexes. For example:
#backup size without --compact2.0G 2018-02-01_10-18-38#backup size taken with --compact option1.4G 2018-02-01_10-29-48
www.percona.com
Preparing Compact Backups
The downside of compact backups is that theprepare phase will take longer, as the secondarykeys have to be regenerated ( --rebuild-indexes ):
# cat xtrabackup_checkpointsbackup_type = full-backupedfrom_lsn = 0to_lsn = 9541454813last_lsn = 9541454813compact = 1
# innobackupex --apply-log --rebuild-indexes/media/backups/2017-07-08_13-22-12
www.percona.com
Rebuilding Indexes
The --rebuild-threads option can be used to speed up the recreation of the keys, by doing it in parallel:
# innobackupex --apply-log \ --rebuild-indexes --rebuild-threads=8 \ /media/backups/2017-07-08_13-22-12
Rebuilding indexes has the side effect of defragmenting them, as they are done in order. We
can rebuild regular full-backups, too.
www.percona.com
Encrypted Backups
Xtrabackup has support for symmetric encryptionthrough the libgcrypt library for both local andxbstream backups
Before using encryption, a key must be generated.For example, to create an SHA256 key: # openssl enc -aes-256-cbc -pass pass:mypass -P -md
sha1 salt=9C8CE740FD0FE6AD key=0A632731C0EABA6131B6683068E04629524FC8FC1E0F69037D5ED03AF126BD0B
iv =03E872849A8DA83647A5FCEFDF1C68F2
* Please note that specifying passwords on the command line input is considered insecure.
www.percona.com
Encrypted Backups (cont..)
We can now use the previous value of iv as ourencryption key:$ innobackupex --encrypt=AES256 \ --encrypt-
key="03E872849A8DA83647A5FCEFDF1C68F2" \/data/backups
Options:--encryption : Defines the algorithm to be used
(supports AES128, AES192 and AES256)--encrypt-key : Defines the encryption key to use
directly on the command line.--encrypt-key-file : Defines the encryption key by
pointing to a file that contains it. This is recommended over the encrypt-key option
www.percona.com
Speeding up Encryption
Parallel encryption is possible by encrypting morethan one file at a time (--parallel) and using more than one
thread for each file (--encrypt-threads):
# innobackupex --encrypt=AES256 \--encryption-key-file=/media/backups/keyfile--stream=xbstream --encrypt-threads=4 /tmp \> /media/backups/backup.xbstream
We can also change the buffer for each encryption thread with --encrypt-chunk-size (default 64K)
www.percona.com
Decrypting Backups
Before preparing it, the backup must be unarchived (ifxbstream was used) and decrypted. We can use --decrypt or
do it for each individual .xbscript file using xbscript (included with xtrabackup):
# innobackupex --decrypt=AES256 \--encrypt-key="A1EDC73815467C083B0869508406637E" \/data/backups/2013-08-01_08-31-35/
# for i in `find . -iname "*\.xbcrypt"`; do \xbcrypt -d --encrypt-key-file=/media/backups/filekey \--encrypt-algo=AES256 < $i \> $(dirname $i)/$(basename $i .xbcrypt) && rm $i; done
--parallel can speed up the process as well
www.percona.com
TROUBLESHOOTING ANDPERFORMANCE
Locking Issues By default, innobackupex will lock the tables (FTWRL)
during the last phase of the backup You can use --no-lock if you are only backing upInnoDB tables and no DDL are executing on that lastPhase* You can use --no-lock if you backup other engines if no
DDLs are executed, and non-innodb tables are not modified (e.g. users are created on the mysql database)*\
You can use --rsync to minimize the copy time whenlocked
* Please note that --no-lock also makes impossible to obtain a reliable binary log coordinates.
www.percona.com
TROUBLESHOOTING ANDPERFORMANCE
File Not Found
2017-07-08 12:56:56 7f0dfe4737e0 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.2017-07-08 12:56:56 7f0dfe4737e0 InnoDB: File name ./ibdata12017-07-08 12:56:56 7f0dfe4737e0 InnoDB: File operation call: 'open'returned OS error 71.2017-07-08 12:56:56 7f0dfe4737e0 InnoDB: Cannot continue operation.innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 389.
Set a correct datadir option on /etc/my.cnf
www.percona.com
TROUBLESHOOTING ANDPERFORMANCE
Permission Denied
InnoDB: read-write can't be opened in ./ibdata1 modextrabackup: Could not open or create data files.[...]xtrabackup: error: xb_data_files_init() failed witherror code 1000innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 389.innobackupex: Error: Failed to create backup directory /backups/2017-07-08_12-19-01: Permission denied at /usr/bin/innobackupex line 389.
Use the Linux root user or any other account with read/write permissions on datadir and on target-dir
www.percona.com
TROUBLESHOOTING ANDPERFORMANCE
MySQL Does Not Start AfterRecovery
Starting MySQL server... FAILEDThe server quit without updating PID file
Did you change file permissions of the files to the mysql user?
Did you remember to apply the log before restart?Failing to do so can also corrupt the backup.
Did you restore an appropriate my.cnf file? Previousversions of MySQL (< 5.6) do not start if transaction log
files have a different size than specified/default
www.percona.com
TROUBLESHOOTING ANDPERFORMANCE
The Backup Takes Too Long
As MySQL performs physical backups, the speedshould be close to to a filesystem copy. If IO impactis not a concern, we can speed it up by:
Using parallelization ( --parallel , --compression-threads , --encrypt-threads , --rebuild-threads ) Using compression ( --compress ) if the extra CPUcompensates the lower bandwidth required Using incremental backups (specially with Perconaextensions) to perform rolling full backups
[email protected]@nilnandan