Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on

Using ZFSTM Snapshots With

Zmanda Recovery Manager for MySQLTM

on OpenSolarisTM 2008.11 OS

Caryl Takvorian

May 2009

Sun Microsystems, Inc.

This article was published by BigAdmin at:http://www.sun.com/bigadmin/features/articles/zrm_mysql_zfs.jsp

To keep track of the latest content published by BigAdmin, subscribe to the BigAdmin newsletter: http://www.sun.com/bigadmin/newsletter/.

http://www.sun.com/bigadmin/features/articles/zrm_mysql_zfs.jsp

http://www.sun.com/bigadmin/newsletter/

Copyright © 2009 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. This distribution may include materials developed by third parties.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open is a registered trademark of X/Open Company, Ltd.

AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation in the United States and other countries.

Sun, Sun Microsystems, the Sun logo, MySQL, OpenSolaris, Solaris, Sun BluePrints, Sun Fire, SunSolve, and ZFS are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries.

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

2 Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on OpenSolaris 2008.11 OS

Table of ContentsIntroduction................................................................................................................................................5Architecture Overview...............................................................................................................................6

OS Configuration..................................................................................................................................6The ZFS File System.............................................................................................................................6

Zpool Configuration.........................................................................................................................7ZFS Configuration............................................................................................................................7

Configuration of Zmanda Recovery Manager 3.0 for MySQL.............................................................8Installing Dependencies Using pkgadd...........................................................................................9Installing Dependencies Using IPS..................................................................................................9Installing ZRM...............................................................................................................................10

Types of Backup Available to ZRM...................................................................................................11Raw-Copy Backup..........................................................................................................................11Logical Backup...............................................................................................................................11Raw-Quick (No-Copy) Backup .....................................................................................................11

Performance Testing Setup......................................................................................................................13System Setup.......................................................................................................................................13MySQL Software Release...................................................................................................................13OpenSolaris Operating System Release..............................................................................................13Zpool Configuration............................................................................................................................13ZFS Configuration...............................................................................................................................14ZRM Software Release.......................................................................................................................14db_STRESS Load Generator...............................................................................................................14

1. Creating the test databases..........................................................................................................142. Creating test tables in pool2/mysql............................................................................................143. Generating the dataset.................................................................................................................154. Loading the dataset into the tables..............................................................................................155. Creating users.............................................................................................................................156. Creating a test scenario ..............................................................................................................15

Testing Methodology..........................................................................................................................15Performance Test Results.........................................................................................................................17

1. MyISAM Results with Idle Database..............................................................................................172. InnoDB Results with Idle Database................................................................................................173. MyISAM Results with a 15 Percent Load on the Database ...........................................................194. InnoDB Results with a 15 Percent Load on the Database ..............................................................205. MyISAM Results with a 90 Percent Load on the Database ...........................................................216. InnoDB Results with a 90 Percent Load on the Database ..............................................................22

Product Information.................................................................................................................................23OpenSolaris Operating System...........................................................................................................23The ZFS File System...........................................................................................................................23The Sun Fire X4500 Server.................................................................................................................23Zmanda Recovery Manager 3.0 for MySQL.......................................................................................23

Summary..................................................................................................................................................24About the Author ....................................................................................................................................24For More Information..............................................................................................................................25Licensing Information..............................................................................................................................27


IntroductionThe objective of this report is to show how to deploy Zmanda Recovery Manager 3.0 for MySQL™ on the OpenSolaris™ 2008.11 OS and how ZRM performs on a Sun Fire™ X4500 system using the ZFS™

file system. In particular this article demonstrates how ZRM takes advantage of ZFS snapshot technology to offer a best-in-class open source data-protection solution for MySQL.

It is intended to be read in conjunction with the article Using Sun Fire X4540 Server With Zmanda Recovery Manager 2.2 for MySQL Database by Thomas Hanvey (Sun Microsystems) and Dmitri Joukovski and Ken Crandall (Zmanda), October 2008, which is available at http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp.

All the steps described in this document were carried out on a single Sun Fire X4500 system which acted as the ZRM backup server and the MySQL database server.

For more information on the OpenSolaris 2008.11 OS and Zmanda technology, see the Product Information section at the end of this article. For further details on ZRM Enterprise, see: http://zmanda.com/zrm-mysql-enterprise.html.

Zmanda Recovery Manager for MySQL is available for download from the Zmanda network: http://network.zmanda.com/.


http://network.zmanda.com/

http://zmanda.com/zrm-mysql-enterprise.html

http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp

Architecture Overview

OS ConfigurationFor this guide, ZRM was deployed on a Sun Fire X4500 server running the OpenSolaris 2008.11 release.

For more information on the Sun Fire X4500 server, see the Product Information section at the end of this article.

The system used during this test is a server with two dual-core AMDTM Opteron 285 processors running at 2593 MHz with 16 Gbyte of RAM and 48 SATA drives of 512 Gbyte each, for a total of 24 Tbyte of storage capacity.

The two boot drives are mirrored using ZFS volume management capabilities and the remaining 46 drives are available for backup data.

The ZFS File SystemFor more information on ZFS, see the Product Information section at the end of this article.


Figure 1: Architecture Overview

One of the most interesting features of ZFS when combined with backup technologies is its snapshot capabilities. A snapshot is a read-only copy of a file system or volume which can be created very quickly. Thanks to ZFS copy-on-write characteristics, snapshots are essentially a free operation, which initially consume no additional space within the pool. However, as data within the active dataset changes, the snapshot consumes space by continuing to reference the old data and so prevents the data from being freed back to the pool.

ZFS snapshots include the following features:

● Persist across system reboots.

● Use no separate backing store. Snapshots consume disk space directly from the same storage pool as the file system from which they were created.

● Recursive snapshots are created quickly as one atomic operation. The snapshots are created together (all at once) or not created at all. The benefit of atomic snapshot operations is that the snapshot data is taken at one consistent time, even across descendent file systems.

Zpool Configuration

For the purposes of this guide, the storage available on the Sun Fire X4500 server was split into two pools:

● A backup pool made of 24 drives and two spares in a RAIDZ2 configuration. This provided 11 Tbyte of capacity for backups with a very high level of protection against drive failure and performance sufficient to handle 300 to 350 Mbyte/sec of backup traffic over the network.

● A mysql pool made of 12 drives and two spares in a RAIDZ2 configuration was used to store the MySQL databases.

First, we created the RAIDZ2 pool zmanda with six drives. Then, we added the other drives by groups of six and verified at the end that we ended up with a pool with 10.9 Tbyte of physical space.

Here are zpool creation commands for pool zmanda:# zpool create -f zmanda raidz2 c3t1d0 c4t1d0 c6t1d0 c7t1d0 c8t1d0 c9t1d0# zpool add zmanda raidz2 c3t2d0 c4t2d0 c6t2d0 c7t2d0 c8t2d0 c9t2d0# zpool add zmanda raidz2 c3t5d0 c4t5d0 c6t5d0 c7t5d0 c8t5d0 c9t5d0# zpool add zmanda raidz2 c3t6d0 c4t6d0 c6t6d0 c7t6d0 c8t6d0 c9t6d0# zpool add zmanda spare c8t0d0 c9t0d0

We used a similar command to create the pool pool2.

Here are the zpool creation commands for pool pool2:# zpool create -f pool2 raidz2 c3t3d0 c4t3d0 c6t3d0 c7t3d0 c8t3d0 c9t3d0 # zpool add -f pool2 raidz2 c3t7d0 c4t7d0 c6t7d0 c7t7d0 c8t7d0 c9t7d0 # zpool add -f pool2 spare c8t4d0 c9t4d0

Finally we can check the ZFS pools just created:

# zpool listNAME SIZE USED AVAIL CAP HEALTH ALTROOT pool2 5.44T 225K 5.44T 0% ONLINE - rpool 464G 13.3G 451G 2% ONLINE - zmanda 10.9T 252K 10.9T 0% ONLINE -


ZFS Configuration

Next, we need to create the ZFS file system which will be used by ZRM and MySQL. We will need three different ZFS file systems:

● zmanda/zrm to hold the ZRM backups

● zmanda/logs to hold the MySQL binlogs

● pool2/mysql to hold the MySQL data files

So, first let's create the ZFS file system where ZRM will store the backup images, and mount it in the default location expected by ZRM:

# zfs create -o mountpoint=/var/lib/mysql-zrm zmanda/zrm

Then, create the ZFS file system where the MySQL data files will be stored, and the third file system to store the MySQL log files.# zfs create zmanda/logs# zfs create pool2/mysql

Finally we can list the file systems available to us:# zfs list NAME USED AVAIL REFER MOUNTPOINT pool2 1.35T 2.22T 1023G /pool2 pool2/mysql 356G 2.22T 356G /pool2/mysql rpool 185G 272G 60K /rpool rpool/ROOT 150G 272G 18K legacy rpool/ROOT/opensolaris-1 150G 272G 121G legacy rpool/dump 7.95G 272G 7.95G - rpool/export 19.1G 272G 19K /export rpool/export/home 19.1G 272G 18.9G /export/home rpool/swap 8.16G 280G 216M - zmanda 50.0G 7.08T 38.0K /zmanda zmanda/logs 815K 7.08T 815K /zmanda/logs zmanda/zrm 87.9K 7.08T 87.9K /var/lib/mysql-zrm

Configuration of Zmanda Recovery Manager 3.0 for MySQLFor more information on ZRM, see the Product Information section at the end of this article.

The Sun Fire X4500 server is configured as a ZRM server with Zmanda Management Console (ZMC). To install ZRM and ZMC, follow these overall steps, which are explained in more detail in subsequent sections.

1. From the Zmanda Network web site (which requires registration), download the following:

● ZRM installer binary for Solaris 10 systems with Intel® or AMD Opteron processors: ZRM-enterprise-3.0-installer-intel.bin

● Prerequisite packages: solaris-10-zrm-dependency-intel.tar.gz

1. Extract the prerequisite packages, solaris-10-zrm-dependency-intel.tar.gz, and install all the included packages.

2. Extract the binary package and run the Zmanda Rapid Installer to install ZRM.

3. After the installation is complete, the ZMC Welcome Page is displayed, and you can begin configuring backup sets.


There are two ways of installing the required dependencies, using pkgadd or using Image Packaging System (IPS).

Installing Dependencies Using pkgadd

The first option is to download the dependency package provided by Zmanda from the Zmanda Network at http://network.zmanda.com. The required dependency packages are provided in a compressed tar archive file, solaris-10-zrm-dependency-intel.tar.gz.

Extract all the packages:# gtar zxvf solaris-10-zrm-dependency-intel.tar.gz

The dependency packages provided are as follows:

● SMCcoreu● CSWgcc3corert● CSWgcc3g++rt● SMCgpgme● SMCgrep● SMClgcc346● SMClgpger● SMCliconv● SMClintl● SMCpcre● SMCsudo● SMCtar

These packages are also available from Sunfreeware at http://sunfreeware.org/ and from Blastwave at http://blastwave.org.

Install each package using pkgadd:# pkgadd -d <pkg_name>

Installing Dependencies Using IPS

The second option is to use the new OpenSolaris Image Packaging System. The most popular Solaris packaging repositories such as http://blastwave.org and http://sunfreeware.org have already converted the software they have on offer to the new IPS package format, and setting OpenSolaris to use their repository is one single step.

Add the repository to the list of authorized package repositories: # pkg set-authority -O http://blastwave.network.com:10000 blastwave

Let's make sure that everything is OK: # pkg authority -H AUTHORITY URL blastwave http://blastwave.network.com:10000/ opensolaris.org (preferred) http://pkg.opensolaris.org:80/

Now we can start installing the CSW packages required by ZRM.


http://sunfreeware.org/

http://blastwave.org/

http://blastwave.org/

http://sunfreeware.org/


Note that the names of the Blastwave packages available for OpenSolaris IPS start with IPS instead of the traditional CSW; for example, IPScommon instead of CSWcommon.

# pkg install IPScommon DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.00/0.00 PHASE ACTIONS Install Phase 146/146 PHASE ITEMS Reading Existing Index 9/9 Indexing Packages 1/1

To show the ease of use of the OpenSolaris IPS packaging system, we'll install the complete list of required packages in one single command:

# pkg install IPScurl IPSbzip2 IPSbdb4 IPScurlrt IPSexpat IPSfindutils IPSgfile IPSggettext IPSgtar IPSiconv IPSlibidn IPSlibnet IPSoldaprt IPSossl IPSossldevel IPSosslrt IPSosslutils IPSreadline IPSsasl IPSsunmath IPSzlib IPSglib2 IPSgcc3g++rt IPSmtx

Once finished, let's check that we have everything. (Once installed, the Blastwave packages use their old CSW names.)

# pkginfo | grep CSW application CSWbdb4 berkeleydb4 - Embedded database libraries and utilities application CSWbzip2 bzip2 - a high-quality block-sorting file compressor system CSWcommon common - common files and dirs for CSW packages application CSWcurl curl - Library and tool for common Internet protocols application CSWcurlrt curlrt - Library for common Internet protocols Runtime application CSWexpat expat - XML Parser Toolkit system CSWfindutils findutils - A set of utilities for searching a file system application CSWgcc3corert gcc3corert - GNU C Compiler Run Time application CSWgcc3g++rt gcc3g++rt - GNU C++ Compiler Run Time system CSWgfile fileutils - GNU file utilities system CSWggettext ggettext - GNU gettext application CSWglib2 glib2 - the low-level core library for GTK+ and GNOME application CSWgtar gtar - GNU tape archiver system CSWiconv libiconv - GNU iconv library application CSWlibidn libidn - Implementation of the Stringprep, Punycode and IDNA system CSWlibnet libnet - the libnet packet construction library application CSWmtx mtx - SCSI Media Changer and Backup Device Control application CSWoldaprt openldap_rt - OpenLDAP runtime libraries (oldaprt) application CSWossl openssl - Openssl meta package application CSWossldevel openssl_devel - Openssl development support application CSWosslrt openssl_rt - Openssl runtime libraries application CSWosslutils openssl_utils - Openssl binaries and related tools system CSWreadline readline - library to enable interactive line editing application CSWsasl sasl - Simple Authentication and Security Layer system CSWsunmath libsunmath - Sun maths library application CSWzlib zlib - Zlib Data Compression Library


Installing ZRM

Installing Zmanda Recovery Manager 3.0 for MySQL is a simple task of running the Zmanda Rapid Installer which contains the ZRM Server, the ZRM Client, as well as the Zmanda Management Console (ZMC) web user interface.

As we are connected to the Sun Fire X4500 through an SSH connection, we'll start the Rapid Installer in text mode, as such:# ./ZRM-enterprise-3.0-installer-intel.bin –-mode text

Once completed, the following software components are installed:

● The ZRM 3.0 for MySQL software

● Zmanda Management Console with MySQL, PHP, Apache, PHPmyadmin, Perl and various Perl modules and their dependencies. These are installed under the /opt/zmanda/zrm and /opt/zmanda/common directories.

To start or stop ZRM run: # /etc/init.d/zmc_zrm_init start | stop | status

Types of Backup Available to ZRMDepending on the datastore selected, ZRM offers a number of different types of backup.

Raw-Copy Backup

A raw-copy backup makes a copy of the binary disk image of databases stored on non-transactional storage engines by using mysqlhotcopy. It works on live databases, but the whole database (or the tables being backed up if only a few tables are selected) is locked for the duration of the backup. Although raw backups can be restored more quickly than logical backups, they can only be restored to the same version of MySQL server on the same platform architecture.

If any of the databases or tables are stored on a transactional storage engine (such as InnoDB), a logical mysqldump backup is taken instead.

Logical Backup

Choosing this option forces a mysqldump backup that copies MySQL binary logs regardless of the storage engine. MySQL binary logs track and save all database server transactions as a list of SQL statements. To implement a logical backup strategy, binary logging must be enabled on the MySQL server, and a path to the log files must be supplied (the default is /var/lib/mysql) in the Binary Log Path field.

Logical backup method works with all MySQL storage engines except the MySQL cluster NDB storage engine. Logical backups can also be restored to any platform architecture or database that supports SQL.

Because logical backups require a read lock on the database(s) or tables being backed up, they can have a greater impact on the applications using the MySQL database. Logical backups also result in increased restore times, as restoring the data is accomplished by re-playing the transactions against the target database instead of just copying files.


Raw-Quick (No-Copy) Backup

As its name suggests, this type of backup is the quickest on offer, but it is only available with raw backups using snapshots (offered by ZFS and other filesystems). When the Quick (No-copy) snapshot option is enabled, ZRM for MySQL uses the snapshot itself as the backup rather than transferring the data into a standard backup archive on the ZRM server.

Quick snapshot backups are appropriate for large databases and for databases that have high transaction rates. In addition to eliminating data transfer bottlenecks during backup, quick snapshot backups also provide much faster database restoration than other backup methods.

However, because quick snapshot backups do not copy the data off of the MySQL server, they do not protect data against server media failure. For this reason, quick snapshot backups can be converted at any time into standard backups stored on the ZRM server by using the Convert Backup option on the Reports menu tab.

Sun Microsystems' ZFS file system includes snapshot capability, which facilitates near-instantaneous hot backups and rapid restores. ZRM makes use of this functionality by using an optional snapshot plug-in that integrates ZRM for MySQL with ZFS. It creates temporary snapshots of the ZFS volumes on which to perform a full backup. When snapshots are enabled, ZRM for MySQL can perform backups with minimal impact on MySQL applications. Database writes will be blocked only during snapshot creation, which typically takes less than a second regardless of database size.


Performance Testing Setup

System SetupThis section describes the system architecture and configuration implemented for the performance test.

The hardware used during this test was a Sun Fire X4500 server with two dual-core AMD Opteron 285 CPUs running at 2593 MHz with 16 Gbyte of RAM and 48 SATA drives of 512 Gbyte each.

The two boot drives are mirrored using ZFS volume management capabilities and the remaining 46 drives are available for backup data.

As a general rule, no specific OS tuning was done on the system, nor were the binaries of both MySQL and ZRM optimized in any way. The objective of the tests was to deploy the solution out-of-the-box.

The only tuning performed was to limit the amount of memory available to the ZFS ARC cache to 4 Gbyte by adding the following line to /etc/system:

set zfs:zfs_arc_max = 4294967296

MySQL Software ReleaseThe performance tests were carried out using MySQL 5.1 for Solaris 10 x64 (or x86_x64)./opt/mysql/mysql/bin/mysql Ver 14.14 Distrib 5.1.30, for pc-solaris2.10 (x86_64) using readline 5.1

OpenSolaris Operating System Release OpenSolaris 2008.11 snv_101b_rc2 X86 Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 19 November 2008

Zpool ConfigurationAs seen previously, the storage available on the Sun Fire X4500 server was split into two pools:

● A backup pool made of 24 drives and two spares in a RAIDZ2 configuration. This provided 11 Tbyte of capacity for backups with a very high level of protection against drive failure and performance sufficient to handle 300 Mbyte/sec to 350 Mbyte/sec of backup traffic over the network.

● A mysql pool made of 12 drives and two spares in a RAIDZ2 configuration was used to store the MySQL databases.


ZFS ConfigurationThree ZFS file systems were created:

● zmanda/zrm to hold the ZRM backups

● zmanda/logs to hold the MySQL binlogs

● pool2/mysql to hold the MySQL data files

ZRM Software ReleaseThe following software components were installed:

● The ZRM 3.0 for MySQL software

● Zmanda Management Console with MySQL, PHP, Apache, PHPmyadmin, Perl and various Perl modules and their dependencies. These are installed under the /opt/zmanda/zrm and /opt/zmanda/common directories.

ZRM was configured with:

● Level 0 backups

● No compression

db_STRESS Load Generatordb_STRESS is an open source tool (http://dimitrik.free.fr/db_STRESS.html) that can be used to quickly generate large MySQL databases. Installing and configuring db_STRESS for our particular needs involved the following steps.

1. Creating the test databases- small_isam

- med_isam

- large_isam

- small_innodb

- med_innodb

2. Creating test tables in pool2/mysql# /opt/mysql/mysql/bin/mysql -u root small_isam < BenchBASE_tab.sql# /opt/mysql/mysql/bin/mysql -u root small_innodb < BenchBASE_tab.sql # /opt/mysql/mysql/bin/mysql -u root large_isam < BenchBASE_tab.sql # /opt/mysql/mysql/bin/mysql -u root med_innodb < BenchBASE_tab.sql # /opt/mysql/mysql/bin/mysql -u root med_isam < BenchBASE_tab.sql


http://dimitrik.free.fr/db_STRESS.html

3. Generating the dataset# /opt/db_STRESS-kit/bin/bench64 3821145 to create a dataset of roughly 10GB# /opt/db_STRESS-kit/bin/bench64 38211450 to create a dataset of roughly 100GB# /opt/db_STRESS-kit/bin/bench64 382114500 to create a dataset of roughly 1000GB

4. Loading the dataset into the tables# /opt/db_STRESS-kit/mysql/load_mysql.sh small_isam # /opt/db_STRESS-kit/mysql/load_mysql.sh small_innodb # /opt/db_STRESS-kit/mysql/load_mysql.sh med_isam # /opt/db_STRESS-kit/mysql/load_mysql.sh med_innodb # /opt/db_STRESS-kit/mysql/load_mysql.sh large_isam

5. Creating users

Creating a user called dim with password dimitri and giving this user access to all the databases:mysql> GRANT ALL PRIVILEGES ON *.* TO 'dim'@'localhost' IDENTIFIED BY 'dimitri' with GRANT OPTION;

6. Creating a test scenario

We created two different scenarios corresponding roughly to 15 percent CPU utilization of the system by starting one WebX client and 90 percent CPU utilization by starting 30 WebX clients. (Note: WebX is a software package bundled with the open source db_STRESS package.)

Scenario 1:Starting one WebX client with the following command generated about 15 percent CPU utilization:

WebX x.db_STRESS -AntiDEAD 2000 -MAX 100000 -Base $dbname@localhost -User "dim" -Passwd "dimitri" -ID 0 -Group Level20 -Timeout 1 -TotalTIME $duration -RW 0 -SHMKEY 555 -MaxID 2000

Scenario 2:Starting 30 WebX clients, all with the following command generated about 90 percent CPU utilization:

WebX x.db_STRESS -AntiDEAD 2000 -MAX 100000 -Base $dbname@localhost -User "dim" -Passwd "dimitri" -ID 0 -Group Level20 -Timeout 1 -TotalTIME $duration -RW 0 -SHMKEY 555 -MaxID 2000

Testing MethodologyTesting the performance of ZRM 3.0 for MySQL was done by conducting backups of several local MySQL databases with tables of various sizes and storage engines, with the database either idle, 15 percent or 90 percent busy.

Each backup was performed as a Level 0 backup with no compression and three types of backups were performed:

● Raw-quick backups using ZFS snapshots

● Raw-copy backups

● Logical backups


Each type of backup was conducted on the main two storage engines available to MySQL tables:

● MyISAM

● InnoDB

What was measured was the overall time taken by ZRM to complete the backup, as reported by ZRM using the mysql-zrm-reporter command:

mysql-zrm-reporter --where backup-set=$dbname --output report.$dbname.$i --fields backup-date,backup-set,backup-size,backup-time,read-locks-time,flush-logs-time,backup-type,raw-databases,raw-databases-snapshot,logical-databases,backup-status

Each test was carried out three times in a row (except for the tests with the large database) and both the slowest and fastest results were discarded.

The following actions were taken before each measured run:

● Delete all previous backups from /var/lib/mysql-zrm.

● Delete all previous ZFS snapshots created by ZRM.

● Stop all active MySQL clients (the db_STRESS clients).

● Stop the MySQL server.

● Unmount / mount the ZFS file systems, in order to remove any caching effect from ZFS from consecutive runs.

● Start the MySQL server.

Note: Time did not allow us to conduct recovery tests.


Performance Test ResultsA test result marked as" n/a "means that this particular test took too long to complete in a reasonable time and was therefore not measured.

1. MyISAM Results with Idle DatabaseThe first set of results were obtained by measuring the throughput while backing up idle databases made of various MyISAM tables sizes.

Table 1: MyISAM Results with Idle Tables

Backup Type Table Engine Size (GB) Time (s) Throughput (MB/s)

raw-quick MyISAM 10.66 1 10,915.84

raw-copy MyISAM 10.66 68 160.53

logical MyISAM 11.44 575 20.37

raw-quick MyISAM 117.41 1 120,227.84

raw-copy MyISAM 117.41 759 158.40


raw-quick MyISAM 1065.84 1 1,091,420.16

raw-copy MyISAM 1065.84 6467 168.77

logical MyISAM 1065.84 n/a 0.00

The first observation that can be made here is that the raw-quick type of backup, which uses the ZFS snapshot capabilities, can be not only very fast but independent of the size of the dataset.

The second observation is that a “logical” backup, also known as hot-backup, takes a lot longer to complete than a “raw-copy” backup, but of course enables the backup to take place while the database is still available to other users.

2. InnoDB Results with Idle DatabaseThe second set of results were obtained by measuring the throughput while backing up idle databases made of various InnoDB tables sizes.


Table 2: InnoDB Results with Idle Tables


raw-quick InnoDB 10.82 1 11,079.68

raw-copy InnoDB 10.82 91 121.75

logical InnoDB 8.08 411 20.13

raw-quick InnoDB 300.41 4 76,904.96

raw-copy InnoDB 300.41 2314 132.94

logical InnoDB 300.41 n/a n/a

Here as well, we can see that the “raw-quick” type of backup makes the most of the ZFS snapshot capabilities.

We can also see that a “raw-copy” backup of an InnoDB table has a lower throughput compared to a MyISAM table. This is because behind the scenes, ZRM uses mysqlhotcopy when dealing with MyISAM tables and mysqldump for InnoDB tables. The “logical” backups however are consistent on both types of storage engines.


Figure 2: Throughput Represented on a Logarithmic Scale: With Idle Database

MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy InnoDB logical

10

100

1,000

10,000

100,000

1,000,000

10,000,000

10,920

161

20

11,080

122

20

120,230

158

20

76,905

133

1,091,420

169

Throughput with Idle Database

small

medium

large

Thr

ough

put (

MB

/s)

3. MyISAM Results with a 15 Percent Load on the Database The next set of results involved repeating the same measurement as previously, but this time while db_STRESS was running MySQL queries on the databases generating a 15 percent load.

First we measured MyISAM results.

Table 3: MyISAM Results with a 15 Percent Load


raw-quick MyISAM 10.66 11 992.35






raw-quick MyISAM 1065.84 102 10,700.20

raw-copy MyISAM 1065.84 9622 113.43

logical MyISAM 1065.84 n/a n/a

“Raw-quick” backups are again much quicker than all other types of backup in this scenario, but because we're backing up a live database under load, we can see the effect of the delay while we're waiting to grab the lock on the tables before performing the snapshots.

The impact of the 15 percent load is quite noticeable on the backup throughput compared to the figures for an idle database but it is still quite reasonable.


4. InnoDB Results with a 15 Percent Load on the Database Then we move on to InnoDB results.

Table 4: InnoDB Results with a 15 Percent Load


raw-quick InnoDB 10.66 67 162.92




raw-copy InnoDB 300.41 2661 115.60


As mentioned above, “raw-quick” backups are again much quicker than all other types of backup in this scenario, but because we're backing up a live database under load, we can see the effect of the delay while we're waiting to grab the lock on the tables before performing the snapshots.

The impact of the 15 percent load is quite noticeable on the backup throughput compared to the figures for an idle database but it is still quite reasonable.


5. MyISAM Results with a 90 Percent Load on the Database Finally, for the last set of results we used db_STRESS to heavily load the databases at 90 percent load and repeated all the measurements.

First are the MyISAM results.

Table 5: MyISAM Results with a 90 Percent Load





raw-quick MyISAM 106.6 40 2,728.96




Figure 3: Throughput Represented on a Logarithmic Scale: With Database at 15% Load

MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy InnoDB logical

10

100

1000

10000

100000

992

115

20

465

72

18

360

85

18

9048

116

18

10700

113

Throughput with Database at 15% Load

small

medium

large

Th

rou

ghpu

t (M

B/s

)


raw-quick MyISAM 1065.84 876 1,245.91

raw-copy MyISAM 1065.84 72202 15.12

logical MyISAM 1065.84 n/a n/a

Apart from the “raw-quick” backups, we can see here that the overall throughput is well below the previous results, since the backup process is competing with the mysqld process for CPU time.

In almost all use-cases, it would be a much better strategy to off-line such a heavily loaded database before taking a backup.

6. InnoDB Results with a 90 Percent Load on the Database Then come the InnoDB results.

Table 6: InnoDB Results with a 90 Percent Load






raw-copy InnoDB 300.41 7722 39.84


The first raw-quick backup result on the 30.43 Gbyte dataset seemed oddly out of line compared to other results collected.



Figure 4: Throughput Represented on a Logarithmic Scale: With Database at 90% Load

MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy

1

10

100

1,000

10,000

79

15

3

3526

2,729

23

40

531

53

1,246

15

Throughput with Database at 90% Load

small

medium

large

Th

rou

ghpu

t (M

B/s

)

SummaryThis guide is a quick introduction to configuring the Sun Fire X4500 server as a backup server for Zmanda Recovery Manager 3.0. We have seen in this document how to deploy the Zmanda Recovery Manager for MySQL on a system running OpenSolaris. We have successively learned how to:

● Create a ZFS storage pool using RAIDZ2

● Create a ZFS file system

● Install packages using the new OpenSolaris Image Packaging System

● Deploy ZRM in this environment

We have demonstrated the use of ZRM's ZFS snapshot integration in protecting two of the major MySQL storage engines under varying workloads to help better understand the value and positioning of this solution for Sun customers.

ZRM's usage of ZFS snapshot technology has shown its benefits to system and database administrators as it can reduce the time needed to back up a database, independently of its size. Table 1 demonstrated that the time needed to make a backup of a 1 TB MyISAM table could be reduced from 1h47mn to 1s. In this test, we could take a backup of a 1 Tbyte database close to 100 percent utilization in just a few minutes, or only a few seconds if the database were not heavily loaded. With ZRM's capability to convert backups, nearline snapshots can be copied to standard backup stored on the ZRM backup server without interfering with database utilization. (Please note: Results may vary.)

Product Information

OpenSolaris Operating SystemOpenSolaris 2008.11 OS supports Dynamic Tracing (DTrace), ZFS, cryptographic infrastructures, IP filter and User and Process Rights Management, and Predictive Self-Healing, which can further enhance system reliability.

The ZFS File SystemZFS is a 128-bit general-purpose file system that spans the desktop to the data center, where space is shared dynamically between multiple file systems from a single storage pool. Physical storage can be added to or removed from storage pools dynamically, without interrupting services. This capability can provide new levels of flexibility, availability, and performance.

The Sun Fire X4500 ServerThe Sun Fire X4500 server has reached end-of-life and is no longer orderable. It is superceded by the next-generation Sun Fire X4540 Server. The Sun Fire X4500 server can deliver the performance of a four-way x64 server and provides 48 SATA drives with a total capacity of 48 Tbyte in 4U of rack space. Its bigger brother, the Sun Fire X4540 server, is powered by two Quad-Core AMD Opteron™ CPUs operating at 2300 MHz and provides the same amount of storage capacity.


http://www.sun.com/servers/x64/x4540/

Zmanda Recovery Manager 3.0 for MySQLZmanda developed ZRM for MySQL with the explicit goal of providing simple, secure, and consistent backups with minimal impact on applications and users. With ZRM for MySQL, DBAs can deploy MySQL databases in mission-critical, high-transaction environments with confidence because they know their data is protected by Zmanda. True to its open source mission, ZRM doesn't use any proprietary data formats. Users can recover their MySQL data even without ZRM being installed.

ZRM eases day-to-day backup and recovery tasks with its web-based Zmanda Management Console (ZMC). The management console integrates with the Zmanda Network to provide on-line documentation and other support services for ZRM. Because it is open source and supports common standards, as well as open data formats for backups, ZRM can free you from vendor lock-in.

Enabling advanced backup and recovery practices, such as raw, logical or hot-backups (Back up Live MySQL databases without impacting applications and users), ZRM provides Continuous Data Protection (CDP) for MySQL by combining file system-level snapshots along with data from MySQL binary logs. File system snapshots help reduce database server downtime, especially when working with large databases. ZRM uses a flexible framework of plug-ins for taking snapshots on a range of operating platforms.

Zmanda offers three versions of ZRM for MySQL: the ZRM Enterprise edition, ZRM Cluster edition and ZRM Community edition. The Enterprise edition includes a graphical installer, Zmanda Management Console, command-line interface and plug-in framework.

About the Author Caryl Takvorian is currently a Staff Engineer at Sun Microsystems. He joined Sun in 1998 in what is now known as ISV Engineering where he assists ISVs to make the best use of Sun's technologies. His technical skills are mainly in the Solaris OS and multithreaded applications and development tools. Caryl also leads the Telecommunications segment for ISV Engineering.

For More InformationZmanda Documentation

● Zmanda Recovery Manager for MySQL web site http://www.zmanda.com/zrm-mysql-enterprise.html, including ZRM datasheet (PDF): http://www.zmanda.com/zrm-datasheet.pdf

● Zmanda Recovery Manager for MySQL is available for download from the Zmanda network: http://network.zmanda.com/

● Zmanda Network documentation for Zmanda Recovery Manager (requires registration): http://network.zmanda.com/documentationEnterprise.php, includes:

● ZRM: Guide to MySQL Backup & Recovery and Backup and Recovery Benchmark for MySQL with ZRM:

http://network.zmanda.com/zrm-whitepapers.php

● BigAdmin article: Sun Fire X4540 Server as Backup Server for Zmanda's Amanda Enterprise 2.6 Software: http://www.sun.com/bigadmin/features/articles/zmanda_sfx4540.jsp


http://www.sun.com/bigadmin/features/articles/zmanda_sfx4540.jsp

http://network.zmanda.com/zrm-whitepapers.php

http://network.zmanda.com/documentationEnterprise.php


http://www.zmanda.com/zrm-datasheet.pdf

http://mysqlbackup.zmanda.com/

http://mysqlbackup.zmanda.com/

● Backup and recovery tests for MyISAM and InnoDB engines with Zmanda Recovery

Manager for MySQL: http://solutions.mysql.com/solutions/white-papers/partner_wp_zmanda_backup.php

● Sun Storage J4400 Array as Disk Storage for Zmanda Recovery Manager for MySQL Database: http://www.sun.com/bigadmin/features/articles/zrm_mysql_j4400.jsp

Sun Fire Server Information

● Sun Fire servers: http://www.sun.com/servers/x64/x4540/

● Sun Fire X4500 server document collection: http://docs.sun.com/app/docs/prod/sf.x4500#hic

● BigAdmin article: Using Sun Fire X4540 Server With Zmanda Recovery Manager 2.2 for MySQL Database: http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp

OpenSolaris Information

● OpenSolaris 2009.06 Image Packing System Guide (PDF): http://opensolaris.org/os/project/pkg/files/ips.pdf

● OpenSolaris web site: http://opensolaris.org

ZFS Documentation

● Solaris ZFS Administration Guide (part of the Solaris 10 System Administration Collection): http://docs.sun.com/app/docs/doc/819-5461

● Sun documents at http://docs.sun.com

● ZFS Best Practices Guide on the Solaris Internals web site: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

MySQL Documentation

● MySQL site on sun.com: http://www.sun.com/software/products/mysql/index.jsp

● mysql.com web site: http://www.mysql.com/

● MySQL documentation site: http://dev.mysql.com/doc/

● MySQL training courses: http://www.sun.com/software/products/mysql/training.jsp

● MySQL download sites:

● Get MySQL: http://www.sun.com/software/products/mysql/getit.jsp

● MySQL Downloads: http://dev.mysql.com/downloads/


http://dev.mysql.com/downloads/

http://www.sun.com/software/products/mysql/getit.jsp

http://www.sun.com/software/products/mysql/training.jsp

http://dev.mysql.com/doc/

http://www.mysql.com/

http://www.sun.com/software/products/mysql/index.jsp

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

http://docs.sun.com/

http://docs.sun.com/app/docs/doc/819-5461

http://opensolaris.org/

http://opensolaris.org/os/project/pkg/files/ips.pdf

http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp

http://docs.sun.com/app/docs/prod/sf.x4500#hic

http://www.sun.com/servers/x64/x4540/

http://www.sun.com/bigadmin/features/articles/zrm_mysql_j4400.jsp

http://solutions.mysql.com/solutions/white-papers/partner_wp_zmanda_backup.php

http://solutions.mysql.com/solutions/white-papers/partner_wp_zmanda_backup.php

Discussions

● Zmanda forums: http://forums.zmanda.com/

● MySQL forums: http://forums.mysql.com/

● Sun Storage forums: http://forum.java.sun.com/category.jspa?categoryID=66

● Sun Hardware - Servers forums: http://forums.sun.com/forum.jspa?forumID=830

Other Sun Resources

● Sun training courses at http://www.sun.com/training/

● Sun BluePrints™ wiki (http://wikis.sun.com/display/BluePrints/Main, especially the following:

● Storage BluePrints page: http://wikis.sun.com/display/BluePrints/Storage+BluePrints

● Sun BluePrints article: Optimize MySQL Server on Sun x64 Servers and Storage: http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Servers+and+Storage

● Sun Storage Administration wiki: http://wikis.sun.com/display/StorageAdmin/Home

● BigAdmin Storage Tech Tips wiki: http://wikis.sun.com/display/BigAdmin/Storage+Tech+Tips

● BigAdmin Storage Resource Collection (includes community content): http://www.sun.com/bigadmin/collections/storage.html

● Sun Storage web site: http://www.sun.com/storagetek/index.jsp

● Storage Stop Blog: http://blogs.sun.com/storage

Support

● Sun resources:

● Register your Sun gear: https://inventory.sun.com/inventory/

● Services: http://www.sun.com/service/index.jsp

● SunSolveSM Online: http://sunsolve.sun.com

● Community system administration experts: http://www.sun.com/bigadmin/content/communityexperts/

Events of interest to users of Sun products

● Sun and MySQL Around the World—Meetup Mashup: http://www.sun.com/events/mysqltour/index.jsp

● Find a MySQL Meetup Group near you: http://mysql.meetup.com/


http://mysql.meetup.com/

http://www.sun.com/events/mysqltour/index.jsp

http://www.sun.com/bigadmin/content/communityexperts/

http://sunsolve.sun.com/

http://www.sun.com/service/index.jsp

https://inventory.sun.com/inventory/

http://blogs.sun.com/storage

http://www.sun.com/storagetek/index.jsp

http://www.sun.com/bigadmin/collections/storage.html

http://wikis.sun.com/display/BigAdmin/Storage+Tech+Tips

http://wikis.sun.com/display/StorageAdmin/Home

http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Servers+and+Storage

http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Servers+and+Storage

http://wikis.sun.com/display/BluePrints/Storage+BluePrints

http://wikis.sun.com/display/BluePrints/Main

http://www.sun.com/training/

http://forums.sun.com/forum.jspa?forumID=830

http://forum.java.sun.com/category.jspa?categoryID=66

http://forums.mysql.com/

http://forums.zmanda.com/

● Worldwide developer events and Sun Tech Days: http://developers.sun.com/events/

● Current events: http://www.sun.com/events/index.jsp

Licensing InformationUnless otherwise specified, the use of this software is authorized pursuant to the terms of the license found at http://www.sun.com/bigadmin/common/berkeley_license.html.


http://www.sun.com/bigadmin/common/berkeley_license.html

http://www.sun.com/events/index.jsp

http://developers.sun.com/events/

Documents

Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on