Basic NetBackup Performance Tuningeval.symantec.com/mktginfo/downloads/DPN7203_FINAL_June... · 2007-06-12 · Basic NetBackup Performance Tuning Bill Roth - Technical Product Manager,

Basic NetBackup Performance TuningBill Roth - Technical Product Manager, NetBackupRay Schafer – Technical Product Manager, NetBackupJune 12, 2007

This forward-looking indication of plans for products is preliminary and all future release dates are tentative and are subject to change. Any future release of the product or planned modifications to product capability, functionality or feature are subject to ongoing evaluation by Symantec, and may or may not be implemented and should not be considered firm commitments by Symantec, and should not be relied upon in making purchasing decisions.

Symantec Vision 2007 2

Agenda

Opening 11

Examining the challenge 22

The data transfer path 33

Bottleneck identification 44

Closing 55


The Situation

• Complaints about slow backups? – Backups don't complete in time and impact production workload– Backups don't complete within a 24 hour window, impacting the next

days backups– Fast tape drives aren't being fully utilized

• Issues with long restores– Database recovery takes days instead of minutes– Full volume recovery takes much longer than expected– File recovery takes a long time to complete


The Situation

• Complaints…– Hello, Has anyone use a LTO Ultrium 2 autoloader tape drive that has

decent backup time? I have one and I can't figure why it take so long just to backup 280GB locally.

• The tape drive is connect directly to the server, a Dell PowerEdge 2950 and all I’m backing up is the C and D drive, about 280g and it takes 6:30:00 to complete, suppose to take about two hours only.

• I call HP up and go through some testing and eventually was told, "the device is working fine, not sure why it takes so long", by Hp Tech support.


The Complication

• Advice is wonderful, but it can complicate the equation…– LTO-2 tend to do 32 MB/s with compression and 16 MB/s without

compression. That means;• 1 hr at 32megs per second you get 115200mb or 115gig per hour• 1 hr at 16megs per second you get half that at 57.6gig.

– Assuming the optimal number it would take 2.4 hours to backup. Figure the same amount of time for verify. This should put your backup at about 5 hours.

• "suppose to take about two hours only“• Can you see now where your calculations are incorrect? Odds are you were not

including verify time nor delays caused by open files. Your tape drive should also be on a SCSI card all by itself.


The Complication

• More advice adds to the complication…

– I have found if you’re backing up a lot of small files it takes a lot longer. I have a DLT S4. It says it will backup 800GB uncompressed (one tape) in 10 hours ...

• It takes 24 usually for me.


The Implication

• The need for basic performance tuning skills is clear– Unless you’re “lucky” considerable time will be consumed, possibly

without positive results– There is a great likelihood that you may face multiple data protection

performance issues, each with a different root cause


Agenda

Examining the challenge 22


The Challenge is Clear

• But where should I start?– Are SLA's for backup and restore in place...

• Some sites have general guidelines stating that all full backups should complete during a weekend window

• Some sites have more granular guidelines that apply to specific data types– These sites may have characterized data according to business value

or importance• Some sites have no documented plan, and wait for complaints and problems

to surface• Some of us may have inherited a data protection infrastructure that has been

ignored or mismanaged in the past

– How do you define "slow backups" and "long restores"?• Unless SLA's have been defined, "slow" or "long" may be a perception issue• Does anybody actually measure and record performance?



• Before trying to address the challenge it might be a good idea to better understand a few key areas:– What backup and restore performance you're trying to achieve?

• Basing this determination on business requirements is recommended• Segregating or characterizing requirements based on data business value is

also recommended

– What are current performance levels?• Measure and record backup and restore performance on a daily, weekly, or

monthly basis• Track both full and incremental backups• Track all restore requests• Symantec recommends restore testing on a frequent basis, measure and

record these tests as well



• Once the challenge is accurately defined it becomes possible to evaluate the situation using a logical sequence of steps– Performance goals / SLA's are known and understood– Current performance is known and non-compliant data protection

operations have been identified


Agenda

The data transfer path 33


Data Transfer Path

• What is the data transfer path for this slow client?

Remote Office 1 Remote Office 2

WAN

Main Campus 1 Main Campus 2

SAN

Backup Infrastructure

• Defining a typical data transfer path


Data Transfer Path

Backup Server NetBackup media server, or master & media server combination

Storage Device Secondary storage used for backup images

TCP/IP network path between the client and the NetBackup server

NetBackup client with data protection requirements

Network

Client

• Defining a typical data transfer path


Data Transfer Path


Data Transfer Path

• Defining the data transfer path– Begin by trying to isolate the data transfer path relative to a

specific client from the bulk of the data protection infrastructure– The basic high level data transfer path includes the client, a

network, a media server, and a secondary storage device

Backup Server Storage DeviceNetworkClient


Data Transfer Path

• Defining the data transfer path– For those of us that don’t appreciate block diagrams:

NetBackup Client

Disk Storage

Network

Tape & Disk

Storage Units

NetBackup Media Server


Data Transfer Path

• Understand the client component of the data transfer path:

– 2x 3.2 GHz Intel Xeon CPU– 2 GB RAM– 2x Intel PRO/1000 MT NIC– LSI PCI-X Ultra320 SCSI Adapter– Emulex LP9000 HBA– Seagate ST373207LC SCSI Disk– 10x Hitachi SCSI Disk (SAN Connected)

MS Windows Server 2003 SP2MS Exchange 2003 SP2NetBackup 6.0 MP4 Client

Client


Data Transfer Path

• Understand the client component of the data transfer path:

3.2 GHz CPU

10/100/1000 NIC

3.2 GHz CPU

10/100/1000 NIC

1 Gb HBA

2 GB RAM

Ultra 320 SCSI

73 GB

6.9 GB6.9 GB

6.9 GB6.9 GB

6.9 GB

6.9 GB6.9 GB

6.9 GB6.9 GB

6.9 GB

NetBackup Client


Data Transfer Path

• Understand the client component of the data transfer path:– Examine additional parameters such as disk drive specific

information– Vendor web pages often provide great detail:


Data Transfer Path

• Understand the client component of the data transfer path:– Examine additional parameters…– LSI PCI-X Ultra320 SCSI Adapter

• View vendor web pages to obtain additional information• View industry web pages for additional information

– www.t10.org– www.scsita.org

– Ultra320 is defined as being able to provide up to 320 MB/s data transfer rates

http://www.t10.org/

http://www.scsita.org/


Data Transfer Path

• Understand the network component of the data transfer path:

Client NIC configurationNetwork

• Parameters– Local settings are easily determined– Other parameters may be more difficult to

determine


Data Transfer Path

• Understand the network component of the data transfer path:– Understand the network path being used to perform backups /

restores• Switches• Hops• Port settings• Other network traffic

NetBackup Client

Network



Data Transfer Path

• Understand the media server component of the data transfer path:

– Attributes:• CPU• Memory• HBAs• NICs• Other adapters, disk drives, etc.

NetBackup 6.0 MP4 media serverBackup Server



Data Transfer Path

• Understand the storage device component of the data transfer path:

– Attributes:• Device type• Rated speed• Capacity

Disk & Tape based secondary storageStorage Device


Agenda

Bottleneck identification 44


Bottleneck Identification

• Before trying to identify a bottleneck:– Characterize performance

• Take a granular look at an individual client relative to the backup infrastructure• Break down the infrastructure into individual components

– Measure & document current performance attributes• Determine & record current data transfer rates, backup & restore duration

– Have a goal, SLA, or performance metric that you’re trying to achieve



• Bottlenecks & current limitations– There’s always a bottleneck in the data transfer path

• It may be a very fast bottleneck, but it exists non-the-less

– Use a variety of tool and utilities to evaluate current performance– Pinpoint performance bottlenecks



• Tools and utlities– On the client, you can use a variety of tools and utilities that can

assist is understanding how fast data can be extracted from the source disk or array:

• Bonnie++ (Unix)• Tiobench (Linux)• HDSpeed (Windows)• Many others available

– Additionally, you can execute the NetBackup bpbkar process in standalone mode

• This is the NetBackup process that is used during backups• The “NetBackup Planning and Performance Tuning Guide” contains

instructions for using this tool



• Tools and utlities– Client data extraction example – IOMeter:

– This utility is able to read file system data at about 54 MB/s



• Tools and utilities– Network tools and utilities can be used to determine if network

bottlenecks are impacting backup performance• Copying data across a network connection may provide a ballpark

performance figure, but the results can be skewed by source and destination disk drives

– IPERF• Can be used to test and isolate network issues• Can measure throughput in either direction• Supported on Unix, Linux, and Windows• Testing does not involve disk subsystems



• Tools and utilities– Network performance test example, IPERF

– Test results indicate an approximate 42 MB/s data transfer rate



• Tools and utilities– What about the media server?

• A variety of tools and utilities are available to assist in bottleneck identification• NetBackup debug logs such as bptm, bpdm, and bpbkar• OS resident utilities that track CPU and memory utilization, etc.

– OS resident logs are helpful in a variety of cases• Windows Event logs, and additionally, viewing the BIOS messages during the

boot process• Unix var/adm/messages



• Tools and utilities– NetBackup media server



• Tools and utilities– NetBackup media server



• Tools and utilities - Buffers– The NetBackup media server uses shared memory to buffer data

between the network and the tape or disk drive• Or between the disk and the tape drive if the NetBackup media server and

client are the same system

– The number and size of these shared data buffers can be configured on the NetBackup media server

– The size and number of the tape and disk buffers may be changed so that NetBackup optimizes its use of shared memory

– Changing the default buffer size may result in better throughput for high-performance tape drives

• These changes may also improve throughput for other types of drives

– Buffer settings are for media servers only and should not be used on a pure master server or client



• Tools and utilities - Buffers



• Tools and utilities - Buffers

• Tools and utilities – Buffers– The block diagram representation

* NetBackup 6.5 uses bptm with disk, prior versions use bpdm

Diskbpdm(child)

SharedBuffers

bpdm/bptm*(parent)

bptm(child)

SharedBuffers

bptm(parent)

Tape





• Tools and utilities – Buffers– There are two schools of thought surrounding the use of buffers

to accomplish the goal of improved performance– The first uses NetBackup logs to view buffer wait and delay

counters• This approach is more time consuming but provides insight into what’s going

on internally within NetBackup• Logs provide buffer usage details on a process by process basis• These are generally used to provide input for bottleneck identification and

additional tuning

– The second takes a more generalized approach of just setting them to an “optimal” value

• We’ve run through an exhaustive series of steps and have come to the conclusion that in our environment, a predetermined quantity of buffers and predetermined buffer size gives us the best performance



• Tools and utilities – Performance Tuning Guide



• Tools and utilities – Performance Tuning Guide



• Tools and utilities– Storage Devices should be matched to HBA’s or SCSI adapters

• Be cognizant of the number of drives and the throughput per drive compared to the speed of the FC/SCSI connection

• Understand rated throughput capabilities of tape drives

– Tape streaming is a good thing• Proper communication bandwidth is one of many factors that can assist in

eliminating “shoe-shining”

– Data Compression• Tape based compression is generally favorable

– Preferred over client side compression



• Tools and utilities– Storage Devices - understand current limitations…– Use “dd” or “tar” to test without reading data

from a disk source• UNIX:• dd if = /dev/zero of = /dev/rmt/0cbn

– Windows:• dd if = NUL of = \\.\TAPE0• Note: Windows may require the installation of a utility or

other software to enable these commands…

NetBackup Master / Media

Server

Tape Storage

Unit



• Tools and utilities– Storage Devices - understand current

limitations…– UNIX measurement tools

• iostat• SAR

– Windows measurement tools• NetBackup includes four NetBackup related

performance Counters:– Disk / Tape Read (2)– Disk / Tape Write (2)


Agenda

Closing 55


Closing

• Improving performance– Develop a plan of attack

• Bottleneck identification should yield a list of areas identified as potential bottlenecks

– Prioritize bottleneck elimination• It often makes sense to eliminate major bottlenecks first• It may not make sense to purchase and deploy faster tape drives if the current

tape drives are not the main bottleneck in the data path, for example

– Execute a methodical plan of attack• As the configuration is altered for bottleneck elimination, make a minimal

number of simultaneous changes and record new performance metrics• Confirm that the appropriate bottleneck has been eliminated• Move on to address subsequent bottlenecks as need to meet business

requirements


Closing

• Additional reference– Advanced NetBackup Performance Tuning

• Session S287204• Today following this session

– “Backup & Recovery”• By W. Curtis Preston• Available on amazon.com

– “Implementing Backup and Recovery: The Readiness Guide for the Enterprise”

• By David B. Little and David A. Chapa• Available on amazon.com

– “Veritas NetBackup™ Backup Planning and Performance Tuning Guide”

• Available on support.veritas.com


ANSWERSQUESTIONS

Bill Roth

[email protected]

Ray Schafer

[email protected]

Copyright © 2007 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. This forward-looking indication of plans for products is preliminary and all future release dates are tentative and are subject to change. Any future release of the product or planned modifications to product capability, functionality or feature are subject to ongoing evaluation by Symantec, and may or may not be implemented and should not be considered firm commitments by Symantec, and should not be relied upon in making purchasing decisions.

Thank You

Documents

Basic NetBackup Performance Tuningeval.symantec.com/mktginfo/downloads/DPN7203_FINAL_June... · 2007-06-12 · Basic NetBackup Performance Tuning Bill Roth - Technical Product Manager,