SAN Client Deployment - Best Practices and Performance Metrics

This document is provided for informational purposes only and is intended for distribution only by Symantec employees to selected partners and customers. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. Copyright 2010 Symantec Corporation. All rights reserved. Symantec, the Symantec logo and NetBackup are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

SAN Client Deployment - Best Practices and Performance Metrics Issue Date: 10th March 2010 Version number: 1.5 Revision Date: 07 July 2015 Applies to: NetBackup 7.0, 7.1, 7.5, 7.6, 7.6.1, and 7.7 Note: This is a living document and will be subject to periodic updates. Please check the date and version number to ensure you are referencing the latest version.

TECH54778 SAN Client Best Practices and Performance Metrics 10th March 2010 Page i

Contents 1.0 INTRODUCTION .................................................................................................................. 1

1.1 ADDITIONAL READING .......................................................................................................... 1 1.2 PERFORMANCE PARAMETERS FOR CONFIGURING THE SAN CLIENT ....................................... 2 1.3 SAN INTERCONNECT ........................................................................................................... 2 1.4 SAN CLIENT SCALABILITY AND ZONING BEST PRACTICES....................................................... 2 1.5 MEDIA SERVER PROFILE ...................................................................................................... 3 1.6 CLIENT PROFILE .................................................................................................................. 4

2.0 BEST PRACTICES .............................................................................................................. 6 2.1 SEGREGATED FT MEDIA SERVERS ....................................................................................... 6 2.2 COMBINED FT/LAN MEDIA SERVERS ................................................................................... 7 2.3 MULTI-STREAMING BACKUP ................................................................................................. 8

2.3.1 Stream distribution ..................................................................................................... 8 2.3.2 Connection restrictions on the FT Media Server ....................................................... 8 2.3.3 Increasing the maximum target ports available to a SAN Client ............................... 9 2.3.4 Increasing the number of SAN clients per target port ................................................ 9

TECH54778 SAN Client Best Practices and Performance Metrics 10th March 2010 Page 1

1.0 Introduction In todays 24/7 business environment, corporations are looking for ways to protect their data without disrupting business operations. This eliminates the options of using the LAN during of peak hours to backup and protect the data.

The NetBackup SAN Client feature takes advantage of Fiber Channel Storage Area Network (SAN) connections to perform a fast backup of a client machine directly to a Media Server, moving data transfers off the LAN and away from conflicting traffic. The SAN Client provides a high performance data transfer for the client and offloads the LAN with the backup traffic.

The primary purpose of the SAN Client is to back up large applications and file systems faster and more efficiently than is possible using LAN based backup or locally attached backup devices (a SAN Media Server). However the SAN Client architecture does not allow the same degrees of parallel backup operation as the alternative approaches and is therefore less suited to handling large numbers of small backup jobs. As a general rule SAN Client should be deployed selectively on servers where the data volumes are such that LAN backup within a reasonable time period is impractical it should not be deployed with a very large number of clients with a wide range of backup sizes and a small number of FT Media Servers target ports (grouped together) as this may lead to excessive resource contention on the FT Media Servers when there is a wide range of job sizes.

This document describes the NetBackup SAN Client feature and provides some information on performance considerations and best practices when deploy the SAN Client. The document is intended to provide backup administrators with the necessary information to help our customers take full advantage of this advanced feature and maximize their ROI.

1.1 Additional Reading The following documents provide more information on the subjects covered in this paper:

NetBackup SAN Client and Fibre Transport Guide: http://www.symantec.com/docs/DOC5332

NetBackup Device Configuration Guide (OS specific configuration steps for SAN Clients): http://www.symantec.com/docs/DOC5332

SAN Client and Fibre Transport troubleshooting tech note: http://www.symantec.com/docs/TECH51454


1.2 Performance Parameters for configuring the SAN Client To implement the SAN Client, the user needs to deploy a Fiber Channel based SAN between the client and the Media Server. The Server and Clients can each have multiple SAN ports zoned together.

A Media Server that can receive data from a SAN Client is referred to as a Fiber Transport or FT Media Server. The HBAs on the Media Server that receive the data are referred to as Fiber Transport Targets and must be of a supported type (the HBAs used on the SAN Client itself can be of any type).

If configured correctly the feature allows a client transfer rate exceeding two Terabytes of data in an hour. Obtaining this type of performance requires using the right hardware not only for the connectivity between the client and the Media Server but also for the front end client data and the back end storage disks. The whole system must be configured and tuned to achieve the desired performance requirement.

The rest of this document lists the profile of the appropriate hardware and how to configure the system to deliver the optimum performance.

Figure 1 - SAN Client Components

This section provides some information about the type of hardware and the main parts that affect the performance of the system.

1.3 SAN Interconnect When configuring the SAN Client, the first step is to deploy a Fiber Channel SAN between the client and the Media Server.

The architecture of the SAN should be such that the throughput between SAN Clients and Fiber Transport Media Servers is maximized and the latency is minimized. For example, one important requirement is a SAN interconnect design that has no points of congestion. One way of achieving no congestion is to avoid using Inter Switch Links (ISL) and trunking. It is therefore recommended the connection between SAN Clients and the Fiber Transport Servers be on the same switch and switch blade wherever possible.

If you have to use ISLs, we recommend that you under subscribe them. At any time, the trunking between switches should be able to handle the throughput required for your environment's Quality of Service requirements.

1.4 San Client Scalability and Zoning Best Practices The larger amount of client initiators with overlapping zones the more likely there will be a widespread reconfiguration notice to all initiators. This can result in all initiators trying to scan all visible Fibre Transport ports at the same time. Because of this potential, there is a technical limitation to how many SAN client initiator ports should be used per Fibre Transport Media Server target port.


Symantec recommends a maximum of 30 SAN Client initiator ports zoned with a Fibre Transport target port because of the effective queue depth of about 60 for a QLogic hardware/firmware target port. Because the NetBackup Fibre Transport media server will simulate two "pseudo tape devices" on each Fibre Transport target port, the number of commands arriving will be two times the number of initiators. Keeping the number of SAN Clients per port at 30 or below prevents each target port from having to queue more than 60 low-level fibre channel commands at any point in time.

Limiting the number of initiator ports for each target port to 30 will prevent the queue from being overflowed. With more than 30 clients per port there is risk that the Fibre Transport Media Server will not respond to a SCSI inquiry. This could have undesirable results on the client operating system during boot or device discovery.

While all inclusive zoning is supported for SAN Client configurations, it is not recommended. An all inclusive zone runs a risk of a single miss-configured client or server affecting all the other SAN Clients in the zone.

1.5 Media Server Profile The FT Media Server is only supported on a limited number of platforms and selected Qlogic2xxx cards and their OEM variants as Fiber Transport Targets on the Media Server. For SAN Clients and for the backup storage devices on the FT Media Server we support QLogic, Emulex, and most OEM Fiber Channel cards. The range of platforms and HBAs supported is constantly being expanded. Refer to the Hardware Compatibility List (HCL) for more information on the supported types of hardware:

http://www.netbackup.com/compatibility

Below is a list of some important parameters for tuning the system for high performance.

1) Use PCI-express 4/8 lane or PCI-x slots with a speed of 133MHz or higher. Avoid oversubscribing your PCI-x and PCI-e buses. If possible use PCI-e slots to connect the backend storage devices. If PCI-e slots are not available make certain that your Media Server hardware has different dedicated PCI-x buses for the backend storage devices and the Fiber Transport targets.

2) Do not use PCI cards on the same bus as the required QLogic FC HBAs. A slower PCI card reduces the speed of the controlling bus and therefore all other cards in that bus. Consequently, data transfer rates are reduced and performance is degraded.

3) Make sure the total aggregated IO throughput is not more than the total memory bandwidth.

4) In general, we recommend that you dont use more than 4 HBA Fiber Transport ports per Media Server, as 4 Fiber Transport ports provide a maximum actual transfer rate of 600MB/second. However dont use more than 8 ports in any circumstance. Since the backend storage must match or exceed the Fiber Transport Data Rate, the bandwidth requirement on a system with 8 Fiber Transport ports would be 2.4 GB/second which is near the maximum I/O load on most systems. Up to 8 Fiber Transport ports are possible; however, there are diminishing returns as the available bandwidth is spread over the ports.

5) The QLA-2344 four-port FC adapter's usable aggregate performance is not significantly greater than a two-port QLA-2342 when the QLA-2344 four-port FC adapter is used in the same PCI-x slot for SAN Client target mode. The advantage that a QLA-2344 HBA offers is the ability to spread its aggregate performance over four ports instead of two.

The QLA-2344 HBA performs similarly to two QLA-2342 HBAs but uses one less PCI slot if the following is true:

You use a direct-connection (rather than FC switches or bridges) between SAN clients and a Fibre Transport (FT) media server.


Only two ports are fully loaded with Fibre Transport traffic at the same time. 6) Use a system test tool (like dd) to make sure the backend disk can write as fast as

the expected FT transfer speed. For example, if you expect to transfer an aggregate of 600 MB/s over 8 streams, verify that your disk can sustain at least 600 MB/s rate using 8 streams with 256k blocks.

7) Finally, a multi-processor system is required. While the actual CPU utilization for Fiber Transport streams is small, keeping the interrupt latency low is required to maintain high transfer rates. This requires a minimum of two CPU. An x86_64 based system that has a constant 600 MB/s of Fiber Transport data will utilize about 50% of a dual core 3 GHz processor. A very general rule of thumb would be that there should be at least 5 MHz of CPU speed for every 1 MB/s of data transfer.

8) Expect the performance of target mode 4 and 8 Gbit/s FC to be similar because the latency of processing each transfer command at both the client and media server end is a significant portion of the transport time. 8 Gbit/s FC target ports can only provide a higher per port aggregate transfer rate when more client ports are concurrently in use and you should not count on this occurring in practice.

1.6 Client profile On the client side three parameters are to be considered:

1) Operating System The SAN Client is supported on the following operating systems: Solaris, Windows, AIX, HP-UX and Linux. Please refer to the release notes and HCLs for details of the operating system versions supported. Note that the SAN Client is generally supported on the same operating system versions as a Media Server, not a LAN Client.

2) CPU - As with the Media Servers, the rule we use is 1MB/s for every 5 MHz of CPU cycle. This means that the CPU use for your application server may be reduced by up to 5GHz during a particularly heavy I/O load.

3) HBA - If the data is stored on a SAN attached disk, do not use the same HBA for the disk and backup SANs. This allows you to better tune your environment including ISL links and HBA parameters. While the same HBA can be successfully used for both SANs, this practice is discouraged from a performance standpoint. Sharing an initiator introduces a potential performance bottleneck issue. Additionally, it complicates any SAN instability issues. SAN reorganizations and fault recoveries tend to spread through all zones (including overlapping zones) that contain an end point affected by the failure or reorganization.

4) Disk Speed - The speed at which the client can read the data is a fundamental factor. Again you can use dd to test the disk performance and tune it. Keep in mind that the Fiber Transport uses 256k block sizes, so use this factor when profiling the performance of your disk arrays.

5) Data Streams - The minimum optimal number of data streams between a SAN Client and a Fiber Transport Media server is two streams per available Fiber Transport target port. For example, a SAN Client backup to a Media Server with 4 available ports should utilize 8 data streams, writing 1 stream to each LUN available between the Media Server and the SAN Client. Writing more data streams will create contention for the LUNs when the data sources are capable of supplying over half of the maximum data rate for the port, and using less streams of data will mean that the ports on the Media Server are not fully utilized.

6) The SAN Client is not fully qualified to support IPv6. In common with all types of backup operation care should be taken when planning backup policies and schedules to avoid situations in which large numbers of small backup jobs monopolize too many resources on all the FT Media Servers and delay the start of large backup


jobs such that one or more of them fail to start early enough to allow them to complete within the backup window.


2.0 Best practices One question that usually comes to mind when designing and implementing SAN Client in a backup environment is should I mix regular clients with SAN Clients on the same Media Servers or should I segregate them. The answer to this question depends on your environment and performance requirements.

2.1 Segregated FT Media Servers In this scenario the FT Media Servers are dedicated for SAN Client backup only. This case is recommended when you have different tiers of backend storage. The high end, high performance disk storages are connected to the FT Media Servers. This provides enough traffic to optimize the usage of the backend storage.

Another benefit of this architecture is that its easy to distribute the clients to the different Media Servers. As they all have the same speed, an equal distribution of the clients to the Media Servers is usually enough.

In a dedicated Fiber Transport backup pool, you must monitor the environment so that you always know that the load on the Media Servers is at an acceptable level. If the load is such that the Media Servers can't handle all the backup requests from SAN Clients in an acceptable amount of time, the pool will have to be grown, and likewise you may consider shrinking the pool of dedicated Media Servers if you find that the servers sit idle or underutilize the available Disk I/O.

The advantage of a segregated FT Media Server pool is that you are better able to guarantee the class of service necessary is available to SAN Clients since you don't have to allow for LAN based transfers. Disadvantages are that more Media Servers are required overall and/or the pool of Media Servers available to the SAN Clients is smaller than it would be in a mixed environment. This means that the failure of a single Media Server will have a larger impact on the overall SAN Client backup window than it would in a shared environment.

Figure 2 - Segregated FT Servers


2.2 Combined FT/LAN Media Servers In this scenario the same FT Media Servers are used to backup both SAN and LAN based clients. Using Media Servers in this way addresses the principle disadvantage of the segregated model by making more Media Servers available for both FT and LAN backup, thus reducing the number of Media Servers required and/or increasing the number available for SAN and LAN backups. This is particularly useful in environments where some client platforms are not supported with the SAN Media Server.

However it should be noted that LAN jobs utilize more CPU per MB of data transferred and the LAN protocol interrupt load can adversely affect interrupt latency and reduce FT data rates. Additional CPU's may or may not help depending on the affinity of the interrupt processing for one CPU on some operating systems.

Figure 3 - Combined FT/LAN Media Servers A major advantage to this model is that it utilizes a larger group of servers with lower performance requirements for a cost savings on a per-server basis. Since the pool of FT servers is more distributed, it is expected that each server will have a smaller data transfer load while still maintaining the same overall throughput performance due to the cumulative effect of having more active Media Servers.

The downside to this model is that it makes it harder to guarantee a higher Quality of Service to individual large application servers. An individual application server may support 600 MB/s of backup I/O, but if the selected Media Server is already writing many streams of LAN based backups from multiple clients there will be a performance impact to both the SAN Clients and LAN Clients.


2.3 Multi-Streaming backup The typical maximum practical transfer rate for a single 2 gigabit Fiber Channel port is around 160 MB/sec and this is the maximum transfer rate you can expect to see between the SAN Client and FT Media Server over a single port.

The read speed of a single backup stream from the disk system on the client may be significantly less than this maximum figure, resulting in apparently poor performance. However it is possible to run multiple backup streams over the same port between the client and server and thus better backup performance may be achieved by using multi-streamed backups.

The performance of individual backup streams will vary depending on the read speed of the source data. This can be established (on UNIX and Linux clients) by running a performance test using a dd command of the form:

time dd if= of=/dev/zero bs=256k count=8192 Specifying the path to be backed up should be specified in place of . This will return a value for the elapsed backup time which, in turn, will enable you to calculate the read speed of that stream. Once you have a feel for the read speed of each stream you should be able to configure a policy with sufficient parallel streams to achieve the maximum throughput on the Fiber Channel port.

Note that in some cases multiple streams may contend with each other for disk controller resources resulting in further performance degradation when streams are run in parallel and further tuning may be required.

If transfer rates greater than 160 MB/s are required it is possible to use multiple HBA Ports for the client and the server in conjunction with multiple streams. The ratio of client ports to server ports depends on the speed of the HBA cards in the client. For example, if you have to transfer 300 Megabytes per second from a SAN Client to an FT Media Server, you will need to have 2 x two gigabit client ports and the Media Server will require 2 x two gigabit target ports.

2.3.1 Stream distribution When multiple streams are used from a client with more than 1 HBA port, the jobs are distributed in a round robin fashion with one stream being assigned to each port in turn. E.g. if 4 jobs are run using 2 ports then job1 will use port 1, job 2 will use port 2, job 3 will share port 1 with job 1 and job 4 will share port 2 with job 2. This means that as job streams complete and new job streams do not take their place the ports will be under-utilized in the same way that tape drives using multiplexing may be under utilized as the number of concurrent jobs decreases.

2.3.2 Connection restrictions on the FT Media Server On important limiting factor to bear in mind when running multiple streams to multiple ports on a the same FT Media Server is that the maximum number of connections allowed to a single FT Media Server is 16. This means that the total number of concurrent backup streams across all the available ports on the FT Media Server cannot exceed 16. This figure is increased to 32 connections in NetBackup 7.0, 7.1, 7.5, 7.6, and 7.6.1. Beginning with the 7.7 release, the following are the maximum concurrent values:

On a Linux FT media server host: 40

Symantec recommends that you use 32 or fewer connections concurrently on Linux.

On Linux hosts, you can increase that maximum by setting a NetBackup touch file, NUMBER_DATA_BUFFERS_FT.

For NetBackup Appliance model 5230 and later: 40

For NetBackup Appliance model 5330 and later: 40


On a Solaris FT media server host: 64

It is also important to remember that each port used by a specific client impacts the performance of other SAN Clients utilizing the same Media Server. If two SAN Clients utilize the same Media Server with only four target ports, and each client claims all four of those ports, the clients will contend for those ports. The effect of this contention is that the client with a lower latency will 'win' and transfer data faster than the higher latency client, sometimes as much as twice as fast. If each client had only claimed two ports, they would have been able to share the available ports and not run into contention issues.

The key to tuning is therefore to limit the number of ports used by a SAN Client while maximizing the number of streams on each port but ensuring that the maximum number of connections across all the available ports does not exceed the maximum.

2.3.3 Increasing the maximum target ports available to a SAN Client By default, SAN Clients supports a maximum of two Fiber Transport ports at any given time; this allows FT Media Servers to balance an I/O load fairly among multiple SAN Clients.

If a particular client requires a higher Quality of Services than is provided by two Fiber Channel ports then it is possible to increase the number of ports per server the client is allowed to use by executing the following command.

nbftconfig -changeclient -np 4 -C The above command changes the number of ports for the specific client . To change it for all clients the setconfig option is used:

nbftconfig setconfig np 4 Note that setting np 4 will have no effect on a client that only has a 2 port HBA.

2.3.4 Increasing the number of SAN clients per target port By default, one Fibre Transport port can only be used by up to two different SAN Clients at any given time; this prevents oversubscribing a FT Media sever port to multiple clients.

In environments with a greater number of clients per Media Server target port, this value can be increased. This may reduce the Quality of Service and should be done only if necessary. This value is set on a Master server and applies to all the Fibre Transport Media Servers in that domain. The value can be configured by executing the following command

nbftconfig setconfig ncp 4 The recommended value is less than or equal to 4.

1.0 Introduction1.1 Additional Reading1.2 Performance Parameters for configuring the SAN Client1.3 SAN Interconnect1.4 San Client Scalability and Zoning Best Practices1.5 Media Server Profile1.6 Client profile

2.0 Best practices2.1 Segregated FT Media Servers2.2 Combined FT/LAN Media Servers2.3 Multi-Streaming backup2.3.1 Stream distribution2.3.2 Connection restrictions on the FT Media Server2.3.3 Increasing the maximum target ports available to a SAN Client2.3.4 Increasing the number of SAN clients per target port

Documents

SAN Client Deployment - Best Practices and Performance Metrics