EMC NetWorker 8.2 Server Disaster Recovery and Availability Best

EMC® NetWorker®Version 8.2

Server Disaster Recovery and AvailabilityBest Practices Guide302-000-693

REV 02

Copyright © 1990-2015 EMC Corporation. All rights reserved. Published in USA.

Published December, 2014

EMC believes the information in this publication is accurate as of its publication date. The information is subject to changewithout notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind withrespect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for aparticular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.

EMC², EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and othercountries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date regulatory document for your product line, go to EMC Online Support (https://support.emc.com).

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 In North America 1-866-464-7381www.EMC.com

2 EMC NetWorker 8.2 Server Disaster Recovery and Availability Best Practices Guide

5

Introduction 9

NetWorker server disaster recovery roadmap.................................................10

Availability and Recovery Options for a NetWorker Server 11

Bootstrap and indexes.................................................................................. 12Bootstrap save set........................................................................... 12Client file index save set.................................................................. 12Bootstrap recommendations and practices...................................... 12How to obtain the bootstrap.............................................................13

Gathering the key information....................................................................... 13Hardware information...................................................................... 13Software information........................................................................14

Disaster recovery scenario review..................................................................14Basic disaster recovery (same host)................................................. 14Advanced disaster recovery (different host)......................................15Ground level preparation for NetWorker server disaster recovery...... 16

Data Storage and Devices 17

Capabilities and considerations.................................................................... 18NetWorker metadata storage.........................................................................18Multi-path access and failover.......................................................................18

Storage devices and media.............................................................. 18Method of connectivity.....................................................................19

Reliability and dependencies........................................................................ 20

Disaster Recovery Use Cases 21

Basic disaster recovery scenario....................................................................22Basic disaster recovery considerations..........................................................24More advanced disaster recovery considerations.......................................... 26Clustered solutions....................................................................................... 28Backup to disk.............................................................................................. 28Index or configuration corruption.................................................................. 30Corruption or loss of SAN storage.................................................................. 30Loss of one server, Data Domain system, or site............................................ 30Replication solutions.....................................................................................30

Replication of the NetWorker Server................................................. 31

Preface

Chapter 1

Chapter 2

Chapter 3

Chapter 4

CONTENTS

EMC NetWorker 8.2 Server Disaster Recovery and Availability Best Practices Guide 3

CONTENTS


Preface

As part of an effort to improve its product lines, EMC periodically releases revisions of itssoftware and hardware. Therefore, some functions described in this document might notbe supported by all versions of the software or hardware currently in use. The productrelease notes provide the most up-to-date information on product features.

Contact your EMC technical support professional if a product does not function properlyor does not function as described in this document.

Note

This document was accurate at publication time. Go to EMC Online Support (https://support.emc.com) to ensure that you are using the latest version of this document.

PurposeThis document describes how to design and plan for a NetWorker disaster recovery.However, it does not provide detailed disaster recovery instructions. The DisasterRecovery section of the EMC NetWorker SolVe Desktop (formely known as the NetWorkerProcedure Generator (NPG) provides step-by-step disaster recovery instructions that aretailored to your environment.

You can download the EMC NetWorker SolVe Desktop from the EMC Online Support Siteat https://support.emc.com/ under the Tools and Utilities section.

AudienceThis guide is part of the NetWorker documentation set, and is intended for use by systemadministrators who are responsible for setting up and maintaining backups on a network.Operators who monitor daily backups will also find this guide useful.

Revision historyThe following table presents the revision history of this document.

Table 1 Revision history

Revision Date Description

02 December 4, 2014 Updated Chapter 5 Replication Solutions and sub-topics.

01 June 18, 2014 First release of this document for EMC NetWorker 8.2.

Related documentationThe NetWorker documentation set includes the following publications:

l EMC NetWorker Online Software Compatibility GuideProvides a list of client, server, and storage node operating systems supported by theEMC information protection software versions. You can access the Online SoftwareCompatibility Guide on the EMC Online Support site at support.emc.com. From theSupport by Product pages, search for NetWorker using "Find a Product", and thenselect the Install, License, and Configure link.

l EMC NetWorker Administration GuideDescribes how to configure and maintain the NetWorker software.

l EMC NetWorker Cluster Installation Guide


https://support.emc.com/



HTTPS://SUPPORT.EMC.COM/

Contains information related to configuring NetWorker software on cluster serversand clients.

l EMC NetWorker Installation GuideProvides information on how to install, uninstall and update the NetWorker softwarefor clients, storage nodes, and servers on all supported operating systems.

l EMC NetWorker Updating from a Previous Release GuideDescribes how to update the NetWorker software from a previously installed release.

l EMC NetWorker Release NotesContains information on new features and changes, fixed problems, knownlimitations, environment and system requirements for the latest NetWorker softwarerelease.

l EMC NetWorker Avamar Devices Integration GuideProvides planning and configuration information on the use of Avamar devices in aNetWorker environment.

l EMC NetWorker Command Reference GuideProvides reference information for NetWorker commands and options.

l EMC NetWorker Data Domain Deduplication Devices Integration GuideProvides planning and configuration information on the use of Data Domain devicesfor data deduplication backup and storage in a NetWorker environment.

l EMC NetWorker Error Message GuideProvides information on common NetWorker error messages.

l EMC NetWorker Licensing GuideProvides information about licensing NetWorker products and features.

l EMC NetWorker Management Console Online HelpDescribes the day-to-day administration tasks performed in the NetWorkerManagement Console and the NetWorker Administration window. To view Help, clickHelp in the main menu.

l EMC NetWorker User Online HelpThe NetWorker User program is the Windows client interface. Describes how to usethe NetWorker User program which is the Windows client interface connect to aNetWorker server to back up, recover, archive, and retrieve files over a network.

Special notice conventions used in this documentEMC uses the following conventions for special notices:

NOTICE

Addresses practices not related to personal injury.

Note

Presents information that is important, but not hazard-related.

Typographical conventionsEMC uses the following type style conventions in this document:

Italic Use for full titles of publications referenced in text

Monospace Use for:

l System code

l System output, such as an error message or script

l Pathnames, file names, prompts, and syntax

Preface


l Commands and options

Monospace italic Use for variables

Monospace bold Use for user input

[ ] Square brackets enclose optional values

| Vertical bar indicates alternate selections - the bar means “or”

{ } Braces enclose content that the user must specify, such as x or y or z

... Ellipses indicate non-essential information omitted from the example

Where to get helpEMC support, product, and licensing information can be obtained as follows:

Product informationFor documentation, release notes, software updates, or information about EMC products,go to EMC Online Support at https://support.emc.com.

Technical supportGo to EMC Online Support and click Service Center. You will see several options forcontacting EMC Technical Support. Note that to open a service request, you must have avalid support agreement. Contact your EMC sales representative for details aboutobtaining a valid support agreement or with questions about your account.

Online communitiesVisit EMC Community Network at https://community.emc.com for peer contacts,conversations, and content on product support and solutions. Interactively engage onlinewith customers, partners, and certified professionals for all EMC products.

Your commentsYour suggestions will help us continue to improve the accuracy, organization, and overallquality of the user publications. Send your opinions of this document to [email protected]

Preface


https://support.emc.com

https://community.emc.com

mailto:[email protected]

Preface


CHAPTER 1

Introduction

This chapter includes the following section:

l NetWorker server disaster recovery roadmap.........................................................10

Introduction 9

NetWorker server disaster recovery roadmapThis guide provides an aid to disaster recovery planning. It does not provide detailedstep-by-step disaster recovery instructions.

The Disaster Recovery section of the NetWorker SolVe desktop provides step-by-stepdisaster recovery instructions that are tailored to your environment.

You can download the NetWorker SolVe Desktop from the EMC Online Support Site.

The following figure lists the high-level steps to follow when performing a disasterrecovery of the NetWorker server.

Figure 1 Disaster recovery roadmap

Introduction


CHAPTER 2

Availability and Recovery Options for a NetWorkerServer

This section provides an overview of the various options that can be used to protect andrecover a NetWorker server.

This chapter includes the following sections:

l Bootstrap and indexes.......................................................................................... 12l Gathering the key information............................................................................... 13l Disaster recovery scenario review..........................................................................14

Availability and Recovery Options for a NetWorker Server 11

Bootstrap and indexesBacking up key configuration information is central to the recovery of a NetWorker server.This configuration information is stored in various locations on the NetWorker server andcan change as different clients, devices, and volumes are used, updated, or changed.

The two main backup components that protect this stored data are Bootstrap save setand Client file index save set.

Bootstrap save setThe bootstrap is a special save set that is generated by the backup server. The bootstrapbackup contains key information about the current state and configuration of NetWorkerclients, devices, volumes, and other important information for backup and recoveryoperations.

The bootstrap consists of three components that reside on the NetWorker server:

l The media database of the NetWorker server.

l The resource database of the NetWorker server that includes the jobs database.

l The server index.

The bootstrap backup typically occurs after each backup or savegroup completes and isgenerally small in size. Backing up this save set is the only guaranteed method to captureconfiguration information in a safe and consistent way. The availability of this save set isrequired to ensure a successful disaster recovery of the NetWorker server, regardless ofany other protection methods that are used.

Client file index save setAfter all of the save sets in a scheduled backup for a client completes, the NetWorkersoftware saves the client-specific backup information to the client file index. Each clienthas a client file index directory which is stored in the nsr/index directory on theNetWorker server. The client file index acts as a record of backup data and enablessimple recovery and the ability to browse and restore the data. A client file index consistsof many separate files and directories, and its size depends on the amount of client databacked up.

Each client file index contains the following information:

l Backups that have been performed for a client

l Backup level and type of backup

l File attributes

The client file index is not always required to recover data. You should back up the clientfile index and ensure that it is available for recovery by using the appropriate bootstrapinformation. The availability of the client file index greatly impacts the full restoration ofbackup and recovery services following a disaster recovery. You can also use the clientfile index to determine the time required to restore a NetWorker server to a fullyfunctional state.

You can use the nsrck command to rebuild the client file index for a client from the indexbackup.

Bootstrap recommendations and practicesBy default, if the NetWorker server is a member of an active group, the bootstrap isbacked up once all of the backups for a save group have completed. If the NetWorker

Availability and Recovery Options for a NetWorker Server


server is not a member of an active group, the bootstrap backup is performed after all ofthe backups for every save group has completed.

To ensure that the latest NetWorker server configuration information is captured:

l Maintain a record of the boostraps for reference. The record should be separate andindependent from the backup server or any of its components. You can retain emailor printed copies of the bootstrap record.

l Provide the following information in the bootstrap record:

n The date and time of the bootstrap backup.

n The volume and location that the bootstrap save set is stored on.

n The save set ID of the bootstrap.

n The starting file and record number of the bootstrap save set on the volume.

l Perform a bootstrap backup regularly, after all of the save sets in a save group havecompleted or at least once every 12 hours.

l Clone bootstrap volumes regularly to ensure that a single media failure or loss doesnot impact the recovery of the NetWorker server.

l Write the bootstrap save set to a device that is local to the NetWorker server.

l Write the bootstrap save set to separate, dedicated media.

l Do not mix the bootstrap save set with client backup data. This procedure speeds upthe recovery process and ensures that the recovery of the NetWorker server is notdependant on client data volumes that might have inappropriate policies orprotection.

l Ensure that the location of the media does not impact the access to the bootstrapdata if a local disaster occurs such as a flood, fire, or loss of power. Although localcopies of the bootstrap data are beneficial, they should maintain multiple copies ofthis information.

How to obtain the bootstrapYou can obtain the bootstrap record in the following ways.

l Configure the bootstrap notification to email or print a copy of the boostrap record.

l Use the mminfo -B command.

l Review the savegroup completion report. This report lists the bootstrap record whenthe save set is generated during a save-group backup.

Gathering the key informationTo aid in quick disaster recovery, maintain accurate records for each hardware, software,network, device, and media component.

Hardware informationMaintain the following hardware information and ensure that is kept up to date:

l Volume or file-system configuration

l Fully qualified domain names, IP addresses, and host names

l References for Domain Name Servers (DNS) gateways, Active Directory, or domainservers


How to obtain the bootstrap 13

l Hard drive configuration

l Media device names and paths

l Hardware vendor contact information and contract numbers

l Configuration information for each piece of hardware, both active and inactive, foreach system.

Software informationMaintain the following software information and ensure that is kept up to date:

l Copies of the original operating system media and patches and where they arelocated

l Software enabler and authorization codes

l Software vendor contact information and contract numbers

l The operating system version and patches that were installed

l Operating system configuration

l Emergency media that can be used to recover a computer if a disaster occurs

l NetWorker bootstrap information for each NetWorker server

l Kernel configuration and location

l Device drivers

l A list of any Windows volume mount points and UNC paths

Disaster recovery scenario reviewThe following disaster recovery scenarios might be encountered. Each scenario requires aa different number of recovery steps and might be easier or more challenging to plan foror recover from. The NPG provides step-by-step guidelines on how to perform a disasterrecovery by using the NetWorker software on different OS platforms.

In the simplest scenario, the same physical server remains in place with little or nochanges to the original configuration or the surrounding environment. This is typical of asimple component failure such as a disk or power supply where the base operatingsystem might have been removed or corrupted. In this scenario, a fresh install of thesoftware is required.

In the more complex scenario, a major event has taken place such as a loss of an entireroom or building due to flood or fire. Here, the same hardware might not be available andthe surrounding environment might be disrupted or changed. The recovery process ismore complex and some elements will need to be adapted or prioritized.

The following sections highlight the considerations to note when recovering theNetWorker server.

Basic disaster recovery (same host)Recovering the NetWorker server to the same host is the simplest way to perform adisaster recovery. This base level of recovery should be planned for and in place for allNetWorker deployments.

In this disaster recovery scenario, the objective is to:

l Restore the NetWorker server as quickly as possible to the latest, last known goodpoint before the server failed.



l Ensure that the original recovery media is available.

l Ensure that the original recovery devices are available.

l Ensure that the original environment such as SAN, IP, and storage units remainunchanged.

This is a simple recovery, if:

l An adequate bootstrap and index backups exists.

l The configuration details have not changed much and are well known or documented.

l You are able to access to the media and devices that are required for the recovery.

l The backup administrator has the appropriate skills and knowledge to perform therecovery task. NetWorker 8.1 and higher includes a command line wizard programnamed nsrdr that automates the recovery of the NetWorker server’s media database,resource files, and client file indexes. The NetWorker Administration Guide providesmore details.

In some cases, the physical host might be subject to external issues that might prevent adisaster recovery to be performed or fully completed. This scenario might require manualadaptations to ensure that adequate or alternative connectivity is made available. Thissituation might not require restoring the bootstrap or client file index. To restore theserver to the original state following a temporary change, you need to know the originalconfiguration.

Advanced disaster recovery (different host)The recovery of a NetWorker server to a different host is more complex than performing abasic disaster recovery to the same host. The effort and skills required to recover to adifferent host is significantly greater than a basic disaster recover to the same host.Recovering to a different host will typically require additional information or resourcescoupled with the appropriate skills set to perform and complete the task.

While the loss of a building or site is less likely to occur, the effort and speed that isrequired to recover the NetWorker server has a direct impact on the time to restore ormaintain critical business services following an incident. Business-critical services mightalso be affected and might require a disaster recovery or failover process that relies onthe backup and recovery services that the NetWorker server provides. It is thereforeessential that an advanced disaster recovery scenario is included in any disaster recoveryor business continuity plan.

Although the objective is the same as for the basic disaster recover to the same host, inthis situation:

l The NetWorker server hardware is likely to be different and its connectivity andconfiguration might be different from the original.

l Simply restoring the bootstrap and client indexes might not be as quick or as easy toperform.

l Additional changes to the configuration might also be required before the backupand recovery service is available.

l Immediate access to the original recovery media and the devices cannot be assumed.

l The environment is likely to be different so that the SAN, IP, and storage units mightnot match the original server.

l Additional steps might be required to make the NetWorker server available.

l The availability of adequate bootstrap and index backups is required, but thesemight be copies of the original save sets.


Advanced disaster recovery (different host) 15

l Additional steps might be required to access the save set backups.

Ground level preparation for NetWorker server disaster recoveryTo optimize your chances for a successful disaster recovery of the NetWorker server, youmust meet the following minimum requirements:

l Back up the bootstrap regularly, at least once per 12 hours.

l Back up the server OS configuration regularly.

l Back up the client file indexes for all clients. A separate, dedicated backup for allclient indexes can be performed before or after a bootstrap. This step provides acomprehensive disaster recovery backup solution.

l Monitor, record, and store the status and contents of each bootstrap backup in aseparate physical location from the NetWorker server.

l Use a dedicated pool for bootstrap backups.

l Clone the bootstrap backups.

l Record and maintain the connectivity and details of the SAN, IP, and all storagecomponents.



CHAPTER 3

Data Storage and Devices


l Capabilities and considerations............................................................................ 18l NetWorker metadata storage.................................................................................18l Multi-path access and failover...............................................................................18l Reliability and dependencies................................................................................ 20

Data Storage and Devices 17

Capabilities and considerationsSuccessful disaster planning and recovery relies on the availability of the media on whichthe data is stored and the availability of the devices to read that data. In some cases, thedisaster might be localized and the devices and connectivity might be available. Othermore serious or catastrophic incidents will impact the environment that the NetWorkerserver relies on. This scenario might render the devices inoperable or prevent access todevices or media.

A number of strategies can be used to cope with these scenarios and range from:

l Having multiple devices and copies of data.

l Ensuring that alternative devices, media, or paths are available within short timeperiods.

These recovery strategies will enable you to restore with minimal disruption, effort, andguess work.

NetWorker metadata storageProtecting the storage or data during its normal life can help to prevent disastersituations from occurring. These steps might also might help to improve the speed orreliability of the disaster recovery.

To help to improve the speed, reliability, scalability, and performance of the backupserver:

l Keep key configuration information and index data on separate LUNs to eliminate OScorruption issues and improve overall system performance.

l Host LUNs on RAID-protected or external storage systems to improve theperformance, reliability, and resilience of this data.

l Ensure that you have the appropriate amount of storage.

l Ensure that the storage is protected and is performing at optimal levels.

l Consider using advanced protection technology such as replication or snapshots ofthis data since they offer additional protection.

Multi-path access and failoverWith any storage device that is used to store bootstrap information, consider theinformation outlined in the following sections.

Storage devices and mediaAs the resilience and ease of deploying storage devices varies, so the disaster recoverystrategies that are used should be changed to suit the circumstances. For example, theability to obtain and move a single tape device is simpler than it would be for a virtualtape library (VTL), where the installation and configuration might take considerable effortand time to achieve.

For traditional tape, you can use a single tape deck that is manually loaded. It can belocated next to or inside the same hardware as the physical server. In some cases, thismight be an autoloader with multiple devices and an automated robotic arm that loadsand unloads the media.



For other storage devices, such as VTL or disk systems, the device might be an appliancethat includes CPU, memory, networking, and multiple disk units.

Method of connectivityThe method of connectivity can vary from a simple cable for a standalone tape device, tomultiple IP or SAN connections. Having the device or media available is of little use if therequired connectivity is unavailable.

The availability of the following components are important aspects of disaster recoveryplanning:

l Spare cables

l Alternative ports and routes

l Resilient networks

Configure devices with dual ports for multipath accessIn some cases, devices can be configured with dual ports or multipath access which istransparent to the backup application and device. However, for other devices thisconfiguration might be more difficult to configure. It is simpler to configure and makeavailable spare ports or alternative host names and routes as a disaster recoveryplanning step than it is to create or configure them at the time of a disaster recovery.

Most manufacturers do not support dual path tape devices or library control ports, or theyimpose limitations that make these options impractical. However, you can reservealternative ports and make available alternative or backup path connections.

Make devices available in multiple locationsIn some cases, you might be required to make devices available in multiple locations andthen move the backup or direct the data to the appropriate devices. This scenario canprovide a faster and more robust backup service. However, these configurations are oftencomplex and might be difficult to configure, maintain, or troubleshoot. In thesesituations, it is often a choice between actively using and configuring the devices fornormal use or having the devices in a standby state for only disaster recovery use.

Normal use is defined as actively using the devices in all locations during normal, non-disaster recovery operations. This can make the configuration more complex andpresents operational and troubleshooting challenges. However, it does provide thebenefit of being able to use the device and ensures that the device is operational at thetime that it is required.

Standby use is defined as leaving the device in a a standby state where it is not used innormal operations. This can simplify the configuration, but the device might beinoperative when it is required. This configuration is also a less efficient use of resourcessince the devices are not used during normal operations.

Device failover

In both normal use and standby scenarios, device failover is an area that is often prone toerror and can require some manual intervention. Although some of these issues are easyto resolve, they should be documented, understood, and practiced.

When planning for disaster recovery, consider the following:

l Device access paths might be different, might change, or might disappear. They canall impact the configuration and might require additional steps to correct.

l Device names should identify the location or use. This can facilitate easiertroubleshooting and more reliable execution of disaster recovery procedures.


Method of connectivity 19

l Check the device status and availability. Devices that are not used regularly are morelikely to exhibit issues at a time when they are most required.Designing resilience into the backup service is a good practice and does not have toinclude idle devices. However, while this solution provides a better return ofinvestment and increases the available capacity and performance for running backupoperations, designing resilience also makes the solution more complex to configureand manage. Clustering and replication technologies are used to enhance resiliencein the backup environment but also to reduce complexity.

Implementing clustering and replication technologies in your disaster recovery planwill:

l Help to manage and automate the different elements such as disk storage andnetwork connections.

l Ensure that resources such as disk storage and network connections are available onthe correct hardware.

l Ensure that the resources and configurations are appropriate for the software servicethat is running.

Reliability and dependenciesThe reliability of a backup and recovery service depends upon the reliability of theindividual components, regardless of the chosen software, devices, and disaster recoveryapproach.

When designing a resilient backup and recovery service:

l Select devices that match the performance and operating expectations of the service.

n You can use multiple devices and multiple paths to improve reliability andavailability. Although this helps to eliminate single points of failure, it does notremove all of the single points of failure since no service can be completelyreliable.

n Careful design with consideration to the various disaster recovery scenarios willhelp to identify and eliminate the most common single points of failure.

l Consider the reliability of the expected duty cycle of the service and the componentsthat are used.

n Some devices cannot operate continuously, or may have limits on performance orfunctionality.

n Using some devices, such as physical tape devices, for excessive periods mightimpact their reliability.

n Appliances such as disk arrays, deduplication systems, or VTLs might also requiremaintenance periods in which backups cannot be performed or perform at areduced speed or rate.

l Consider regular maintenance.

n Issues might arise that will require some disruptive maintenance to the service.

n The ability of a service and subcomponents of that service to be taken offline,failed over, or to recover automatically will also ensure that the maintenance isboth performed and performed with minimal disruption to the service.

n Software patches and updates will be required to ensure optimum performance,reliability, and support.



CHAPTER 4

Disaster Recovery Use Cases


l Basic disaster recovery scenario............................................................................22l Basic disaster recovery considerations..................................................................24l More advanced disaster recovery considerations.................................................. 26l Clustered solutions............................................................................................... 28l Backup to disk...................................................................................................... 28l Index or configuration corruption.......................................................................... 30l Corruption or loss of SAN storage.......................................................................... 30l Loss of one server, Data Domain system, or site.................................................... 30l Replication solutions.............................................................................................30

Disaster Recovery Use Cases 21

Basic disaster recovery scenarioThis section describes a basic NetWorker implementation to highlight important disasterrecovery focus areas.

The following figure provides an example of a basic NetWorker solution that works wellfor a small office. If the server is powerful enough and the storage and connections aresized appropriately, it can protect 100 clients and a number of business systems.

In this example, the NetWorker server configuration offers very little resilience andhighlights a number of disaster recovery issues that might make recovery difficult or evenimpossible:

l The NetWorker server:

n Has a single ethernet connection and therefore is a single point of failure.

n Is using internal disks and therefore is a single point of failure.

n Has no mirroring or storage replication.

n Is contained to a single space within a room or a data center and therefore is asingle point of failure.

l The bootstrap email has not been configured and is not monitored, so the bootstrapbackup emails are lost.

l The bootstrap and index backups are written to a single tape, which has three yearsof bakups on it. The volume has not been changed or cloned and therefore is a singlepoint of failure.

l The single copy of the bootstrap is created for disaster recovery purposes every threemonths and is stored in the office Administrators desk in a different building.However, the secretary does not know the purpose of this tape and keeps it in alocked desk, in an office a few miles away from the main building and therefore is asingle point of failure.

Basic NetWorker solutionUnfortunately, in this example the management of the NetWorker server has been poorand little regard has been paid to the protection of the server.



In this example, the following issues might impede disaster recovery:

l Lack of resilience or redundancy in the backup environment. The NetWorker server isa single system and it uses RAID protected storage, but it is located locally through adirect attachment. This is the same for the tape devices that are located in a smallautoloader near the system.

l A loss of the site might result in a loss of the tape devices, the server, and thestorage. The customer in this situation only has one data room, so the use of asecond site is not viable.

l The customer does not remove tapes from the site. The tapes are cycled on a monthlybasis, but this is limited to a small number of monthly backups of key systems, withmost tapes remaining on site.

l Bootstrap backups have been configured to run daily and are written to an index andbootstrap tape. This tape is changed, but with staff changes and an increasingworkload, it is often left for several weeks. When it is changed, a new tape is labelledand the old tape is given to the office Administrator for storage. However, the officeAdministrator does not know the purpose of this tape and keeps it in a locked desk,in an office a few miles away from the main building.

l The bootstrap notifications have been configured to be sent by email. Unfortunately,no one monitors the email alias.

l The bootstrap notifications emails have failed for months and no one is aware of thissituation.

In the event of a significant disaster, the company in this example will find it extremelydifficult to recover its data and systems. Although some data is held offsite, the ability torecover it will rely on the NetWorker server and the infrastructure to be available.

While the hardware components may be quickly found, the ability to recover theNetWorker server to its previous state remains a challenge. The bootstrap tape from theoffice administrator’s desk can be used and may only recently have been changed. Theability to use this tape depends on someone knowing where the tape is and who to ask


Basic disaster recovery scenario 23

and the office administrator being available to unlock the desk and deliver the tape.Unfortunately, without any records of the bootstraps, the entire tape will have to bescanned to rebuild the records on the new NetWorker server which is a time-consumingprocess. Since the tape was stored in an area that fluctuated in temperature, read errorsmight occur and the recovery might not be possible.

Although this situation may seem extreme, it highlights the ways in which, withoutcareful consideration, a disaster recovery situation can have a major impact on thebusiness.

If the following procedures were put in place, the recovery would have been much easierand faster to achieve:

l Regularly change the bootstrap tape

l Clone copies of the bootstrap and client file indexes

l Save the bootstrap notifications

Although some data is likely to have been lost forever, key data could have allowed thebusiness to resume. Although it might not have been practical to have a second site withresilient links or remote storage, some simple measures with good management wouldhave made the recovery situation far easier and faster.

Now that we have considered a poor example, the following examples provideinformation on improved levels of disaster recovery protection.

Basic disaster recovery considerationsThe following steps to improve the availability of a NetWorker server can be simple andcost effective:

l Multiple paths for both network and storage connections are common and can helpto reduce the likelihood of a failure that is due to a bad connection or failed NIC orHBA.

l Most storage systems use RAID to prevent one or more disk failures from impactingthe system. These storage systems come in a range of sizes that suit any budget.Implementing these procedures should be considered as a no-cost option, althoughthe ongoing maintenance and management is likely to incur some expense. However,these options are very simple and cost effective and will have a big impact on thespeed and ease that a disaster recovery demands.

The following figure highlights some of basic steps that can be used to improve theavailability and disaster recovery capability of a NetWorker server. It shows a singlesite that is used for backup and recovery.

This example shows how a backup environment can be optimized to reduce singlepoints of failure and improve the speed and ability of a recovery, should a disasterrecovery be required:

l The bootstrap and index backups are cloned daily.

l Copies of the bootstrap and index backup clones are removed from the site andstored in a secure remote location.

l Dual Path Ethernet with automatic failover is configured and managed by a switch.This provides a single resilient IP connection.

l Email notifications are captured and stored in several locations and are availablefrom an archive.

l The backup service and backup operations are monitored daily for nonfatal errorsand warnings.



l A dual path SAN with a storage array that offers RAID protection, replication, andsnapshot capabilities is used.

Standard disaster recovery deploymentIn this example, the backup environment has been optimized to improve disasterrecovery performance in the following ways:

l The same single NetWorker server is made to be more resilient and robust by addingsome additional network and SAN links.

l The storage is RAID protected and has additional protection through snapshots,replication, and mirroring.

l Email notifications are sent to an alias that allows them to be accessed remotely.Email notifications are saved and monitored.

l Logs are monitored for errors so that issues can be detected early.

l Tapes are removed from site on a daily basis because there is only one site available.

l Tapes are stored in a secure and controlled location.

l Some data is cloned to ensure that multiple copies exist. This steps aids in recoveryand limits any exposure to media failure or loss.

l Bootstraps are cloned daily so that two copies always exist.


Basic disaster recovery considerations 25

More advanced disaster recovery considerationsThis section lists other options that build on resilience and offer higher levels ofprotection or recovery speed. In many cases, the recommendations from the previoussection will provide adequate protection and allow the backup service to be recovered ina reliable manner and in a reasonable period of time. For others, this might not provideenough protection or might not deliver a solution that is as quick or as resilient as thebusiness demands.

One of the best ways to improve recoverability and resilience is to introduce a secondsite. This practice allows the infrastructure and data to be present in two locations, whichhelps to mitigate the impact of an issue in a single site or with a single component withina site.

Single NetWorker server configured for two sitesThis figure provides an example of a basic layout of a single NetWorker server that isconfigured to use two sites, where:

l The same key infrastructure, such as SAN and network, is used.

l The infrastructure is configured with dual paths.

l The storage can be duplicated to provide the ability to replicate the NetWorkerconfiguration on the second site.

l Tape devices are used to store the bootstrap and index backups. These devices arelocated in a different building.

l To reduce recovery time significantly, the index storage can be replicated or madeavailable to the second site.

l To further reduce the unavailability of the backup and recovery service, add andcluster a second NetWorker server to make it highly available.



In this example:

l One of the sites has a passive or stand-by server, which sits idle until it is required.However, a similar configuration can also allow a clustered solution where both siteshave a node in which a cluster service is configured to run.

l The tape autoloader is the single point of failure in this example because it is locatedin one site. Although a second autoloader helps, it adds to the complexity of theconfiguration. Backup to disk solutions coupled with deduplication are better optionsin this environment.One of the challenges with using this configuration, or any configuration in which aproduction backup server must be protected, is the ability to capture the system in aconsistent manner. With backup and recovery operations taking place, the state ofthe server and the backup configuration files are in a constant state of change. Theonly way to reliably capture this information is to use the built-in bootstrap backupprocess.

While replicating the configuration files is possible, the operation might result in acrash-consistent state. The bootstrap backup is the only method to ensure that thedata is able to be recovered.

In this configuration, the SAN storage can be used to provide space for an AFTDdevice. These can be used for bootstrap backups and be cloned to the second site toensure that a consistent copy is available.


More advanced disaster recovery considerations 27

Clustered solutionsThe following figure illustrates how to maximize the benefit of having two sites whereeach site has a physical node with identical hardware.

In this example, the nodes are clustered together to provide a highly available servicethat is able to be active on either site. However, the tape device configuration is complexand it might be challenging to capture the system configuration in a consistent state.

Figure 2 NetWorker servers in a clustered environment

Backup to diskBackup to disk based solutions simplify the configuration and help to improve the abilityto capture the system configuration in a consistent state.

Although the diagram looks to be more complex, it provides a valid solution that helps tominimize the configuration complexity and helps to maintain a quick and easy failover inthe event of a disaster in one of the sites.

NetWorker server with backup to disk solutionThe following figure provides an example of a clustered solution that is similar to theprevious example, where a two node cluster is configured to host a NetWorker servicethat can run on either site. Although this example might seem excessive, it meets therequirements of a number of disaster recovery scenarios.



In this example:

l The primary backup storage devices have been replaced with Data Domain systemsthat allow backup to disk functionality with AFTD or DD Boost devices.

l The client file index information, media database, and various configuration files areall located on SAN storage which is presented to the appropriate node through a SAN.

l The SAN storage is replicated between the sites. This steps ensures that the storageis available even in the event of a site loss.

l There is still a requirement to store long term retention data on tape. Thisrequirement is achieved by using the secondary site.

l A tape unit is available to the local storage node in this site for tape out purposes.

l A weekly copy of the bootstrap and index backups is cloned to tape and sent offsite.

l The bootstrap and index backups are cloned to both Data Domain systems to ensurethat they are available in both sites.

l The bootstrap and index backups are performed regularly to the AFTD device. Thisstep ensures that the environment can become consistent in the event of a failoverand protects the backup service.


Backup to disk 29

Index or configuration corruptionBacking up the bootstrap and index backups on the AFTD will allow for rapid andimmediate recovery, if the media database, or configuration areas could be corruptedbecause of a fault or due to human error. Backing up the bootstrap and index backups onthe AFTD will allow for rapid and immediate recovery.

Consider that configuration corruption might make access to the DD Boost devicesdifficult, where an AFTD device is relatively easy to reconfigure.

Corruption or loss of SAN storageIf SAN storage is lost or corrupt, you can:

l Reconfigure the DD Boost devices.

l Configure the tape device, since you will have bootstrap backups on both DataDomains systems as well as the autochanger.

Loss of one server, Data Domain system, or siteIf the server, Data Domain system, or site is lost, it will not result in the loss of backupand recovery service.

If the site or single server loss is the result of a network, power, or cooling event, then theother site should allow the backup service to remain functional after a short delay toallow for the failover to occur. The loss may be temporary, in which case additionalrecovery actions might not be necessary. You can restore the replication and fail over sothat the main site is used once the problem resolved.

If the two sites are within a few miles of each other, you can use the tape out and offsitestorage.

Replication solutionsReplication is a term that is used differently by different vendors and replication solutionsare rarely the same. The features that are offered can be subtly different and requiredifferent parameters to operate. This section provides some basic background on thesupport and qualification of the various replication, mirroring, and snapshot features youneed to consider for disaster recovery of the NetWorker server.

When planning a replication solution, consider the potential impacts on the NetWorkerserver, which is constantly at work reading, changing, and updating information:

l Log files are updated with events and errors.

l Client file indexes are updated to reflect new backups or to remove backups thathave reached their expiration polices.

l The media database is updated to reflect the location and state of each volume used.

l Save set information is created, deleted, or changed.

l The general configuration is updated to reflect the current state of the NetWorkerserver with all its storage nodes, devices, and clients.

These activities require many IO operations on the server's disk. Any impacts on thespeed and reliability of the IO operations will impact the performance and reliability ofthe NetWorker server and the disaster recovery.



Replication, mirroring, and snapshot operations all require interception and capture ofany requested read, write, and change IOs that occur during the operation. Write IOsrequire extra processing not only for the disk updates but to confirm that the updates aresuccessful.

If the replication disks are local, the IO activity might take very little time, especially withadvanced array technologies. However, if the replication requires operations on systemsthat are separated by distance, the time required to perform and confirm the operationscan have a significant impact.

The NetWorker Performance and Optimization Planning Guide provides details on specificperformance requirements.

You can validate the performance impacts and support of replication solutions by aRequest for Product Qualification (RPQ), which you can submit through EMC ProfessionalServices.

Replication of the NetWorker ServerReplication is typically used in cluster configurations or where two separate hosts can actas the NetWorker server with one server being active and the other ready to take over theNetWorker services in the event of a primary server failure.

Composite hostids for the NetWorker serverA composite hostid is a single ID that enables both the active and the passive NetWorkerservers to use a single license. Active-passive configurations enabled in a clustered orreplicated NetWorker server environment are covered. Ensure that all NetWorker clientswithin the datazone have both NetWorker server nodes in their servers file.

The NetWorker Cluster Integration Guide provides details.

Synchronous replication technologiesAll synchronous replications add operational latency that can potentially impair theNetWorker server recovery. In typical service, the NetWorker server performs largeamounts of small random IO operations with 98% of writes below 1Kb. Even a smallincrease in disk request service time will impact performance and can lead to serverreliability issues. Only synchronous replications that prove they do not significantlyincrease service time can be qualified by EMC.

Note

If IP-based links or SAN routing exists in the synchronous replication environment, anRPQ is required. Examples of configurations that require RPQs include:

l SRDF/S or MirrorView/S over IP-based replicas (FCoE, FCIP, Ethernet, and so on).

l SRDF/S or MirrorView/S over remote FC-based replicas (SAN routing, DWDM, and soon).

Submit RPQs through EMC Professional Services.

Asynchronous replication technologiesAny hardware-based asynchronous or near-synchronous replication is supportedprovided the replication link has sufficient bandwidth for the replication to completewithout a restart. Storage failover during a restart period is not supported.

This support applies to any asynchronous replication technologies such as SRDF/A orMirrorView/A over any type of link.


Replication of the NetWorker Server 31

Network attached storageNAS device native replication technology is supported for NetWorker server databaseslocated on NFS (Unix/Linux) shares or on CIFS (Microsoft Windows) shares provided theconnection for both the NAS device and the secondary storage meets NetWorkerperformance requirements.

The NetWorker Performance Optimization and Planning Guide provides details.

Geo-replication technologiesBecause of the many possible variations, any array-based replication solution used withgeo-clustering such as SRDF/CE must be qualified on a case-by-case basis.

In general, the qualification is similar to an asynchronous replication configuration. Ifreplication is continuous, without restarts due to failure in link reliability or due toinsufficient bandwidth, geo-replication configurations are supported.

Submit RPQs through EMC Professional Serives.

Host-based replication technologiesAny host-based replication, also known as software-based replication, that uses softwarerunning on a host cannot be qualified for NetWorker server disaster recovery.

Software-based remote mirroring or replication of the NetWorker server databases willsignificantly impact IO latency and introduce known incompatibilities with some filter-level drivers. This includes such solutions as Symantec Veritas VxVM remote replica andEMC RepliStor.

Note

Do not attempt to recover the /nsr folder with a copy by using an operating system copycommand. The process can corrupt the media database files. Use a qualified replicationsolution.



Documents

EMC NetWorker 8.2 Server Disaster Recovery and Availability Best