Using the SRDF Adapter with VMware Site Recovery Manager 5.1

Using EMC SRDF Adapter for VMware vCenter Site Recovery Manager 5

Version 2.0

• SRDF Overview

• Installing and Configuring the SRDF SRA

• Initiating Test Site Failover Operations

• Performing Site Failover/Failback Operations

Cody Hosterman

Using EMC SRDF Adapter for VMware vCenter Site Recovery Manager 5.02

Copyright © 2010 - 2012 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information issubject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NOREPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THISPUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.

For the most up-to-date regulatory document for your product line, go to the Technical Documentation andAdvisories section on support.emc.com.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

Part number H10553.1

Contents

Preface

Chapter 1 Symmetrix Remote Data FacilityIntroduction ....................................................................................... 14SRDF overview.................................................................................. 15Three-site SRDF................................................................................. 17

Concurrent SRDF........................................................................17Cascaded SRDF...........................................................................18

SRDF/Star overview ........................................................................ 21SRDF/Star operations ................................................................23Concurrent SRDF/Star ..............................................................24Cascaded SRDF/Star..................................................................25

Chapter 2 Installation and ConfigurationIntroduction ....................................................................................... 28EMC SRDF Storage Replication Adapter prerequisites .............. 29

Supported Symmetrix arrays ....................................................29Solutions Enabler ........................................................................29

Supported SRDF Topologies ........................................................... 31Installing/configuring EMC Solutions Enabler ........................... 39

Configuring EMC Solutions Enabler on the VMware vCenterSRM server...................................................................................40Configuring a EMC Solutions Enabler SYMAPI server........40

Installing the EMC SRDF Storage Replication Adapter .............. 48Installing a new EMC SRDF Adapter for VMware vCenterSRM...............................................................................................48Upgrading an existing installation of EMC SRDF Adapter forVMware vCenter SRM ...............................................................50

Using EMC SRDF Adapter for VMware vCenter Site Recovery Manager 5.0 3

Contents

Configuring the EMC SRDF Storage Replication Adapter......... 53Array Manager configuration................................................... 53

Installing/configuring EMC VSI Symmetrix SRA Utilities........ 58vCenter permissions................................................................... 59VSI configuration........................................................................ 61VSI logging .................................................................................. 63

Configuring SRDF SRA behavior................................................... 67SRDF SRA Logging levels ......................................................... 71

Discovering and configuring Symmetrix replicated devices ..... 75Device discovery......................................................................... 75Device discovery advanced options ........................................ 79Consistency protection with EMC VSI Symmetrix SRAUtilities ......................................................................................... 85Consistency groups .................................................................... 86VSI consistency group creation algorithm.............................. 89Creating consistency groups with VSI .................................... 93Assisted troubleshooting of SRDF configurations with VSI ....107Recovery site device masking requirements ........................ 111

Configuring VMware vCenter SRM protection groups............ 113Configuring VMware vCenter SRM recovery plans ................. 116Configuring advanced SRM options ........................................... 117

Chapter 3 Testing Recovery PlansIntroduction..................................................................................... 124Test failover workflow in VMware vCenter SRM...................... 125

Requirements ........................................................................... 125Test ............................................................................................. 126Cleanup..................................................................................... 129

EMC VSI Symmetrix SRA Utilities with Test Failover.............. 135Test Failover options file ......................................................... 135EMC VSI Symmetrix SRA Utilities for vSphere Client ....... 137

Device Write Pacing and TimeFinder-based Test Failover ....... 149SRDF/A and device-level write pacing ................................ 149

Testing recovery plans using EMC TimeFinder/Clone ............ 154Testing recovery plans using EMC TimeFinder/Snap.............. 160Testing recovery plans using EMC TimeFinder/VP Snap........ 166

TimeFinder/VP Snap Overview ............................................ 166Configuring TimeFinder/VP Snap for test failover ............ 167

Test Failover Advanced Options .................................................. 174IgnoreActivatedSnapshots ...................................................... 175TestFailoverForce ..................................................................... 177


Contents

CheckForVirtualDisks ..............................................................178TerminateCopySessions...........................................................182

Test Failover without TimeFinder................................................. 183TestFailoverWithoutLocalSnapshots overview andconfiguration..............................................................................184TestFailoverWithoutLocalSnapshots with two-site SRDF..186TestFailoverWithoutLocalSnapshots with ConcurrentSRDF/Star ..................................................................................188TestFailoverWithoutLocalSnapshots with CascadedSRDF/Star ..................................................................................193TestFailoverWithoutLocalSnapshots with Concurrent SRDF(Non-Star)...................................................................................198TestFailoverWithoutLocalSnapshots with Cascaded SRDF(Non-Star)...................................................................................200TestFailoverWithoutLocalSnapshots andIgnoreActivatedSnapshots.......................................................202Protection during test failover with R2 devices ...................203

Testing recovery plans using GNS-enabled groups ................... 206Mirroring remote device group definitions ......................... 207

Test Failover with non-VMware devices ..................................... 210

Chapter 4 Gold Copy Protection During FailoverIntroduction ..................................................................................... 216Gold Copy considerations and requirements ............................. 217

Advanced option: FailoverIfGoldCopyFails.........................221Advanced option: CheckForVirtualDisks .............................222

Configuring recovery site gold copies ......................................... 228Device Write Pacing and recovery site gold copies .............239

Configuring protected site gold copies ........................................ 240

Chapter 5 Recovery Operations with Two-site SRDFIntroduction ..................................................................................... 248Recovery ........................................................................................... 249

Planned Migration—Two-site replication.............................250Disaster Recovery—Two-site replication ..............................256

Reprotection with two-site SRDF.................................................. 265Reprotect after Planned Migration .........................................265Reprotect after a temporary failure ........................................269Reprotect after a failover due to unrecoverable failure.......271

Failback............................................................................................. 273Recovery with non-VMware devices ........................................... 274

5Using EMC SRDF Adapter for VMware vCenter Site Recovery Manager 5.0

Contents

Chapter 6 Recovery Operations with Concurrent SRDF/StarIntroduction..................................................................................... 278Recovery........................................................................................... 279

Concurrent SRDF/Star configuration ................................... 280Planned Migration—Recovery with Concurrent SRDF/Star ..281Disaster Recovery ..................................................................... 289

Reprotection..................................................................................... 298Reprotect after Planned Migration ........................................ 298Reprotect after a temporary failure........................................ 300Reprotect after a failover due to unrecoverable failure ...... 303

Failback ............................................................................................ 304Failover with non-VMware devices............................................. 305

Chapter 7 Recovery Operations with Cascaded SRDF/StarIntroduction..................................................................................... 308Recovery........................................................................................... 309

Cascaded SRDF/Star configuration ...................................... 310Planned Migration—Recovery with Cascaded SRDF/Star311Disaster Recovery ..................................................................... 320

Reprotection..................................................................................... 330Reprotect after Planned Migration ........................................ 330Reprotect after a temporary failure........................................ 332Reprotect after a failover due to unrecoverable failure ...... 335

Failback ............................................................................................ 336Failover with non-VMware devices............................................. 337

Chapter 8 Recovery Operations with Concurrent SRDFIntroduction..................................................................................... 340Recovery........................................................................................... 341

Planned Migration—Recovery with Concurrent SRDF...... 342Disaster Recovery ..................................................................... 347

Reprotection and Failback ............................................................. 353Failover with non-VMware devices............................................. 354

Chapter 9 Recovery Operations with Cascaded SRDFIntroduction..................................................................................... 358Recovery........................................................................................... 359

Planned Migration—Recovery with Cascaded SRDF......... 360Disaster Recovery ..................................................................... 365


Contents

Reprotection and Failback.............................................................. 371Failover with non-VMware devices ............................................. 372

Chapter 10 Symmetrix Security with EMC SRDF AdapterIntroduction ..................................................................................... 376Symmetrix Access Controls ........................................................... 377

EMC SRDF SRA ........................................................................382EMC VSI Symmetrix SRA Utilities.........................................387

Symmetrix Authorizations............................................................. 389EMC SRDF SRA ........................................................................392EMC VSI Symmetrix SRA Utilities.........................................395

Appendix A Testing Recovery Plans with TimeFinder/MirrorTesting recovery plans using EMC TimeFinder/Mirror ........... 402

TimeFinder/Mirror and IgnoreActivatedSnapshotsadvanced option........................................................................408

7Using EMC SRDF Adapter for VMware vCenter Site Recovery Manager 5.0

Contents


Preface

VMware vCenter Site Recovery Manager (SRM) provides workflow andbusiness continuity and disaster restart process management for VMwarevSphere. The software leverages storage array replication and communicatesto replication management through a well-defined set of specifications. ThisTechBook discusses the best practices for using VMware vCenter SRMversion 5.1 with EMC SRDF Adapter version 5.1.

For information on VMware vCenter SRM version 4.1, refer to Using EMCSRDF Adapter for VMware vCenter Site Recovery Manager 4.1,located at http://support.emc.com.

As part of an effort to improve and enhance the performance and capabilitiesof its product lines, EMC periodically releases revisions of its hardware andsoftware. Therefore, some functions described in this document may not besupported by all versions of the software or hardware currently in use. Forthe most up-to-date information on product features, refer to your productrelease notes.

Audience This document is part of the EMC Symmetrix documentation set, andis intended for use by storage administrators, system administratorsand VMware vSphere administrators.

Readers of this document are expected to be familiar with thefollowing topics:

◆ EMC Symmetrix system operation.◆ EMC Solutions Enabler and EMC Unisphere for VMAX.◆ VMware vSphere products and their operation.

Organization The document is divided into ten chapters:


10

Preface

Chapter 1, “Symmetrix Remote Data Facility,” introduces the readerto SRDF technology with a focus on SRDF/S, SRDF/A and three-siteSRDF

Chapter 2, “Installation and Configuration,” discusses theinstallation, setup and configuration of the EMC SRDF StorageReplication Adapter for VMware vCenter Site Recovery Managerversion 5.1. This chapter also presents detailed best practices forinstalling and configuring Solutions Enabler for management ofSymmetrix storage arrays.

Chapter 3, “Testing Recovery Plans,” presents how to configure andexecute test recovery plans.

Chapter 4, “Gold Copy Protection During Failover,” presents how toconfigure gold copies for use during recovery plans failover.

Chapter 5, “Recovery with Two-Site SRDF,” discusses how toconfigure and execute failover with two-site SRDF configurations.

Chapter 6, “Recovery with Concurrent SRDF/Star,” discusses how toconfigure and execute failover with Concurrent SRDF/Starconfigurations.

Chapter 7, “Recovery with Cascaded SRDF/Star,” discusses how toconfigure and execute failover with Cascaded SRDF/Starconfigurations.

Chapter 8, “Recovery with Concurrent SRDF,” discusses how toconfigure and execute failover with Concurrent SRDF configurations.

Chapter 9, “Recovery with Cascaded SRDF,” discusses how toconfigure and execute failover with Cascaded SRDF configurations.

Chapter 10, “Symmetrix Security with EMC SRDF Adapter,”discusses how to use Symmetrix Access Controls and SymmetrixAuthentication to manage access and control of the Symmetrixstorage arrays visible to the VMware vCenter Site Recovery Managerserver.

Note: Examples provided in this guide cover methods for performingvarious VMware vSphere activities using Symmetrix systems and EMCsoftware. These examples were developed for laboratory testing and mayneed tailoring to suit other operational environments. Any proceduresoutlines in this guide should be thoroughly tested before implementing in aproduction environment.


Preface

Related documents ◆ Using EMC SRDF Adapter for VMware vCenter Site RecoveryManager 4.1

◆ VSI for VMware vSphere: Storage Viewer Version 5.4 Product Guide

◆ VSI for VMware vSphere: Storage Viewer Version 5.4 Release Notes

◆ VSI for VMware vSphere: Symmetrix SRA Utilities Version 5.4Product Guide

◆ VSI for VMware vSphere: Symmetrix SRA Utilities Version 5.4 ReleaseNotes

◆ VMware Site Recovery Manager Administration Guide

◆ VMware vSphere 5 Documentation: Introduction to VMware vSphere

◆ VMware vSphere 5 Documentation: Basic System Administration

◆ VMware vSphere 5 Documentation: Fibre Channel SAN ConfigurationGuide

◆ EMC Symmetrix Remote Data Facility (SRDF) Product Guide

◆ EMC Solutions Enabler Installation Guide

◆ EMC Solutions Enabler Symmetrix Array Management CLI ProductGuide

◆ EMC Solutions Enabler Symmetrix SRDF Family CLI Product Guide

◆ EMC Solutions Enabler Symmetrix TimeFinder Family CLI ProductGuide

Authors Cody Hosterman is a Senior Systems Integration Engineer in theEMC Enterprise Business Technology Integration team focusing onVMware and Symmetrix integration. He has been with EMC since2008. Cody has a bachelor's degree in Information Sciences &Technology from The Pennsylvania State University.

We'd like to hear from you!

Your feedback on our TechBooks is important to us! We want ourbooks to be as helpful and relevant as possible, so please feel free tosend us your comments, opinions and thoughts on this or any otherTechBook:

[email protected]


[email protected]

12

Preface


1

This chapter presents the following topics:

◆ Introduction ........................................................................................ 14◆ SRDF overview................................................................................... 15◆ Three-site SRDF.................................................................................. 17◆ SRDF/Star overview ......................................................................... 21

Symmetrix RemoteData Facility

Symmetrix Remote Data Facility 13

14

Symmetrix Remote Data Facility

IntroductionVMware vCenter Site Recovery Manager leverages storagearray-based replication such as EMC® Symmetrix® Remote DataFacility (SRDF®) to protect virtual machines in VMware vSphereenvironments. The interaction between VMware vCenter SiteRecovery Manager (SRM) and storage array replication (SRA) isgoverned through a well-defined set of specifications. TheseVMware-defined specifications are implemented by the storage arrayvendor as a lightweight application referred to as the storagereplication adapter.

EMC SRDF Adapter is an SRA that enables VMware vCenter SiteRecovery Manager to interact with an EMC Symmetrix storageenvironment. It allows VMware vCenter Site Recovery Manager toautomate storage-based disaster restart operations on Symmetrixarrays in a SRDF configuration. The EMC SRDF Adapter for VMwarevCenter Site Recovery Manager supports SRDF/Synchronous,SRDF/Asynchronous, Concurrent SRDF, Cascaded SRDF1 andSRDF/Star modes of operation.

The functionality of the adapter is enhanced through the use of theEMC Virtual Storage Integrator. The EMC Virtual Storage Integrator(VSI) Symmetrix SRA Utilities is an EMC plug-in to the VMwarevSphere Client that provides the ability to configure SRDF SRAoptions and features without requiring the user to resort to commandline utilities or applications outside of the VMware application suite.

IMPORTANT

This book assumes the reader has at a minimum generalknowledge of the VMware vCenter Site Recovery Managerproduct. For in-depth information refer to VMware documentationon www.vmware.com.

1. Support for non-Star Cascaded SRDF or non-Star Concurrent SRDF issomewhat restricted. Refer to chapter 2 for further details on supportedconfigurations.



SRDF overviewThe EMC Symmetrix Remote Data Facility (SRDF) family of productsoffers a range of Symmetrix-based disaster recovery, parallelprocessing, and data migration solutions.

SRDF disaster recovery solutions are based on active remotemirroring and dependent-write consistent copies of data maintainedat one or more remote locations. A dependent-write is a writeoperation that cannot be issued by an application until a prior, relatedwrite I/O operation completes. Dependent-write consistency isrequired to ensure transactional consistency when the applicationsare restarted at the remote location.

SRDF configurations require at least two Symmetrix systems. Intwo-site configurations, these systems are also known as the primaryand the secondary systems. Both sites can be located in the sameroom, in different buildings within the same campus, or hundreds tothousands of kilometers apart.

The EMC SRDF SRA supports two modes of two-site replication:

◆ SRDF/Synchronous

◆ SRDF/Asynchronous

SRDF/Synchronous (SRDF/S) is a disaster-restart solution thatoperates in synchronous mode and maintains a real-time(synchronous) mirrored copy of production data (R1 devices) in aphysically separated Symmetrix system (R2 devices) within an SRDFconfiguration. SRDF/S is a method of replicating production datachanges from locations less than 200 km apart. Synchronousreplication takes writes that are inbound to the source Symmetrix andcopies them to the target Symmetrix. The resources of the storagearrays are exclusively used for the copy. The write operation from thevirtual machine is not acknowledged back to the host until bothSymmetrix arrays have a copy of the data in their cache. SRDF/S is abuilding block of several multisite disaster-restart options such asSRDF/Star.

SRDF/Asynchronous (SRDF/A) mode provides a dependent writeconsistent copy on the target (R2) device, which is only slightlybehind the source (R1) device. SRDF/A session data is transferred tothe remote Symmetrix array in predefined timed cycles or delta sets,which minimizes the redundancy of same track changes beingtransferred over the link.

SRDF overview 15

16


SRDF/A provides a long-distance replication solution with minimalimpact on performance that preserves data consistency. In the eventof a disaster at the R1 site or if SRDF links are lost during datatransfer, a partial delta set of data can be discarded, preservingconsistency on the R2 with a maximum data loss of two SRDF/Acycles or less.

For more information on SRDF technology, including setup andconfiguration, refer to EMC Symmetrix Remote Data Facility (SRDF)Product Guide available at http://support.emc.com.



Three-site SRDFSRDF supports a variety of three-site solutions including Concurrentand Cascaded replication.

Concurrent SRDFEnginuity versions 5567 or higher support concurrent primarydevices (R11) that are configured with two R1 SRDF mirrors. Two R1SRDF mirrors allow the R11 devices to be mirrored concurrently totwo R2 devices that can reside in one or two remote Symmetrixarrays. Each R1 SRDF mirror is configured to a different SRDF group.

Note: The R11 devices have to be assigned to two different SRDF groupseven if the two partner R2 devices reside on the same remote Symmetrixarray.

Concurrent SRDF topologies are supported on Fibre Channel, GigabitEthernet (GigE), and ESCON SRDF topologies. Basic concurrentSRDF solutions require that each R1 SRDF mirror operate in the sameprimary mode, either both synchronous or both semi-synchronous,but allows either or both SRDF mirrors to be placed into one of theadaptive copy modes. Advanced concurrent SRDF solutions supportone or both R1 SRDF mirrors of the R11 device operating inasynchronous mode.

Figure 1 shows a concurrent SRDF topology in which the R11 devicecommunicates with the R2 device in Symmetrix B in synchronousmode. Concurrently, the same R11 device communicates with the R2device in Symmetrix C in one of the adaptive copy modes.

Concurrent SRDF supports the following topologies:

◆ Both legs are replicating in Synchronous mode

◆ Both legs are replicating in Asynchronous mode

◆ One leg is Asynchronous and the other leg is Synchronous1

◆ Adaptive Copy is also supported in place of Asynchronous orSynchronous.

1. This is the only topology supported by the SRDF SRA.

Three-site SRDF 17

18


Note: Not all revisions of Enginuity support all topologies. Please refer tothe SRDF Product Guide for specific restrictions.

Figure 1 Concurrent SRDF

Cascaded SRDFCascaded SRDF is a three-site disaster recovery solution where datafrom a primary (R1) site is synchronously mirrored to a secondary(R21) site, and then asynchronously mirrored from the secondary(R21) site to a tertiary (R2) site. The core benefit behind a cascadedtopology is its inherent capability to continue mirroring, withminimal user intervention, from the secondary site to the tertiary sitein the event that the primary site fails. This enables a faster recoveryat the tertiary site, assuming that the tertiary site is where you wantto restart production operations.



Cascaded SRDF uses dual-role SRDF devices (R21 devices) on thesecondary site which acts as both an R2 to the primary site and an R1to the tertiary site. The R2 SRDF mirror of the R21 device inSymmetrix B receives data from the R1 SRDF mirror of the R1 devicein Symmetrix A. The R2 SRDF mirror of the R2 device in SymmetrixC receives data from the R1 SRDF mirror of the R21 device inSymmetrix B. When data from Symmetrix A reaches Symmetrix B,both SRDF mirrors of the R21 device receive updates, but theirbehavior differs:

◆ After the R2 SRDF mirror of the R21 device receives updates, datais written to drives in Symmetrix B. If SRDF operates insynchronous mode, the SRDF emulation sends theacknowledgment to Symmetrix A.

◆ After the R1 SRDF mirror of the R21 device receives updates, theSRDF emulation sends data across the SRDF links to the R2 SRDFmirror of the R2 device in Symmetrix C as shown in Figure 6 onpage 34. When data is received in Symmetrix C, the R1, R21 andR2 device are synchronized.

Figure 2 shows an example of a Cascaded SRDF configuration.

Figure 2 Cascaded SRDF

The benefits of cascaded SRDF are:

◆ Faster recovery times at the tertiary site enabled by thecontinuous mirroring from the secondary site to the tertiary sitein the event of the primary site failure.

◆ Tight-integration with TimeFinder product family.

Three-site SRDF 19

20


◆ Management capability by using the current storage managementportfolio of software products.

◆ Geographically dispersed secondary and tertiary sites.



SRDF/Star overviewSRDF Symmetrix Triangular Asynchronous Replication (SRDF/Star)is a data protection and failure recovery solution that covers threegeographically dispersed data centers in a triangular topology.SRDF/Star protects business data against a primary site failure or aregional disaster using concurrent SRDF or cascaded SRDFcapabilities. These technologies mirror the same production datasynchronously to a nearby remote site and asynchronously to adistant remote site. This architecture can be expanded to includemultiple Symmetrix triangles.

SRDF/Star consists of three sites. The terminology that will be usedthroughout this book to refer to these three sites are:

◆ Workload site: The primary data center where the productionworkload is running.

◆ Sync target site: The secondary site usually located in the sameregion as the workload site. The production data is mirrored tothis site using synchronous replication.

◆ Async target site: The secondary site in a distant location. Theproduction data is mirrored to this site using asynchronousreplication.

SRDF/Star provides consistent data protection and incremental datarecovery between target sites in the event of a workload site failure ortransient fault (link failure). If a workload site becomes inoperable,SRDF/Star provides failover capability through SRDF/A recoverylinks that quickly reestablish data replication between the target sites.One target site resumes data replication for the workload site whilethe other resumes as a protected secondary target site. SRDF/Starrequires SRDF/A recovery links and an additional control host at oneof the target sites.

Each Symmetrix array in an SRDF/Star configuration uses dynamicSRDF devices that can function as either an R1 or an R2 device.During failure recovery, the R2 devices at either the synchronoustarget site or the asynchronous target site are dynamically convertedto R1 devices to become production devices at the new workload site.

SRDF/Star data consistency is achieved through a composite group(CG) that leverages Multi-Session Consistency or EMC Enginuity™

Consistency Assist technology to ensure all members in the CG areeither replicating or not replicating at any given point. A composite

SRDF/Star overview 21

22


group enabled for MSC or RDF-ECA consistency is referred to as anRDF consistency group. Devices in an RDF consistency group areprotected to preserve the dependent-write consistency of a database,which may be distributed across multiple SRDF platforms.

IMPORTANT

Since SRDF/Star environments are built on RDF consistency grouptechnology, an RDF daemon must be running on at least onecontrol host attached locally to each site. EMC stronglyrecommends running redundant RDF daemons on multiple controlhosts to ensure that at least one RDF daemon is available toperform time-critical, consistency monitoring operations. Byrunning redundant RDF daemons, you avoid service interruptionscaused by performance bottlenecks local to a control host, and linkfailures of the redundant RDF daemons and control hosts.

SRDF/Star environments provide advanced multi-site businesscontinuity protection. They combine SRDF technologies to enableconcurrent or cascaded SRDF/S and SRDF/A operations from thesame source volumes. The SRDF/Star operation is a combination ofhost software and Enginuity functionality that operates in aconcurrent or cascaded SRDF configuration.

SRDF/Star provides the following benefits and features:

◆ Sites can be geographically dispersed.

◆ SRDF/Star can span multiple RDF groups and Symmetrix arrays.

◆ RDF consistency groups maintain data consistency acrossSRDF/Star.

◆ In the event of a workload site failure, SRDF/Star enables you tofailover and resume asynchronous data transfer between theremaining target sites.

◆ Data is synchronized differentially, so the time to establish remotemirroring and consistency is minimal.

◆ In the event of a rolling disaster affecting the workload site, youcan determine which of the target sites (synchronous orasynchronous) holds the more current data and switch workloadoperations to that site.

The EMC SRDF SRA supports two types of SRDF/Starconfigurations; Concurrent SRDF/Star and Cascaded SRDF/Star.Diskless Cascaded SRDF/Star, otherwise known as SRDF/EDP



(Extended Distance Protection) is not supported with the SRDF SRA.The two supported configurations are discussed briefly in thefollowing subsections. For detailed information on setting up andconfiguring Star, refer to SRDF documentation.

SRDF/Star operationsControlling SRDF/Star involves tasks such as bringing up theSRDF/Star sites for normal operation, isolating one or more sites fortesting or other purposes, or switching the workload to one or moreof the remote sites after workload site failure. Understanding theseoperations are integral to understanding the different operationsperformed by the SRDF SRA. Users can perform these and otherSRDF/Star operations using the Solutions Enabler symstar commandor Symmetrix Management Console GUI1. Common operations arelisted and described in Table 1, “SRDF/Star operations.”

1. Unisphere for VMAX 1.5 does not yet support SRDF/Star operations

Table 1 SRDF/Star operations (page 1 of 2)

Name Results

Cleanup Cleans up internal meta information and Symmetrix cacheat the remote site after a failure at the workload site.

Configure Upgrades or transitions an existing SRDF/Star environment to employ R22devices, provided the current SRDF/Star environment is operating innormal condition.

Connect Starts the SRDF data flow

Disable Disables SRDF/Star consistency protection across the three sites.

Disconnect Suspends the SRDF data flow and transitions the path to adaptive copydisk mode.

Enable Enables complete SRDF/Star consistency protection across the three sites.

Halt Used to prepare the system for a planned switch of the workload to a targetsite. This action write-disables the R1 devices, drains all invalid tracks andMSC cycles so that Site A=Site B=Site C, suspends SRDF links, disablesall consistency protection, and sets adaptive copy disk mode.

Isolate Isolates one target site from the SRDF/Star configurationand makes its R2 devices read/write enabled to their hosts.


24


Concurrent SRDF/StarIn a concurrent SRDF configuration, data on a single source device isremotely mirrored to two target devices at the same time, providingtwo available copies of data. These mirrors operate independentlybut concurrently using any combination of SRDF modes.

Concurrent SRDF/Star environments provide a rapidreestablishment of replication operations in the event of a workloadsite failure. Rather than performing a full resynchronization betweenthe asynchronous and synchronous target sites, concurrentSRDF/Star performs a differential synchronization, which

List Lists each SRDF/Star composite group configuration,including workload name, mode of operation, CG and Starstates, and target names and states.

modifycg Maintains consistency protection when adding or removingdevice pairs from an SRDF/Star consistency group.

Protect Synchronizes devices between the workload and target sites and enablesSRDF/Star consistency protection to the specified target site.

Query Displays the status of a given SRDF/Star site configuration.

Reconfigure Transitions the SRDF/Star setup from concurrent SRDF tocascaded SRDF or vice versa after a site or link failure, or aspart of a planned event.

Reset Cleans up internal meta information and Symmetrix cacheat the remote site after transient fault (such as a loss ofconnectivity to the synchronous or asynchronous targetsite).

Show Displays the contents of the internal definition for a givenSRDF/Star site configuration.

Switch Transitions workload operations to a target site after a workload site failureor as part of a planned event.

Unprotect Disables SRDF/Star consistency protection to the specified target site.

Verify Returns success if the state specified by the user matchesthe state of the star setup.

Table 1 SRDF/Star operations (page 2 of 2) (continued)

Name Results



dramatically reduces the time it takes to remotely protect the newworkload site after the primary site failure. With ConcurrentSRDF/Star you can determine which site contains the most currentdata in the event of a rolling disaster that affects the workload site.Figure 3 illustrates a Concurrent SRDF/Star configuration. Theworkload site (New York) transfers data synchronously to the targetsite (New Jersey) and asynchronously to the distant target site(London). The recovery links are between London and New Jersey,and if an outage happens at New York, an SRDF/A session can beestablished between these two target sites.

Figure 3 Concurrent SRDF Star

Concurrent SRDF/Star is valuable for duplicate restarts and disasterrecovery, and provides increased flexibility for data mobility andapplication migrations.

Cascaded SRDF/StarCascaded SRDF is a three-way data mirroring and recovery solutionthat provides enhanced replication capabilities, greaterinteroperability, and multiple ease-of-use improvements. A cascaded


26


SRDF configuration does not require three separate site locations,although that is the most common configuration for a disasterrecovery solution.

Cascaded SRDF introduces the concept of the dual role R1/R2 device,referred to as an R21 device. The R21 device is both an R1 mirror andan R2 mirror, for use only in cascaded SRDF operations. Whenthinking of the R21 device, it is easier to understand the concept if itis thought of as a mirror type, instead of as a device. The controls forthese devices are relationship-based.

Figure 4 illustrates a Cascaded SRDF/Star configuration. In the eventof a synchronous target site loss, a differential data resynchronizationcan be performed between the workload site and the asynchronoustarget site. This is achieved by using change tracker technology at theworkload site to keep track of the data difference between theworkload site and the asynchronous target site. In the case ofCascaded SRDF/Star, the synchronous target site is always morecurrent than the asynchronous target site. In all cases, however, thechoice of which site to use in the event of a failure is left to thediscretion of the customer.

Figure 4 Cascaded SRDF/Star


2


◆ Introduction ........................................................................................ 28◆ EMC SRDF Storage Replication Adapter prerequisites................ 29◆ Supported SRDF Topologies ............................................................ 31◆ Installing/configuring EMC Solutions Enabler ............................ 39◆ Installing the EMC SRDF Storage Replication Adapter ............... 48◆ Configuring the EMC SRDF Storage Replication Adapter .......... 53◆ Installing/configuring EMC VSI Symmetrix SRA Utilities ......... 58◆ Configuring SRDF SRA behavior .................................................... 67◆ Discovering and configuring Symmetrix replicated devices ...... 75◆ Configuring VMware vCenter SRM protection groups ............. 113◆ Configuring VMware vCenter SRM recovery plans................... 116◆ Configuring advanced SRM options............................................. 117

Installation andConfiguration

Installation and Configuration 27

28

Installation and Configuration

IntroductionIn order to enable the processes and functionality offered by VMwarevCenter Site Recovery Manager that leverage SRDF, special softwaremust be installed and configured to create the interface betweenVMware and the underlying storage and replication.

EMC provides three main applications to enable, simplify andmanage this interface:

◆ EMC Solutions Enabler (required)

◆ SRDF Storage Replication Adapter (required)

◆ EMC VSI Symmetrix SRA Utilities (optional but recommended)

The installation and configuration of these tools is covered in thischapter. The installation and configuration of SRM itself is notcovered in this book with the exception of portions directly related tostorage and replication. For guidance on SRM installation andconfiguration not covered in this book please refer to VMwaredocumentation.

This book discusses using SRDF with VMware vCenter SRM version5. Throughout the book SRM will typically be referred to genericallyas version 5 and, unless otherwise stated, all discussions in this bookrefer to all minor versions of SRM between 5.0.0 to 5.1.0. Features orbehaviors introduced in a particular version of SRM 5 will bespecifically called out as unique to version 5.0.1 or 5.1.0.

Note: Examples of SRDF management in this book utilize either SolutionsEnabler CLI or Unisphere for VMAX. Unisphere for VMAX is thereplacement software for the Symmetrix Management Console. WhileUnisphere for VMAX includes the functionality that meets the need of mostcustomers, there is still advanced functionality that has yet to be included inUnisphere for VMAX. For those scenarios, examples will use SolutionsEnabler CLI or Symmetrix Management Console.



EMC SRDF Storage Replication Adapter prerequisitesThis book assumes that the user is running the 5.1 release of the EMCSRDF SRA Adapter and VSI Symmetrix SRA Utilities 5.4. As newversions are released, functionality and requirements can change, soit is important to consult the individual SRA release notes to verifysupport information.

Supported Symmetrix arraysThe EMC SRDF Adapter version 5.1 for VMware vCenter SiteRecovery Manager supports the following arrays1:

◆ EMC Symmetrix VMAX 40K™ storage arrays runningEnginuity™ version 5876 or later

◆ EMC Symmetrix VMAX 20K™ storage arrays running Enginuityversion 5874 or later.

◆ EMC Symmetrix VMAX 10K™/VMAXe storage arrays runningEnginuity version 5875.231 or later.

Refer to the EMC Support Matrix and SRDF Product guides forin-depth interoperability information.

IMPORTANT

The bit settings on the front-end Fibre Channel adapter play acrucial role in the proper operation of EMC SRDF Adapter. Pleaseconsult the EMC Support Matrix on www.emc.com for furtherinformation.

Solutions EnablerSolutions Enabler version 7.5 is the minimum version of the softwarerequired for EMC SRDF Adapter version 5.1.

The 32-bit version of Solutions Enabler must be installed on the SRMserver (even though the OS itself must be 64-bit) before installation ofthe SRDF Adapter. The adapter can be configured to use a remote

1. DMX support has been dropped in version 5.1 of the SRDF SRA. Userswho wish to use DMX arrays with SRM must remain on the SRDF SRAversion 5.0.1 with SRM 5.0.1.

EMC SRDF Storage Replication Adapter prerequisites 29

30


installation of Solutions Enabler which can then be 32 or 64-bit.Nevertheless, regardless of whether or not the SRDF SRA is using thelocal installation for SYMAPI calls or a remote one, the localinstallation of Solutions Enabler is still required and must be 32-bit.



Supported SRDF TopologiesThe SRDF SRA supports a wide variety of SRDF topologies thatcustomers can leverage to protect their VMware environment. TheSRDF SRA 5.1 adds four newly supported configurations to theexisting set.

Supported two-site SRDF configurations:

1. Synchronous SRDF (Figure 5)

2. Asynchronous SRDF (Figure 6)

Supported three-site SRDF configurations:

1. Concurrent SRDF/Star

• Recovery between workload and synchronous site (Figure 7)

• Recovery between workload and asynchronous site (Figure 8)

2. Cascaded SRDF/Star

• Recovery between workload and synchronous site (Figure 9)

• Recovery between workload and asynchronous site (Figure10)

3. Concurrent SRDF (Figure 11)

4. Cascaded SRDF (Figure 12)

Since VMware vCenter Site Recovery Manager is, by design, atwo-site paradigm, only two of the three SRDF sites in a three-sitesolution can be directly managed by the SRDF SRA. In other words,recovery can only occur between two of the three SRDF sites andwhich sites those are depends upon by where the compute resources1

are physically located.

While only the recovery ESX hosts must be in the same physicallocation as the configured recovery array, the vCenter server andSRM server can be located anywhere, this is highly discouraged. Ifthese management applications are in a physically separate locationfrom the recovery storage and compute, it is possible that they couldbe in the failure domain of the protected site resources. If so, a

1. In three-site SRDF configurations, (Star or non-Star) there is norequirement by SRM or the SRDF SRA to have any compute ormanagement applications at the bunker site.

Supported SRDF Topologies 31

32


disaster would render failover through SRM and the SRDF SRAimpossible. It is important to locate the vCenter and SRM servers inthe same domain as the recovery storage and compute.

The SRDF SRA only supports managing certain types of three siteconfigurations. The following diagrams lay out the supported SRDFconfigurations and recovery locations. In all of the diagrams, SRMserver 1 is the protected site SRM server and SRM server 2 is therecovery site SRM server, further more, the storage, compute andmanagement applications (Solutions Enabler, SRM, vCenter) are alllocated in the same physical location.

IMPORTANT

Although the SRDF SRA supports failing over to different sites ofSRDF/Star, the choice of which Star site is the recovery site is not adecision made at the time of failover but rather dictated by thelocation of the compute resources. The SRDF SRA can only failovervirtual machines to the array where the recovery hosts arephysically located.

Figure 5 Synchronous SRDF Support



Figure 6 Asynchronous SRDF Support

Figure 7 Concurrent SRDF/Star Support--Failover to Synchronous site


34


Figure 8 Concurrent SRDF/Star Support--Failover to Asynchronous site



Figure 9 Cascaded SRDF/Star Support--Failover to Synchronous site


36


Figure 10 Cascaded SRDF/Star Support--Failover to Asynchronous site



Figure 11 Concurrent SRDF Support--Failover to Asynchronous site


38


Figure 12 Cascaded SRDF Support--Failover to Asynchronous site



Installing/configuring EMC Solutions EnablerThe EMC SRDF Adapter for VMware vCenter Site Recovery Managerutilizes Solutions Enabler software to perform the discovery andmanagement of the Symmetrix storage arrays on behalf of VMwarevCenter Site Recovery Manager. The Solutions Enabler software usesin-band commands to small devices called gatekeepers to manipulatethe Symmetrix storage arrays. Therefore, Solutions Enabler must beinstalled on a host that has access to gatekeepers from the storagearray that hosts the replicated devices presented to the VMwareenvironment. The simplest configuration provides the server runningVMware vCenter Site Recovery Manager with direct connectivity tothe relevant storage arrays.

The more common deployment of Solutions Enabler for EMC SRDFAdapter for VMware vCenter Site Recovery Manager is anticipatedto be a client-server model. In this model, the EMC SRDF Adapterutilizes the Solutions Enabler API server running on a remote hostthat has direct connectivity to the storage arrays. The communicationbetween the Solutions Enabler commands executed by the EMCSRDF Adapter for VMware vCenter Site Recovery Manager andSolutions Enabler API server occurs over a TCP/IP network. For easeof management, setup, and use, EMC offers the Solutions EnablerVirtual Appliance which can be used for the remote SYMAPI servers.Solutions Enabler can also be installed on any virtual or physicalmachine that is running a supported operating system. For details onthese methods refer to the Solutions Enabler Installation Guide.

Note: The terms SYMAPI server is simply the name given to a SolutionsEnabler installations that has its server daemon turned on and configured. ASYMAPI server is a Solutions Enabler installation that can service remote APIrequest from remote clients.

Installing/configuring EMC Solutions Enabler 39

40


Configuring EMC Solutions Enabler on the VMware vCenter SRM serverThe EMC SRDF Adapter for VMware vCenter Site Recovery Managerdoes not utilize Solutions Enabler CLI commands to manage theSymmetrix storage arrays. The adapter utilizes API calls to performany storage control operations. This feature optimizes theperformance of the adapter for large replicated environments. Theseare the types of environments frequently encountered whenSymmetrix storage arrays are deployed in enterprise solutions.

Although the EMC SRDF Adapter uses the Solutions Enabler API, theadapter still requires the Solutions Enabler libraries that are installedwith Solutions Enabler to execute the API calls. Therefore, SolutionsEnabler software has to be installed on the VMware vCenter SiteRecovery Manager server.

Configuring a EMC Solutions Enabler SYMAPI serverRegardless of whether the SYMAPI server is locally installed on theSRM server or is on a separate, dedicated server there are a fewsimple configurations required before it can service the EMC SRDFSRA. Note that these steps are only required on the Solutions Enablerinstallation with gatekeepers and are not required on the client-sideSolutions Enabler install in a client/server configuration.

These steps should be performed on both the protected and recoverySYMAPI server installations. For more information on any of thesesteps or options please refer to the Solutions Enabler InstallationGuide on support.emc.com

Gatekeepers Solutions Enabler is an EMC software component used to control thefeatures of Symmetrix arrays. Solutions Enabler receives userrequests from CLI, GUI, or other means, and generate low-levelcommands (syscalls) that are transmitted to the Symmetrix array foraction.

Gatekeeper1 devices are Symmetrix devices that act as the target ofcommand requests from Solutions Enabler. These commands arrivein the form of disk I/O requests (and so a “disk” must be named bythe host as the “address” or target of that command). The morecommands that are issued from Solutions Enabler, and importantly

1. For more detailed information on gatekeepers refer to EMC knowledgebase article emc255976 on support.emc.com.



the more complex the actions required by those commands are, themore Gatekeepers and/or array processor resources that are requiredto handle those requests in a timely manner.

Gatekeepers require only a small amount of space, 3 MB (3 cyl). Usersare discouraged from building Gatekeepers in larger sizes as thesmall size is used by Enginuity to automatically identify and usedevices as Gatekeepers.

While in most cases Gatekeeper devices must be mapped andmasked to single servers only and should not be shared acrossservers, virtualized environments offer an exception. VMwareenvironments permit the movement of a virtual machine from onephysical ESX server to another (VMware vMotion) and if that virtualmachine is executing Symmetrix management or control operations,the Gatekeepers being used must be visible to that virtual machine onwhichever physical server it is running upon. As a result, theGatekeeper devices must be made visible to any ESXi server the guestmay operate upon. Note that while each ESX server must be able tosee the same Gatekeepers, they may not be shared among the virtualmachines.

The minimum requirement of six Gatekeepers should provesufficient for most SYMAPI servers, except, for instance, thosemanaging Consistency Groups. Since most SRM SYMAPI serversutilize Consistency Groups, additional devices are usually required.Here is how to calculate the required Gatekeeper count:

[Number of CGs] * 0.3 = Gatekeepers needed for Consistency Groupoperation

Round up the result of calculations to whole integers and then do thefollowing addition:

6 + (GKs needed for CG operation) = total number of Gatekeepers toprovision.

Daemonconfiguration

The storsrvd daemon (allows for and receives remote SYMAPIconnections) and storrdfd daemon (provides consistency protectionfor RDF device configurations) should be installed and configured.Since these daemons are not enabled by default in standard SolutionsEnabler installations, with the notable exception of the vApp version,they may not be active. These daemons can be installed by executingthe following commands on the server(s):

stordaemon install storsrvd -autostartstordaemon install storrdfd -autostart


42


These commands must be run from binary location of the SYMCLIcommands (if not already in the path), usually /usr/symcli/bin inUnix or C:\Program Files\EMC\SYMCLI\bin in Windows.

The USE_RDFD parameter must also be enabled in the options file in/var/symapi/config in Unix or C:\ProgramFiles\EMC\SYMAPI\config1 in Windows to enable the use of thestorrdfd.

Nethost configuration In order to control and restrict trusted-user and node access2 anoptional nethost file can be created in the SYMAPI serverconfiguration directory. When this file exists, only the nodes/userslisted in this file are allowed to connect to the server to executeremote SYMAPI functions. If the security functionality provided bythe nethost file is not desired, the file itself can just be deleted fromthe SYMAPI server.

The identity of a client host may vary from the perspective of theserver, since the server can accept connections from IPv4 and IPv6clients. Thus the network address of the client could be an IPv4 orIPv6 address. If you have decided to specify the network address inthe nethost file instead of the node name, then the exact syntax of theaddress is important. If you incorrectly specify an address,connections from some clients may be denied (see the end of thissection for more information on troubleshooting). In general,specifying the node name (or the FQDN) is advised, since properDNS configuration will usually ensure that the name of the host willbe consistent, regardless of the network address of the client.

In addition to specifying the node, each user that requires access froma particular host must also be authorized.

For ease of use, an asterisk (*) can be entered as the user or host andthis will allow all users or all hosts (or a combination of the two) tohave API access. It is important to note that the Solutions Enablervirtual appliance does not support entering an asterisk for the host

1. The root drive letter may change depending on the location of theinstallation. C:\Program Files is the default.

2. The nethost functionality is a per-SYMAPI server feature. It is unrelated toand performs a different function than Symmetrix Access Control Listsand Symmetrix Authorizations. These features are discussed later in thisbook. Nevertheless, these three security features can be used in acomplementary fashion.



entry—this is only currently supported in standard installations. Thevirtual appliance does support entering an asterisk as the userthough.

For SRM environments, any user accounts that operate VSI and theaccount running the VMware vCenter SRM service (this is theaccount that the SRDF SRA, when called, runs under) must beentered into the nethost file (only if the file exists OR the vApp is inuse). The protection and recovery side SRM server FQDNs and therespective accounts used on both should be added to both theprotection and recovery side SYMAPI servers. This will allow for fulluse of VSI and the SRDF SRA.

Traditional installations of Solution Enabler require the user tomanually edit or create the file in a standard text editor using thesyntax displayed in Figure 13.

Figure 13 Nethost file

If the file is not present in the config folder the user must create a newfile entitled nethost with no file extension/type.

As an example, imagine the following SRM environment:

Protected side information

◆ SRM/VSI admin username: john

◆ SRM service account name: srm1

◆ SRM server FQDN: srm1.ebc.emc.local

Recovery side information

◆ SRM/VSI admin user username: lisa

◆ SRM service account name: srm2

◆ SRM server FQDN: srm2.ebc.emc.local


44


If the storage admin wanted to limit SYMAPI access to just the SRMservice accounts, Lisa and John, the nethost file on both SYMAPIservers should look like Figure 14.

Figure 14 Nethost example with limited user access

If the storage admin wanted to limit SYMAPI access to the SRMservers and anyone who was able to log on to them, the nethost fileswould look like what is shown in Figure 15.

Figure 15 Nethost example with full user access

Note: The nethost file does not distinguish between domain and local hostaccounts and therefore no domain prefix for the user is needed or allowed.

The Solutions Enabler Virtual Appliance is slightly different in that itoffers a web-based interface to edit the nethost file as seen inFigure 16.



Figure 16 Solutions Enabler Virtual Appliance nethost entry

Unlike standard installations of a SYMAPI server, the virtualappliance enforces the use of the nethost file. Since the file system isprotected, the nethost file is present upon initial deployment of thevApp therefore enforcing its use.

Troubleshooting nethost permissionsIf the nethost information is incorrect or missing, VSI or the SRDFSRA will report an error such as the one below in their log files:

Failed connection to SYMAPI server(192.168.160.127:2707). Error = The trusted hostfile disallowed a client server connection.

The trusted host file is another name for the nethost file. This errormeans that either the host or user was incorrectly entered or notentered at all. Verify the listings in the nethost file or refer to thestorsrvd log on the target SYMAPI server for a message such as theone below that indicates a blocked session due to a nethost issue:

<Error> [1232 API_Listener] Jul-09 16:46:44.488 :ANR0018E Rejecting session 927 [email protected]: The trusted hostfile disallowed a client server connection

Note the bold portion which indicates the user (before the @ symbol)and the host FQDN (after the @ symbol). If what is displayed in thestorsrvd log is the correct user/host, ensure that they are entered intothe nethost file in the proper format as discussed above.


46


Troubleshootingremote SolutionsEnabler service

When connecting remote clients to a remote SYMAPI server, commonsituations can occur that prevent remote access from a client. Table 2shows the typically encountered errors and the usual resolution tothe issue.

Certificate errors can be handled in one of two ways; either the clientcertificate must be fixed or the SYMAPI server can be altered toignore invalid certificates. To alter SYMAPI certificate handling thefollowing process should be used:

1. Log on to the SYMAPI server

2. Navigate to <install location>\EMC\SYMAPI\config

3. Open the daemon_options file (using a standard text editor likeVI or Wordpad)

4. Find the line storsrvd:SECURITY_CLT_SECURE_LVL = VERIFY

Table 2 Common client/server SYMAPI connection errors

Error message Common explanation

The remote client/server handshakefailed.

Typically caused by inconsistentinformation in the client hostsecurity certificates.

The remote connection is refused.The server may not be running.

Caused by the storsrvd servicenot running on the target serveror the incorrect port was entered.In some occasions SYMAPIservers are configured to not usethe default port of 2707.

A network data operation failed Caused by incorrect firewallsettings or a network outage.

The server refused a connection dueto a security level mismatch

Caused by a difference in settingsbetween the target SYMAPIserver's storsrvd daemon securitylevel and the client's securitysettings. It is recommended thatboth be set to SECURE.

Host system not recognized orauthenticated

The hostname entered wasincorrect or has no DNS entry.

The trusted host file disallowed aclient server connection

The SYMAPI server’s nethost fileis incorrectly configured.



5. Uncomment the line by removing the preceding pound sign (#)and change the value from VERIFY to NOVERIFY

6. Restart the storsrvd service with the following solutions enablercommands:

stordaemon shutdown storsrvd

stordaemon start storsrvd

If there is a security mismatch, the SYMAPI server behavior mayneed to be changed. To alter SYMAPI certificate handling thefollowing process should be used:

1. Log on to the SYMAPI server

2. Navigate to <install location>\EMC\SYMAPI\config

3. Open the daemon_options file (using a standard text editor likeVI or Wordpad)

4. Find the line storsrvd:SECURITY_LEVEL = SECURE

5. Uncomment the line by removing the preceding pound sign (#)and change the value from SECURE to NONSECURE or ANY.

6. Restart the storsrvd service with the following solutions enablercommands:

stordaemon shutdown storsrvd

stordaemon start storsrvd

Changing the SYMAPI security setting mentioned above in step 4may affect/block other applications remotely accessing that SYMAPIserver. Change this option with care.


48


Installing the EMC SRDF Storage Replication AdapterThe EMC SRDF Adapter allows for a new installation or an upgradeof an existing installation. Depending on which method is used, theconsiderations for installation are slightly different. This sectiondiscusses both options.

Installing a new EMC SRDF Adapter for VMware vCenter SRMThe EMC SRDF Adapter for VMware vCenter Site Recovery Managerhas to be installed on both servers running the VMware vCenter SiteRecovery Manager software.

InstallationThe installation of the adapter is executed using the standardWindows InstallShield utility and must be run on both the protectedand the recovery site. The software distribution consists of a singlefile that performs the installation of EMC SRDF Adapter.

Version 5.1 of the SRDF SRA automatically determines the directoryin which the adapter has to be installed. This is determined by usingthe information in the registry and the configuration file for VMwarevCenter SRM. The installation directory is displayed by the installerbut cannot be changed.

The successful installation of version 5.1 of the adapter results in thecreation of two files in the install directory/directories. Theinstallation places the adapter executable itself (EmcSrdfSra.exe) andan associated interface file (command.pl) within the SRA installdirectory indicated by SRM.

In the SRM 5.1 release, the architecture of the SRM software has beenupgraded to 64-bit. This means two things; the SRM server must beinstalled on a 64-bit OS and the install file directories of SRM and theSRA will be in different locations as compared to earlier versions.Therefore, a new install of the SRDF SRA in 5.0.x and 5.1.x willbehave slightly different. Those differences are outlined below.

SRM 5.0.xThe two files placed during the SRA install for SRM 5.0.x(EmcSrdfSra.exe and command.pl) will be saved together in the samedirectory under:



<install location>/VMware\VMware vCenter Site RecoveryManager\storage\sra\EMC Symmetrix\

SRM 5.1.xThe two files placed during the SRA install for SRM 5.1.x(EmcSrdfSra.exe and command.pl) will be saved in separate places inthe file system.

The command.pl file that is used by SRM to call the SRDF SRAexecutable is in the SRA directory of SRM:

<install location>/VMware\VMware vCenter Site RecoveryManager\storage\sra\EMC Symmetrix\

The SRA itself (EmcSrdfSra.exe) is placed in a separate directorycreated by the SRA installation process:

C:\Program Files (x86)\EMC\sra\

Options filesFurthermore, the installation places four options files into the EMCapplication folder deemed by the operating system(“%programdata%\EMC\EmcSrdfSra\Config” in Windows 2008).Unlike the install files of the SRA, the location of the options files isthe same regardless of the version of SRM (5.0 or 5.1).

These files are:

◆ EmcSrdfSraGlobalOptions.xml —Global options for adapterbehavior during SRM operations. These are advanced settingsthat should only be edited with discretion.

◆ EmcSrdfSraTestFailoverConfig.xml —File for test failoverdevice configuration pairings

◆ EmcSrdfSraProtectionSiteGoldcopyConfig.xml —File for goldcopy device configuration pairings of R1 devices

◆ EmcSrdfSraRecoverySiteGoldcopyConfig.xml—File for goldcopy device configuration pairings of R2 devices

These configuration files are discussed in detail in the relevantsections later in this book.

Installing the EMC SRDF Storage Replication Adapter 49

50


Upgrading an existing installation of EMC SRDF Adapter for VMware vCenter SRMThe installer for EMC SRDF Adapter supports an upgrade of the SRAfrom earlier versions to the latest, version 5.1. The process for theupgrade is similar to a fresh install of the adapter discussed in theprevious section with the exception that the old files will be removedand the new SRA 5 files will be loaded into the new directories.

The 5.1 version of the SRDF SRA introduces a number of new globaloptions and other enhancements so current options files will bereplaced with new versions. When upgrading from a previousversion of SRDF SRA 5, the older XML files will not be deleted butwill be renamed to .bak files. If the user needs to reference oldconfigurations these archived files can be viewed with any standardtext editor.

Upgrading SRM 4.x toSRM 5.x

It is important to note that protection groups and recovery plans willnot be affected by the upgrade of SRM 4 to 5. SRM 5 includesmethodologies to ensure these remain intact after upgrade.Conversely, the information populated into the options files will notbe preserved after upgrade. Test Failover pairings, gold copyinformation or advanced global options must be reconfigured afterinstall.

After the upgrade of SRM from 4 to 5, an option to export arraymanager information is provided. If the array manager informationhas not changed, utilize this XML file to ensure proper configuration.Refer to SRM 5 documentation for detailed information on upgradingSRM.

Note: SRM 4 or 4.1 cannot be upgrade directly to SRM 5.1. SRM 4.xinstallations must be first upgraded to SRM 5.0 or 5.0.1 and then can beupgraded to 5.1.

Upgrading from SRM5.0.x to SRM 5.1

As mentioned previously, SRM 5.1 is now a 64-bit1 application andthe file directories of SRM and the SRA will be in different locationsas compared to earlier versions.

1. The 64-bit architecture of SRM 5.1 requires a 64-bit open databaseconnectivity (ODBC) database source name (DSN) to connect to the SRMdatabase. The creation of a 64-bit DSN to the current database is requiredfor an upgrade to SRM 5.1.



An upgrade to SRM 5.1 will preserve protection group and recoveryplan information as well as any configured array managers. Noreconfiguration of these items is normally required by this upgradepath. For more information please refer to VMware documentation.

SRM 5.1 upgrade preserves the following settings and configurationsthat were created with the previous release.

◆ Datastore groups

◆ Protection groups

◆ Inventory mappings

◆ Recovery plans

◆ IP customizations for individual virtual machines

◆ Custom roles and their memberships

◆ SRM object permissions in vSphere

◆ Custom alarms and alarm actions

◆ Test plan histories

◆ Security certificates

◆ Mass IP customization files (CSVs)

IMPORTANT

During an upgrade, SRM 5.1 preserves only protection groups andrecovery plans that are in a valid state. SRM discards protectiongroups or recovery plans that are in an invalid state.

Since the architecture of SRM has changed, the install files will berelocated and accordingly the directory for SRAs will be in a newlocation. Therefore, even if the SRDF SRA is currently installed priorto the upgrade, the SRA will always (regardless of version) need to bere-installed after SRM has been upgraded to 5.1. The SRA installerwill recognize the new install location of SRM and will place itscommand.pl file in the appropriate folder in the SRM SRA directory.Since the SRDF SRA itself remains a 32-bit application the SRDF SRAwill be placed in a separate directory in the 32-bit program files folder(default is C:/Program Files (x86)/EMC/sra/).

Note that until the SRDF SRAs are installed on both sides and thearray managers configured, previous protection groups will appearas invalid. Once array managers have been configured and a devicediscovery has been run the protection groups will be resolved.

Installing the EMC SRDF Storage Replication Adapter 51

52


IMPORTANT

Only the SRDF SRA 5.1 and later is supported with SRM 5.1.Earlier versions of the SRDF SRA are not compatible with the64-bit architecture of SRM 5.1.



Configuring the EMC SRDF Storage Replication AdapterThe configuration of the EMC SRDF Adapter is performed throughthe Array Managers wizard in the VMware vCenter Site RecoveryManager plug-in. The wizard should be invoked only after theconnection between the VMware vCenter Site Recovery Manager atthe protected site and recovery site has been established.

Array Manager configurationBefore protection groups or recovery plans can be configured, anarray manager must be configured for each instance of the SRDF SRAto provide connectivity information for device discovery. The AddArray Manager wizard for configuring the array managers isinvoked by right-clicking on a SRM server under the Array Managersmenu. In the array manager wizard enter a descriptive name for thearray manager. The option EMC SRDF Adapter for VMware SRM isdisplayed if the EMC SRDF Adapter for VMware vCenter SiteRecovery Manager has been properly installed. The process forconfiguring the EMC SRDF Adapter for VMware vCenter SiteRecovery Manager is detailed in Figure 17. Readers can perform thisin their environment by following the numbers associated with eachstep in the process. This must be completed for both SRM serversbefore continuing on.

Configuring the EMC SRDF Storage Replication Adapter 53

54


Figure 17 Initiating the Configure Array Managers wizard

The configuration of the storage arrays connected to the protectedsite can be determined by entering the hostname or IP address of thelocal and remote servers providing the EMC Solutions Enabler APIservices and selecting Next. The text local can be entered in lieu of ahostname if the VMware vCenter Site Recovery Manager host hasdirect access to Gatekeepers.

Note: If the Solutions Enabler API service is configured to listen on anon-default TCP port, a non-default port number can be optionally specifiedby appending the port number after a colon following the hostname/IPaddress of the SYMAPI server. For instance if the SYMAPI server isconfigured to listen at a non-default port such as 2739 the entry would besimilar to this: symapiserver1.emc.com:2739



Once the array manager has been configured on the first site, areciprocal array manager must be configured on the remote site. Inthe second array manager, the SYMAPI servers should be reversedfrom the first (the local SYMAPI server address is now the remoteaddress and vice versa).

IMPORTANT

It is sometimes tempting to have just a single SYMAPI serverprovide API services for both the local and remote Symmetrixarrays to provide simplicity. This is not usually possible inlong-distance configurations but in the case of campusconfigurations or iSCSI Gatekeepers it may be. DO NOT do this—itis not supported and WILL cause the SRDF SRA to malfunction.The local and remote SYMAPI servers must be distinct servers inorder to provide resiliency for disaster recovery (among otherreasons). Each SYMAPI server must see its proximate arrays as localand their respective remote arrays as remote.

After initial setup of the array managers the SRA queries for activeRDF groups from the discovered arrays on the local SYMAPI server.The SRDF SRA log will report messages, such as the one below, if anRDF group is successfully discovered.

[DiscoverArraysCommand::Run] Found a remote Symmetrix

[000192603603] connected through director [RF-8H], remote

director [RF-8H] and RA group [30], remote RA group [30]

After both array managers have been configured the array pairs mustbe enabled within SRM. Select one of the paired array managers andnavigate to the Array Pairs tab on the right. This process is shown inFigure 18 and only needs to be performed once per array pair on oneof the two array managers (it doesn’t matter which one). Once a pairhas been successfully enabled for one array manager that pair will beenabled on the other respective array manager.

It is important to note that these array pairs must be enabled in orderto discover devices from them. Therefore, for two site replicationenvironments the array pair including the source and target arraymust be enabled. For three site solutions only the array pair including


56


the workload site and the chosen target site must be enabled. Anypair including the third “bunker” site does not need to be (andtypically can’t be) enabled.

Figure 18 Enabling array pairs for the SRDF SRA

If no array pairs show up this usually indicates a configurationproblem. Refer to the SRDF SRA logs for detailed information buttypically it is one of the following reasons:

1. Incorrect SYMAPI server address or lack of Gatekeepers from thecorrect array(s) to the SYMAPI server.

2. The same SYMAPI server is configured as both the local andremote SYMAPI servers

3. Symmetrix array is not properly configured.

a. Symmetrix Remote Adapters are not online

b. SPC-2 is not enabled on the Fibre Adapters or Initiators

c. No RDF groups are configured.

4. An array manager is not configured on one of the two SRMservers.

5. The environment is set to failover to the Asynchronous site of aStar environment and the advanced option to allow this is notenabled. The global option FailoverToAsyncSite must be enabledat both sites for the SRDF SRA to discover the correct arrays.Instructions on configuring this option is discussed in the sectionentitled “Configuring SRDF SRA behavior” on page 67 of thischapter.



If one or more array pairs fail to enable, it means that the remote SRMserver does not have an array manager that has discovered the samepairs. Ensure that the remote SYMAPI server has the proper access tothe correct arrays as described in the boxed note on page 55.

Array Manager configuration with three site SRDFFor SRDF/Star or standard three-site SRDF environments, theprotected site array manager should always have the workload siteSYMAPI server designated as the local SYMAPI. The remote SYMAPIserver is a different situation.

Typically, three site topologies have three SYMAPI servers, one foreach respective site. The SYMAPI server that is to be designated asthe recovery site SYMAPI server can therefore either be the SYMAPIserver located at the synchronous site or the asynchronous site. ForStar, the correct one to be indicated in the array managers depends onif the users intends for the synchronous site or the asynchronous siteto be the location of the DR site. Wherever the recovery VMwareinfrastructure resides dictates which SYMAPI server is to be used.

The SRDF SRA only supports non-Star Cascaded or non-StarConcurrent failover to the asynchronous site. Consequently, unlikeStar, there is no decision to be made as to which SYMAPI server touse. The SYMAPI server that is always to be designated as therecovery SYMAPI server must always be the one local to theasynchronous site.

For Star and non-Star alike, if there is a tertiary SYMAPI server thereis no need (and frankly no mechanism) to configure it within SRM.The SRA does not require access to the third SYMAPI server.


58


Installing/configuring EMC VSI Symmetrix SRA UtilitiesWith the emergence of VMware's vSphere as a true computingplatform, users are spending increasingly more time withinVMware's vSphere Client. As such, users no longer want outsidemanagement applications and would prefer to remain inside ofvSphere Client and complete all of their tasks through a “single paneof glass”. The EMC Virtual Storage Integrator for VMware vSphereleverages the existing vSphere Client plug-in framework, to helpunify multiple EMC plug-ins to vSphere Client by presenting them as“features.” One such feature dedicated to the SRDF SRA is the EMCVSI Symmetrix SRA Utilities.

IMPORTANT

For Windows 2008 and later, if User Account Control (UAC) isenabled, users must run the vSphere Client as an administrator byright-clicking the vSphere Client icon and choosing the “Run asAdministrator” option. Otherwise, the Symmetrix SRA Utilitiesmay fail to edit and save SRA options files.

The Symmetrix SRA Utilities includes five separate functions withinthe Virtual Storage Integrator version 5.4. Discussion on how to useeach of these features is discussed in-depth in the relevant chapters.These features are:

◆ Automated consistency groups creation—In order to enableproper discovery of devices and assist with test recovery andrecovery configuration, VSI includes the ability to easily andautomatically create and configure Symmetrix device consistencygroups.

◆ Assisted troubleshooting—VSI assists with troubleshootingcommon configuration issues in the SRDF and VMwareenvironment that prevent the creation and/or operation ofrecovery plans.

◆ Test failover configuration—This offers the ability to help userswith the task of defining device pairs to be used by EMC SRDFAdapter when testing recovery plans with EMC TimeFinder®

technology.



◆ Gold copy configuration—This offers the ability to help userswith the task of defining device pairs to be used by EMC SRDFAdapter when creating gold copies during failover to a recoverysite.

◆ SRA global options editor—This offers the ability to edit theglobal options for the SRA without having to manually edit anXML file to alter SRA behavior.

The SRA Utilities must be installed locally on the SRM server as itrequires the SRA executable and access to SRM to function properly.Furthermore, the SRA Utilities should be installed on both SRMservers to enable the consistency group creation feature andbi-directional support for the other features. Once installed, the SRAUtilities plug-in is automatically enabled and ready for use.

Note: The 5.4 version of the SRA Utilities is required in order to leverage thenew features of the 5.1 version of the SRDF SRA and is the only version ofVSI that is supported with SRM 5.1.

vCenter permissionsFor a user to be able to successfully use the EMC VSI SymmetrixUtilities their user account must have at least certain permissionswithin vCenter. Depending on what tasks the user wants to do thepermissions required are slightly different. The permissions that arelisted are the minimum amount of control you can assign to a user’saccount that will allow them to use the SRA Utilities.

For Global Options editing, Test Failover Configuration or GoldCopy Configuration and NOT the ability to create consistencygroups, the following roles/permissions must be configured:

◆ An administrator must create a custom role and assign it to theuser on the vCenter object that has these permissions:

• Extension

– Register extension– Unregister extension– Update extension

• Tasks

– Create Tasks– Update Tasks

Installing/configuring EMC VSI Symmetrix SRA Utilities 59

60


◆ The user also needs the "Read Only" role associated with theiraccount on the Cluster and ESXi host objects.

◆ No other permissions are required so all remaining sub-objectscan have the "No Access" role associated with the user’s account.

The permissions for Consistency group creation are slightly differentand more importantly these permissions must be configured onBOTH the local and remote vCenter servers in the SRM environment.For consistency group configuration and NOT the other previouslydiscussed functionality, the following roles/permissions must beconfigured:

◆ An administrator must create a custom role and assign it to theuser on the vCenter object that has these permissions:

• Extension

– Register extension– Unregister extension– Update extension

• Tasks

– Create Tasks– Update Tasks

• Sessions

– Validate session◆ The user needs the "Read Only" role associated with their

account on all ESXi host objects.

◆ Unlike the Test Failover permissions, the user does not need“Read Only” access to the Cluster objects. “No Access” on theCluster objects will be suitable if the user only needs to createconsistency groups and does not need to perform the otherfunctions.

◆ No other permissions are required so all remaining sub-objectscan have the "No Access" role associated with the user’s account.

If the user wants to be able to use all features of the EMC VSISymmetrix SRA Utilities with one account they need to apply all ofthe Test Failover/Global Options permissions listed above inaddition to the “Sessions - Validate Sessions” to the vCenter objectson both the local and remote vCenter servers in the SRMenvironment.



For instructions on creating roles and assigning permissions withinvCenter, please refer to VMware documentation on vmware.com.

VSI configurationBefore any of the EMC VSI Symmetrix SRA Utilities features can beused, the SYMAPI connection must be configured within VSI. Toconfigure this connection, login locally to the SRM server andnavigate within the vSphere Client to the “Home” screen. At thispoint, locate the VSI icon in the “Solutions and Applications” listtowards the bottom of the screen as seen in Figure 19.

Figure 19 EMC VSI configuration location

As can be seen from Figure 20, select the “Solutions Enabler” linkunder the Features > Storage Viewer pane. In the pane that appears,enter in the server name or IP address of the configured SYMAPI


62


server. The configured SYMAPI server should be the same one that isused for the local SYMAPI server in the local array manager withinSRM.

If the SYMAPI server is local to the SRM server do not enter ahostname or IP address and leave the box blank. A blank boxindicates to VSI that it is to use local API calls to perform deviceinquiries.

Figure 20 EMC VSI SRA Utilities Solutions Enabler (SYMAPI) configuration

It is important to note that this configuration parameter is shared byall users of the local install of vSphere Client. If another user changesit, all users will be affected. Consequently users should verify thissetting before each use.

If the consistency group creation feature is going to be used it isrequired that the SYMAPI server be the same one that is configured inthe array manager. This is due to the fact that the group that is createdmust be located on the SYMAPI server shared by the SRA itself1. Thisrequirement is elucidated in Figure 21. The consistency groupcreation feature (discussed in detail later in this chapter) also requiresthat the SYMAPI server be configured within VSI on the remote SRMserver as well. This means that the SRA Utilities must be installedand configured on both SRM servers before consistency groups canbe created. This requirement is unique to this feature of the SRA

1. Unless Group Naming Service (GNS) is enabled and properly configured.GNS will propagate the groups to the other SYMAPI servers.



Utilities. Likewise, the SYMAPI server configured within VSI on theremote SRM server must be the same SYMAPI server configured foruse for the remote SRDF SRA.

Config

Figure 21 Configuring the SYMAPI server for the SRA Utilities

VSI loggingVSI for vSphere Client provides a feature logging function forconfiguring, viewing and exporting logs for all VSI features. Thelogging feature can be located (only after successful installation of atleast one VSI feature) within the vSphere client under Home >Solutions and Applications > EMC. To view the logs, select Loggingfrom the Settings > General tree menu on the left of the screen todisplay logging configuration options as shown in Figure 22.

Figure 22 VSI logging feature


64


The General Logging display provides the following options, asshown in Figure 23. Each option is discussed further following thefigure.

Figure 23 VSI logging options

◆ Select a log from the Log Files drop down box for the desired VSIfeature. Only logs of locally installed features will appear.

◆ All logs can be set to one of several different log levels:

• Off—No log entries will be written.

• Fatal—Only fatal errors will be written.

• Error—Only error messages will be written.

• Warn—Warning and error messages will be written.

• Info—All information including error and warning messageswill be written. This is the default value.

• Debug—Informative messages as well as errors and warningswill be written for any general operations that occur. This isthe recommended setting for troubleshooting issues with theSRA Utilities. When an error or issue is encountered, set thelogging level to debug for the SRA Utilities. This is achieved



by selecting EMC.VSI.VSphere5.Features.SPO.SRAUtilities.txtin the “Log Files” drop down and then select “Debug” in the“Log Levels” drop down. Repeat the process that encounteredthe previous error or issue and revisit the SRA Utilities log.Revert the log level to Info when the issue has been resolved tominimize the file size of the log.

◆ Select the auto-load icon to enable/disable auto-load. Whenenabled a log file will be monitored and loaded into the viewer asit is modified.

◆ Select the scroll lock icon to enable/disable scroll lock. Whenenabled, the scroll lock feature causes the the log viewer to notautomatically scroll to the bottom of the log file whenever it isappended.

◆ Select the zip icon to export all of the logs in a zip file.

◆ Select the folder icon to open the directory that contains the logfiles. The directory is at: %APPDATA%\EMC\Virtual StorageIntegrator\vSphere4\Logs1

◆ Select the search icon to search the text of the logs with findfeature.

The logs for the SRA Utilities are spread out over three different filesdepending on the type of interaction:

1. EMC.VSI.VSphere4.Features.SPO.SRAUtilities.txt—This logcontains normal logging information for standard operations runby the user such as what operations were executed and whatwere the end results.

2. SEWrapper.-yyyymmdd.log—This log contains detailed SYMAPIinteraction and describes the decision making and individualcalls made by the SRA Utilities.

3. EmcSrdfSraUtilities-yyyymmdd.txt—This log details out theintricacies of the consistency group feature that are not loggedelsewhere. This log is not located in the log viewing feature ofVSI. It can be found by manually navigating the file system to%APPDATA%\EMC\Virtual StorageIntegrator\vSphere4\Logs\sraUtilities. Alternatively this can be

1. VSI logging does not include the logs for SRM or for the EMC SRDF SRA.For SRM log locations, refer to VMware documentation. The EMC SRDFSRA logs are located in the SRM log directory under “SRAs” in the “EMCSymmetrix” folder.


66


quickly located by selecting the folder icon described in thesecond to last bullet above and opening the folder calledsraUtilities.



Configuring SRDF SRA behaviorBy default the EMC SRDF SRA is configured to act a certain wayduring discovery, Failover or Test Failover operations. The defaultoptions are configured in accordance to recommended best practicesor to accommodate the most typical environments and, in general, donot need to be changed. Nevertheless, there are occasions when thebehavior of the SRDF SRA must to be altered. Previously, alteringthese options required manual edits to an XML file calledEmcSrdfSraGlobalOptions.xml as seen in Figure 24.

Figure 24 EMC SRDF SRA global options file

With the release of the EMC VSI Symmetrix SRA Utilities for vSphere,this file can now be easily edited from the vSphere Client. The globaloptions file offers the following options shown in Table 3. In-depthdiscussions on each option will be reserved for the relevant respectivesections in this book.

Configuring SRDF SRA behavior 67

68


Table 3 Global options for the EMC SRDF SRA 5.1

Option name Default valueDescription of SRA behavior when the option isenabled

Does the optionsetting need to beidentically configuredfor both SRAs?

SYMAPI debug Disabled The target SYMAPI server logs extremely detailedinformation about SYMAPI interactions from the SRA.Should only be enabled at the direction of EMC Support.

No

Test failover force Disabled Allows Test Failover operations to continue when the RDFPair State is “Split” or “Suspended”.

No

Terminate copysessions

Disabled The SRA will terminate TimeFinder/Snap or Clone replicasafter a Test Failover operation.

No

Test Failover withoutlocal snapshots

Disabled Allows Test Failover operations to be run off of the R2devices instead of creating a TimeFinder replica.

No

Failover if gold copyfails

Enabled If local or remote gold copy creation fails during failover theFailover operation will continue regardless.

No

Ignore activatedsnapshots

Disabled Allows the user to activate TimeFinder/Clone, Snap or VPSnap devices at a a scheduled time instead of having theSRA perform the activate during Test Failover. This optionalso allows the user to manually split the RDF link in casethe option TestFailoverWithoutLocalSnapshots is enabledas well.

No

Filter non-VMwaredevicesa

Enabled Filter out RDF devices that are not presented to theVMware environment during discovery.

Yes

Check for virtualdisksb

Disabled Adds an additional filter to Test Failover and Gold Copyoperations to ensure input replica devices are not in use inthe recovery environment as an RDM or a VMFS volume.

No

Failover to Async site Disabled Allows for the discovery and failover of Asynchronous targetdevices of SRDF/Star configurations.

Yes

SetReplicaTargetToReady

Disabled Enables the SRDF SRA to support replication targets thatbeen made “not ready” by an admin

No

a.Further configuration of the local Solutions Enabler installs is required for filtering and is dis-cussed on page 81b.Further configuration of the local Solutions Enabler installs is required for validating devicereplicas and is discussed in the chapter on test recovery and in the chapter on gold copy creation.



In order to make these changes in VSI, navigate to the home screenwithin the vSphere Client and select VSI in the Solutions andApplications panel as shown in Figure 25.

Figure 25 VSI home screen

To view the global options configuration screen, select “SRDF SRAGlobal Options” in the “Features” panel on the left under the “SRAUtilities” heading. The Global Options configuration screen can beseen in Figure 26.


70


Figure 26 Global options configuration with VSI Symmetrix SRA Utilities

Before making any changes, select the “Refresh” link on the top of thepanel to make sure that VSI is displaying the latest savedconfiguration. The panel lists the option name, the default value, thecurrent value and the description of the option. In order to change thecurrent value select the down arrow and change it to “Enabled” or“Disabled”. If the value selected differs from the current saved value,the value will be highlighted in orange indicating that there exists anunsaved change. The orange will persist until the “Save” link isselected or the user navigates to a new screen. If the user wishes torevert to defaults1, select the “Restore Defaults” link and then select“Save”. Once “Save” is selected the global options file is updatedwith the altered settings. This process is shown in Figure 27.

Editing the global options with VSI only alters the SRA options on thelocal SRM server for the locally-installed SRA. So if the change inbehavior is desired for both SRM servers and both SRA’s then theuser must log into each SRM server and change the options for each.

1. FailoverToAsyncSite is ignored by the “reset to defaults” operation andwill remain at its currently configured setting.



Figure 27 Altering global options with VSI Symmetrix SRA Utilities

SRDF SRA Logging levelsThe support for setting log levels in the SRDF SRA was far lessflexible prior to the 5.1 release of the SRA. The log level could be setvia one of the following ways:

1. Setting the logging level of SRM by modifying thevmware-dr.xml file in the SRM config directory

2. Setting the value (of debug) in the SRA Global options filemanually or through VSI. Only two values were supported - 0and 1

Since the SRA didn't actually support the many different log levels ofSRM almost all information was always logged. This is no longer thecase as logging granularity has been enhanced in the 5.1 release of theSRDF SRA. The SRA now implements the five different levels oflogging supported by SRM. It should be noted that the legacy debug1

option from SRDF SRA global options file is no longer supported andhas been removed in the 5.1 release.

1. The SYMAPI debug option however still persists and is not affected bythe logging levels mentioned here. The enabling/disabling of SYMAPIdebug must still be set in the global options file.


72


The five logging levels are:

1. ERROR—Only Errors

2. WARNING—Only Errors and Warnings

3. INFO—Used for brevity of the logs and therefore logs only majoractions done during any step

4. VERBOSE—Detailed, granular log for each action taken

5. TRIVIA—Extended verbose for support purposes and containsextra information which can prove helpful for debugging issues

ERROR is the lowest level and TRIVIA is the highest. When a loglevel is set to a higher level - for example when it is set to VERBOSE,then log messages having log level ERROR, WARNING, INFO andVERBOSE will be printed.

The logging level can be set by modifying the vmware-dr.xml in theSRM config directory. This file cannot be edited from the SRM GUIand has to be manually edited via a standard text or XML editor. Thelocation of the file is <install directory>\VMware\VMware SiteRecovery Manager\config\vmware-dr.xml.

The vmware-dr.xml file contains important configurationinformation for Site Recovery Manager. It is advisable to onlychange the logging level under the direction of VMware or EMCSupport. When doing so, edit with extreme care and always make abackup of the file before any edits are made and committed.

By default, the SRA logging level is set to verbose. To change thelogging level add the exact following lines to the vmware-dr.xml file:

<level id="Storage">

<logName>Storage</logName>

<logLevel>trivia</logLevel>

</level>

<level id="SraCommand">

<logName>SraCommand</logName>

<logLevel>triva</logLevel>

</level>



The “Storage” log level controls the early query commands, likequeryInfo and queryStrings. The SraCommand log level controlseverything else, mostly the more relevant SRA operations, fromdiscoverArrays to reverseReplication. Enter the desired logging levelfor each in the <loglevel> tags as marked in red boxes in Figure 28and save the file. While it is recommended to keep both of these loglevels the same to provide consistent logging information they can beset differently if needed.

Figure 28 Altering the SRM/SRA logging level

Each time a log level is changed, the VMware SRM service has to berestarted for the changes to be reflected.

Restarting the SRM service will halt any currently executing SRMoperations and prevent other operations from starting until theservice is fully back online. Restart the service with care andcalculation to ensure that no operations will be disturbed by theprocess.

The SRM logs are entitled vmware-dr-xx.log(.gz) and are located at:

%PROGRAMDATA%\VMware\VMware vCenter Site RecoveryManager\Logs\

The SRDF SRA logs are entitled EmcSrdfSra_yyyymmdd.log and arelocated at:

%PROGRAMDATA%\VMware\VMware vCenter Site RecoveryManager\Logs\SRAs\EMC Symmetrix\


74


The SRDF SRA log is newly created each day and the file name isdistinguished by the appending of the current date in the formatshown in the log name. If a clean log is desired, the current log can berenamed to a distinct name and the SRA will automatically create anew log file at the outset of its next operation.



Discovering and configuring Symmetrix replicated devicesOnce the SRDF SRA, VSI Symmetrix SRA Utilities and SRM ArrayManagers are configured, SRDF-replicated devices can be discoveredand managed. This section describes the discovery and configurationprocess that allows these devices to be eventually included in SRMProtection groups and Recovery Plans.

Device discoveryVMware vCenter Site Recovery Manager automatically invokes theEMC SRDF Adapter to perform a discovery of the replicated LUNsbetween the relevant storage arrays as soon as a new array pair hasbeen enabled. A successful completion of the process results in thedetailed discovery of the replicated environment including the serialnumber of Symmetrix storage arrays at the protected site and therecovery site, and the number of replicated devices on the protectedsite Symmetrix storage array. For each device pair the target andsource Symmetrix device number is displayed as well as the directionand status of the replication. If the device pair hosts a VMFS volumethe volume name will appear.

Furthermore, if the devices are a part of a device group or compositegroup on BOTH the local and remote SYMAPI server the name of thegroup will be listed. If only the R1 devices are grouped or only the R2devices are grouped the devices will not be discovered as validcandidates and errors such as below will be displayed in the SRDFSRA log:

[WARNING]: Skipping this source device as the peerdevice is contained within a remote group [<remotegroup name>]

Note: The local (RDF1) and remote (RDF2) groups for a given set of devicesdo not need to be named identically to be accepted as valid group pairs bythe SRDF SRA.

The device discovery tab is shown in Figure 29. When new replicateddevices are added, the “Refresh” link should be selected within the“Devices” tab—this will run a discover LUNs operation on the arraymanagers on the local and remote SRM server.

Discovering and configuring Symmetrix replicated devices 75

76


Figure 29 Discovered Devices tab

If a device does not meet the SRDF SRA requirements it will not beshown in the discovered devices list. Devices must meet thefollowing requirements to be discovered as candidate devices by theSRA1:

◆ Devices must be on enabled Symmetrix arrays

◆ Must be replicated using SRDF

1. These requirements may change with different releases of the SRA. Therequirements listed are for the 5.1 release of the SRDF SRA adapter. Pleaserefer to the release notes of the installed adapter to check for differingrequirements.



◆ Desired R1 devices must be masked to the protected sideVMware environment only if the advanced optionFilterNonVmwareDevices is enabled and configured. If thisoption is disabled the SRDF SRA will still discover them. Ingeneral though the R1 devices should always be masked as theycannot be added to a protection group is the devices are notmasked to the protection environment. For more information onthe advanced option FilterNonVmwareDevices refer to thesection, “FilterNonVmwareDevices” on page 81

◆ Desired R2 (or R21 depending on the configuration) devicesshould be masked to the recovery side VMware environment1.

◆ Uses one of the following replication modes:

• SRDF/Asynchronous

• SRDF/Synchronous

• Concurrent SRDF/Star

• Cascaded SRDF/Star

• Cascaded SRDF (non-Star)

• Concurrent SRDF (non-Star)

◆ Dynamic RDF devices

◆ Adaptive Copy is NOT supported with two-site SRDF

The SRDF SRA version 5.1 does not filter out devices according toRDF Pair state so all states are valid: Synchronized, Consistent, Split,Partitioned, TransmitIdle, SyncInProg, Suspended, etc.

Furthermore, devices from three-site SRDF solutions can bediscovered with the following rules:

◆ SRDF EDP (Star or non-Star) known as Diskless Cascaded orExtended Distance Protection is not supported

◆ The SRDF/Star workload devices are presented to the protectedsite

◆ The SRDF/Star synchronous OR asynchronous devices arepresented to the recovery site (the advanced optionFailoverToAsyncSite must be set accordingly)

1. Refer to the section “Recovery site device masking requirements” onpage 111 for more details on recovery site target device maskingrequirements.


78


◆ Non-Star Cascaded or Concurrent are only supported if theworkload Symmetrix is at the protected site and the async targetSymmetrix is at the recovery site. Failover is only supported tothe async site for non-Star Cascaded or Concurrent SRDF

◆ Non-Star Cascaded or Concurrent cannot have both replicationlegs configured for synchronous or asynchronous replication. InCascaded SRDF, the first leg must be synchronous and the secondmust be asynchronous. For Concurrent SRDF, the order does notmatter but it cannot have the same replication mode on both legs.

IMPORTANT

During a discovery operation, if the SRDF SRA encounters SRDFStar environments with sites in a “disconnected” state, the SRAwill attempt to connect and protect these sites as well as enableStar. Sites in an “isolated”, “halted”, “tripped”, “pathfail”, or“connected” states will be ignored. Discovery operations thatexecute these Star protection workflows will experience longerthan average discovery durations.

EMC requires that all devices are in device groups or compositegroups with consistency enabled (if consistency is applicable) prior tocreating protection groups. When devices are not grouped, eachreplicated device is presented in its own datastore group since SRMand the SRA do not know of device dependencies and applicationrelationships beyond that of a virtual machine spanning multipledevices. This can lead to issues when creating protection groups andrecovery plans as required devices may not be properly grouped.Furthermore, when devices are not grouped, consistency cannot bemaintained in multi-RA group situations. Also, the R1 devices mustbe in a RDF1 group on the local SYMAPI and the R2 devices must bein a RDF2 group on the remote side. If a group is configured withouta remote respective pair group, the devices will be filtered out of thediscovered replicated device list.

Note: With SRDF/A, all devices in the RA group must be failed over inunison. Failover of a subset of devices in a RA group is not allowed.Therefore, if there are SRDF/A devices in the same RA group as VMwaredevices that are unrelated that are not to be failed over with them, it isstrongly recommended to put those devices in a separate RA group.



In order to help users create the correct device groups, EMC stronglyencourages using the EMC VSI Symmetrix SRA Utilities to create thegroups. Use of this tool is discussed later in this chapter.

The SRDF SRA presents the devices to SRM in a standard format suchas the example below:

<SourceDevice id="014C" state="read-write"><Name>014C</Name><TargetDevice key="0181" /><Identity>

<Wwn>60:00:09:70:00:01:92:60:36:03:53:30:30:31:34:43</Wwn>

</Identity></SourceDevice>

SRM takes the WWNs of the devices and attempts to match themwith devices that contain a VMFS volume or are in use as an RDM.SRM examines virtual machine/device correlations and consistencygroupings to create the datastore groups presented in SRM foraddition to a protection group.

Device discovery advanced optionsThe 5.1 version of the SRDF SRA offers two new advanced optionsthat control the behavior of device discovery:

1. FailoverToAsyncSite—allows discovery and failover to theAsynchronous target of a SRDF/Star configuration.DEFAULT=DISABLED

2. FilterNonVmwareDevices—Filters out non-VMware devicesduring device discovery. DEFAULT=ENABLED

FailoverToAsyncSiteBefore the 5.1 version of the SRDF SRA, three site failover withSRDF/Star configurations could only be executed between theworkload site and the Synchronous target site. The SRDF SRAversion 5.1 now allows discovery and failover to the Asynchronoussite of SRDF/Star environment through the use of theFailoverToAsyncSite global advanced option.

By default, this option is disabled and when it is, the SRDF SRA willfilter Asynchronous devices from discovery. When this option isenabled, the SRDF SRA will permit the discovery of theAsynchronous target site devices and conversely exclude anySynchronous target site devices. Furthermore, in order to discover


80


Asynchronous devices (or Synchronous devices for that matter) theFailoverToAsyncSite option must be set identically for both theprotection and recovery site SRDF SRAs.

It is important to note that this option is ONLY for SRDF/Starconfigurations. For non-Star Cascaded or Concurrent SRDF the SRAonly allows failover to their Asynchronous devices and will thereforebe the behavior regardless of this setting.

DO NOT change the FailoverToAsyncSite setting after protectiongroups and recovery plans have been configured. Doing so willinvalidate all protection groups containing SRDF/Star devices uponthe next device discovery operation as shown in Figure 30. This willprevent all failover or test failover operations until the setting hasbeen restored and the devices rescanned.

Figure 30 FailoverToAsyncSite option error



FilterNonVmwareDevicesIn versions prior to the SRDF SRA version 5.1 the number ofreplicated devices reported by the EMC SRDF Adapter in the SRMGUI was not necessarily equal to the replicated devices presented tothe VMware virtualization platform. The discover LUNs process didnot filter out devices according to masking information, so if thedevice met the requirements of the SRA, regardless of whether or notit was part of the VMware environment, it would appear in the list.This did not mean that the SRA or SRM could put them in aprotection group and control these devices. The SRA presented all ofthe valid replicated devices that it had discovered to SRM and thenSRM was responsible for deciding if it is a valid device from theVMware perspective (a RDM or a VMFS). If the device did not satisfyeither of these two requirements it could not and would not be acandidate for a protection group.

To simplify management, discovery behavior has been enhanced andnow reduces the number of returned devices. The behavior of theSRDF SRA external RDF device discovery can now be controlledthrough the use of the global option, FilterNonVmwareDevices. Thisoption is enabled by default in the 5.1 version of the SRDF SRA. Thiswill cause the SRA to filter out devices not part of the VMwareenvironment and they will no longer be listed in the discovereddevices list in the SRM GUI and will not be exposed to SRM in anyway.

This option can be enabled/disabled using the VSI Symmetrix SRAUtilities 5.4. As shown in Figure 31, VSI reminds the user to configurethe SYMAPI authorizations discussed next.

Figure 31 Enabling FilterNonVmwareDevices in VSI Symmetrix SRA Utilities


82


Note: When disabling or enabling the FilterNonVmwareDevices option, itmust be disabled or enabled consistently for both SRAs. If these are setdifferently it could cause the discovery operation to encounter errors or fail.

While it is true that this option is enabled by default, there is,however, further configuration required to provide the SRDF SRAaccess to the information it needs to filter out non-VMware devices.The SRDF SRA does not examine masking information on theSymmetrix to verify which devices are presented to the VMwareenvironment. Instead it connects its local vCenter andqueries/compares the devices in vCenter inventory with thereplicated devices that the SRDF SRA has discovered. Devices thatare not present are then filtered out from the discovery list andtherefore not output to SRM.

In order to enable this behavior, an administrator must grant vCenteraccess for each Solutions Enabler install local to the SRM servers totheir respective vCenter server. It is important to note that this mustbe done on the local installations of Solutions Enabler on the SRMservers and not the remote SYMAPI server. This is regardless ofwhether or not a remote SYMAPI server is configured for use bySRDF SRA control operations.

For example:

In this environment, the FQDN of protection site vCenter server isch-vcenter3.ebc.emc.local. The FQDN of the recovery site vCenterserver is ch-vcenter4.ebc.emc.local. Both vCenter servers can beaccessed via a domain account “hostec”.

Note: The user account(s) must have at least Read Only permissions(assigned to the user account within vCenter) to the vCenter server objectand to the ESXi1 server objects that host (or will host in the case of therecovery side) the virtual machines using the SRDF devices to be managed bySRM.

Authorizations are to be added using a Solutions Enabler CLIcommand shown below (bolded portions are unique and are to bechanged for each given environment):

1. The configured account does not need login access to the ESX(i)servers—it only needs vCenter permissions for the ESX(i) object.



symcfg add authorization -host <FQDN of vCenterserver1> -username <username2> -password <password>-vmware -namespace “vmware/vc”

In order to run Solutions Enabler commands from the SRM serverWindows Command Prompt, the user must either be issuingcommands from the C:\Program Files (x86)\EMC\SYMCLI\bin\directory or have the local Windows “Path” variable configured withthis location. Furthermore, the environmental variable“SYMCLI_CONNECT_TYPE” on the SRM server must either be setto LOCAL or unset. This will ensure that the authorizationcommands are issued to the local installation of Solutions Enablerand not to a remote SYMAPI host.

An administrator must log in the protected site SRM server and runthe following command:

symcfg add authorization -hostch-vcenter3.ebc.emc.local -username hostec-password password3 -vmware -namespace “vmware/vc”

An authorization must also be added for the recovery site SRDF SRA.An administrator must log in to the recovery site SRM server and runthe following command:

symcfg add authorization -hostch-vcenter4.ebc.emc.local -username hostec-password password -vmware -namespace “vmware/vc”

Once these authorizations are added, no further configuration isrequired and the protected and recovery SRDF SRAs will use theseauthorizations to query the vCenter servers to filter devices fromdiscovery.

1. The IP address of the vCenter server can be used in the case of a lack of aconfigured/available DNS server. Alternatively, the local hosts file on theWindows server can be configured with IP address/FQDN translations inplace of a DNS server. Refer to Windows server documentation forinformation on editing the Windows hosts file.

2. The user account does not have to be the account running the SRMservice. It can be any local or domain account with sufficient access tovCenter.

3. Optionally the password can be omitted and once the command has beenexecuted a prompt will appear asking for a password in which the inputpassword will be masked from view.


84


If non-VMware devices continue to show up there is usually an issuewith one or both authorizations. If the SRDF SRA is unable toconnect, the SRA will log a series of messages in the EmcSrdfSra logas shown below:

[WARNING]: Failed to connect to Virtual Center.Please check if you have provided vCentercredentials in the Symmetrix Authorization DB.

[WARNING]: SRA will continue without filtering NONVMware devices.

For further information on the error users can consult the currentviclient log created by the SRDF SRA. This log is located on the SRMserver in the below location:

C:\Program Files\EMC\SYMAPI\log

The logs are entitled viclient-YYYYMMDD.log. So if the log wascreated by the SRDF SRA on September 10th, 2012 the log would beentitled viclient-20120910.log.

Common scenarios for errors can be seen in Table 4.

Table 4 Troubleshooting vCenter authorizations

Message logged in viclient.log Common reason

SymPwdDbEntryList2() Failed: Noobjects of the selected type were found

The -vmware option was missed in thecommand

RetrieveCredentials Failed: No recordswere found in the authorization database.

The -namespace option is missing orincorrect

ViClient::Connect: SOAP 1.1 fault:"":ServerFaultCode [no subcode]"Cannot complete login due to anincorrect user name or password."

Incorrect user name or password



Consistency protection with EMC VSI Symmetrix SRA UtilitiesA typical SRDF configuration contains applications writing data onmultiple Symmetrix LUNs at the protected (R1) site which arereplicated to a remote (R2) sites over SRDF/A or SRDF/S links. Datareplication is a complex process that very often requires preservingthe order of writes on these LUNs. A typical example is a databaseapplication writing data and transaction logs across multiple LUNs.It is very critical to preserve the order of writes in such cases. In SRDFenvironments, this can be achieved by using a feature known asconsistency groups.

Similarly, in a VMware environment, virtual machines are runningapplications that are writing data to one or more Symmetrix LUNsand it is important that these virtual machines are able to correctlyrestart on the recovery site in case of protected site failure. To achievethis, EMC strongly recommends having consistency protection acrossall of the related Symmetrix LUNs being used. VSI includes a featurethat allows users to easily create the correct LUN groupings andenable consistency.

This section will discuss what consistency groups are, the algorithmsVSI uses to create these consistency groups and how VSI assists in thetroubleshooting of common problems involved with creatingconsistency groups for VMware environments.

ViClient::Connect: SOAP 1.1 fault:SOAP-ENV:Client [no subcode]"Host not found"Detail: get host by name failed intcp_connect()

Incorrect host name or the host name isnot configured in DNS or the local hostsfile

ViClient::Connect: SOAP 1.1 fault:"":ServerFaultCode [no subcode]"Permission to perform this operation wasdenied."

The account supplied is valid but does nothave permissions to access the vCenterserver

No error listed in viclient log but certainexpected replicated devices are missingfrom device discovery list

User account entered has access tocertain resources in vCenter but does nothave access to the correct clusters




86


Consistency groupsA consistency group is a device or composite group comprised ofSymmetrix RDF devices, which has been enabled for remoteconsistency. The devices in the consistency group are speciallyconfigured to act in unison to maintain the integrity of data whendistributed across multiple devices within an array. RDF consistencyprotection software preserves the dependent-write consistency ofdevices within the group by monitoring data propagation fromsource devices to their corresponding target devices. If a source R1device in the consistency group cannot propagate data to itscorresponding R2 device, RDF consistency software suspends datapropagation from all the R1 devices in the group. This allows you toquickly recover from certain types of failures or physical disasters byretaining a consistent environment.

RDF consistency group protection is available for both synchronousmode (SRDF/S) and asynchronous mode (SRDF/A). RDFconsistency protection for SRDF/S devices is provided through RDFEnginuity Consistency Assist (RDF-ECA). RDF consistencyprotection for SRDF/A devices is provided through Multi SessionConsistency (MSC)1. Consistency is managed and maintained by theRDF Daemon running on the SYMAPI servers.

Prior to VSI, the creation of consistency groups was a manual process.The storage admin needed to identify the virtual machines beingprotected, identify the datastores that were holding the virtualmachines and determine the list of Symmetrix devices that werebeing used by these datastores (or RDMs if applicable). Once thedevices were determined, a composite group was to be created intowhich the devices identified were added, followed by enabling RDFconsistency on the group. This entire process had to be repeated ifthere was a slight change in the device configuration. Now with VSI,consistency group creation can be completed in an assisted andautomated fashion allowing for extremely simplified setup forfailover and failback.

1. For consistency groups using MSC/ECA to provide consistency, the RDFDaemon on both the protected and recovery SYMAPI server must berunning. It is highly suggested that the RDF Daemon be set to autostart onboth sides in case the SYMAPI server is rebooted.



Depending on the configuration and combination of the devices to begrouped, either a device group or a composite group must be created.When using VSI to create the groups, VSI will intelligently decide onthe appropriate type of group to create on the SYMAPI servers. Notethat even though VSI assigns default names of CG1 or CG2 to thegroups (which can be renamed as the user wishes) CG does not standfor (c)omposite (g)roup, but for (c)onsistency (g)roup. A consistencygroup can refer to both a device group or a composite group. Thedifferences between the two types of groups are described below:

Device group: A device group is a user-defined group comprised ofdevices that can belong to a locally-attached Symmetrix. A controloperation can be performed on the group as a whole, or on theindividual device pairs that comprise it. RDF devices contained in adevice group are limited to one RA group.

Device groups containing SRDF/S devices that do not span multipleRA groups do not need consistency to be enabled. While notrequired, device groups containing SRDF/A devices that do not spanmultiple RA groups are recommended to have consistency enabled.SRDF/A device groups can have consistency enabled through thecommand symrdf enable and this operation does not require theuse of the RDF daemon.

Composite group: A composite group1 (CG) is a user-defined groupwhose members can be individual devices or device groups spanningRA groups.

Composite groups, and especially composite groups that have SRDFdevices that span multiple RA groups must have consistency enabledand this is achieved via the command symcg enable with theconsistency provided through MSC or ECA via the RDF daemon.

VSI consistency group creation is only supported with two-siteSRDF/A and SRDF/S devices. Initial SRDF/Star configurationincludes the process to create the proper consistency groups on allSYMAPI servers and therefore is neither supported or needed withinVSI2. Consistency group creation for non-Star Concurrent or

1. Manually created composite groups must ALWAYS be created with the-rdf_consistency flag regardless of device configuration. It also musteither be of type RDF1 or RDF2.

2. If a Star composite group is deleted, the symstar Solutions Enablercommand offers a simple mechanism to completely recreate the groupwith the “symstar buildcg” command.


88


Cascaded SRDF is discussed in the section, “Creating consistencygroups for Concurrent and Cascaded SRDF” on page 100 of thischapter.

Redundant consistency protectionEMC strongly recommends running redundant SRDF daemons on atleast two control hosts at each site. This ensures at least one SRDFdaemon is available to perform time-critical, consistency monitoringoperations.

For redundant consistency protection of composite groups,environments must simultaneously run two instances of the SRDFdaemon on separate control hosts. Each control host must have acommon view of the composite group being monitored, which can beaccomplished by doing one of the following:

1. Running the GNS daemon on each control hosts

2. Manually defining the composite group on all control hosts.

SRDF daemons running simultaneously on two different hostsperform independent monitoring and switching operations. If onefails, the other SRDF daemon takes it place1, completing all pendingtasks, such as committing the last cycle of data to the target site. Byrunning redundant SRDF daemons, you avoid service interruptionscaused by performance bottlenecks local to a control host, and linkfailures of the redundant SRDF daemons and the control hosts.

The storrdfd daemon runs on each host for which SRDF consistencyis required. If the Group Naming Services (GNS)2 daemon is enabled,storrdfd relies on GNS to propagate updated CG definitions to allhosts locally attached to the same set of Symmetrix arrays. If GNS isnot enabled, you must manually recreate the updated CG definitionon each one of these hosts. Administrators can enable GNS on eachhost by installing and starting the daemon with the followingcommands:

stordaemon install storgnsd -autostart

stordaemon start storgnsd

1. If the primary SYMAPI server fails and cannot be recovered, the arraymanagers in SRM will need to be reconfigured with the backup SYMAPIserver.

2. The EMC Solutions Enabler Symmetrix Array Management CLI Product Guideexplains GNS in detail.



And by using the following SYMAPI options file setting:

SYMAPI_USE_GNS=ENABLE

As shown in Figure 32, Host-1 and Host-2 contain sets of the basedaemon, the SRDF daemon, and the GNS daemon to ensure dataconsistency protection.

Figure 32 Running redundant hosts with GNS to ensure consistency protection

VSI consistency group creation algorithmIn order to perform failback or failover, VSI must figure out whatdevices belong in what consistency groups. To accomplish this, VSIfollows certain algorithms to decide what devices are related andwhat devices are not. Depending on the SRDF replication type(SRDF/A or SRDF/S) the algorithm is slightly different. Thisdifference is important to note because due to SRDF/A restrictions,unintended or non-VMware replicated devices may be included inthe consistency groups.

The SRDF/S algorithm is as follows:

◆ VSI starts by enumerating the list of replicated datastores present


90


on the ESX server/cluster.◆ For each datastore, the underlying Symmetrix devices are

determined.◆ The set of virtual machines (VMs) is generated for that datastore.◆ If any VM spans multiple datastores, all of the underlying

Symmetrix devices are noted (Steps 1-4 are repeated for each ofthose datastores).

◆ If any of the VMs have RDM mappings with additionalSymmetrix devices, these devices are also noted.

◆ The devices in the set from steps 2, 4, 5 are all considered to bepart of one consistency group.

◆ If the included devices span multiple RA groups, VSI will create acomposite group. If there is only one involved RA group a devicegroup will be created instead.

◆ Similarly, consistency groups are calculated for all of theremaining datastores.

Figure 33 shows an example configuration that will be used in thenext part to understand how VSI creates consistency groups.

Note: The sole purpose of this example is to show how the VSI algorithm forconsistency group creation works. The example environment, therefore, may notfollow recommended Symmetrix/VMware best practices.



Figure 33 Virtual machine storage mapping example

Using the algorithm for SRDF/S the configuration shown in Figure33 would result in the following (assuming all devices are SRDF/S):

◆ VSI finds three datastores “VMFS 1” (one extent), “VMFS 2” (twoextents1) and “VMFS 3” (one extent).

◆ “VMFS 1” is on Symmetrix device A1. At this point only deviceA1 is in the consistency group, which we will call CG1.


92


◆ VM1 and VM2 have virtual disks that reside on VMFS 1.Additionally, VM1 has an RDM association with Symmetrixdevice G1. As a result, G1 will be added to the list.

◆ VM2 has a virtual disk on “VMFS 2”. Therefore, the Symmetrixdevices contained in “VMFS 2” will also be added to theconsistency group.

◆ The consistency group CG1 will contain Symmetrix devices A1,C1, D1 and G1.

◆ Datastore “VMFS 3” contains Symmetrix device E1. There is oneVM, VM3, which uses “VMFS 3”. It does not span multipledatastores and does not have any RDMs.

◆ VSI creates a second consistency group CG2 with device E1.

◆ VSI presents the two consistency groups to the user as possibleoptions. The user can select all or some of the groups forconsistency group creation.

◆ Based on user selection, VSI creates the desired consistencygroups, adds the corresponding devices to the groups, andenables RDF consistency.

The above algorithm for SRDF/S also works well for SRDF/A.However, there is an additional restriction that must be taken intoaccount. SRDF/A operations cannot be performed on a subset ofdevices contained in a single RA group. This means, all of the deviceswithin a RA group must be part of any SRDF/A operation.

With that important caveat in mind, here is how the algorithm wouldexecute if the environment in Figure 33 was configured withSRDF/A:

◆ VSI finds three datastores “VMFS 1”, “VMFS 2” and “VMFS 3”.

◆ “VMFS 1” is on Symmetrix device A1.

◆ VM1 and VM2 have virtual disks that reside on “VMFS 1”.

◆ VM1 has a RDM to Symmetrix device G1. As a result, G1 will beadded to the list. Due to the extra restriction on SRDF/A devices,the Symmetrix device F1 will be added to CG1 as well.

1. Multi-extent datastores are not recommended and not supported withSRM by EMC prior to SRM 5.1. To enable support in SRM 5.1, theadvanced SRM option, AutoDetachLUNsWithDuplicateVolumes, must beenabled.



◆ Again, due to the extra restriction on SRDF/A devices, theSymmetrix devices B1 and C1 will also be part of the sameconsistency group CG1 because they are all part of the same RAgroup RA1. Even though B1 is not a Raw Device Mapping or aVMFS volume, it will be included.

◆ VM2 also has a virtual disk on “VMFS 2”. Therefore, theSymmetrix devices contained in “VMFS 2” will also be added tothe consistency group. Step number 5 already identified thisdevice so no changes are made.

◆ The Symmetrix device E1 is part of the RA group RA2. Due to theSRDF/A restriction, it will be part of the same consistency groupCG1. As a result, the Symmetrix device E1 will be added toconsistency group CG1.

◆ At this point, all discovered Symmetrix devices are all going to bepart of the consistency group CG1.

◆ Therefore VSI presents one consistency group CG1 that willcontain Symmetrix devices A1, B1, C1, D1, E1, F1 and G1.

◆ Based on user selection, VSI creates the desired consistencygroups, adds the corresponding devices to the groups, andenables RDF consistency.

Creating consistency groups with VSIAs discussed, the ability to create consistency groups is built into VSI.This allows automated creation of consistency groups from directlywithin the vSphere Client without manual configuration on the array.

VSI allows the creation of consistency groups for both test failoverand failover operations. To ensure proper configuration of Symmetrixdevice consistency for native failover operations (especially SRDF/A)the use of VSI is strongly encouraged. The use of array managementtools is also acceptable, but VSI is preferred as it streamlines andsimplifies the process for VMware environments. In this section, theprocess for creating consistency groups for failover will be outlinedas an example of the creation process.

To create consistency groups in preparation for discovery followthese steps:


94


1. Connect to the Protected vCenter and navigate to the EMC VSItab as shown in Figure 34.

Figure 34 Accessing the consistency group creation panel in VSI

2. The Remote Site Connection wizard, shown in Figure 35, ismodeled after the VMware vCenter Site Recovery Managerremote site connection wizard. The connection wizard does notpop-up automatically when the Failover Configuration panel isselected. To initiate this wizard the “Login” link must be selectedas shown in Figure 34. This wizard allows VSI to connect to andauthenticate with the remote vCenter server and the remotevCenter Site Recovery Manager server. If they are installed on thesame server only the vCenter server needs to be specified,otherwise both server names or IP addresses must be entered. Inthe latter case, the credentials provided for the vCenter servermust also be authorized for the vCenter Site Recovery Managerserver. In the example in the figure, they are running on the sameserver so only the vCenter server name is required. Once theconnection has been made, the “Login” link disappears and it isreplaced with a “Refresh” link to refresh the view.



Figure 35 VSI Remote Site connection wizard

3. For security reasons, the remote vCenter/SRM server connectioninformation is not persistent over vSphere Client log-ins/log-offs.Therefore, each time the Remote Site information must be enteredand authenticated. Once the connection wizard is complete, theFailover Configuration panel will populate with suggestions forcandidate consistency groups (CGs).


96


IMPORTANT

Every time VSI lists consistency groups they are suggestionswithout regard to currently configured groups on the SYMAPIserver. VSI does not take into account any groups previously madeby VSI or other array management tools. Previous groups will notshow up in VSI as VSI does not query the SYMAPI databases forpreviously configured groups. To view currently configured groupsuse Symmetrix management applications like Solutions EnablerCLI or Unisphere for VMAX.

4. Not all of the suggested consistency groups actually have to becreated, only the groups that are actually selected will be created.Furthermore, suggested CGs can be merged into one larger CGbefore creation. In this example, shown in Figure 36 on page 97,VSI suggests the creation of four separate consistency groupsnamed CG0, CG1 and CG2 and CG3. CG0, CG2 and CG3 arevirtual machines protected by SRDF/S replication and CG1contains virtual machines entirely protected by SRDF/A. For thisenvironment all three SRDF/S groups will be merged into one.The SRDF/A group will remain as originally suggested. Figure 36on page 97 shows these CGs being merged into one group.

It is never a requirement to merge consistency groups; thedecision to merge or not to merge is the user’s choice anddepends on the logical relationships between the protectedvirtual machines. VSI will never require this because anymandatory virtual machine groupings have already beenenforced. The consistency groups in this example are beingmerged just to exemplify the concept.

It is also important to note that VSI never prevents mergingconsistency groups in memory within VSI. Two CGs can belogically merged in memory irrespective of the any issues presenton the various components within each of the CGs being merged.However, checks are run on the merged CG and thencorresponding flags are set based on the devices contained in themerged CG. Some of these flags may prevent actual creation, butwill not prevent in-memory merges. For more information onthese flags refer to “Assisted troubleshooting of SRDFconfigurations with VSI” on page 107.



Figure 36 Merging consistency groups with VSI

5. Once the desired consistency group configuration has beenconfigured, the groups can be right-clicked and renamed toconform with an organization’s naming conventions or to simplysupply more meaningful names than CGx. The process to renamethe groups can be seen in Figure 37.


98


Figure 37 Renaming consistency groups

6. At this point, as seen in Figure 38, the desired consistency groupscan be checked and created by selecting the “Create CheckedCGs” link. VSI will commit these groups on both the local andremote SYMAPI servers with the group name specified by theuser. If a device is already present in one or more consistencygroups on the target SYMAPI server, those consistency groupswill be deleted and the device(s) will be added to the newconsistency group. After successful completion of the task, theadvanced details of the groups can viewed using Unisphere orthe Solutions Enabler command line interface. One of thenewly-created groups, the RDF1 group, can be seen in Unispherein Figure 39.



Figure 38 Creating consistency groups


100


Figure 39 Verifying consistency groups with Unisphere for VMAX

IMPORTANT

After successful creation of a consistency group pair with VSI, thenewly created group, its associated virtual machines and replicateddevices will no longer appear in the VSI consistency groupwindow. Only when the “Refresh” link is selected will VSI rescanthe environment and the virtual machines/devices will re-appear.

Creating consistency groups for Concurrent and Cascaded SRDFThe current release of the VSI does not support the automatedcreation of consistency groups for non-Star Concurrent or CascadedSRDF. Therefore the groups must be created with Symmetrixmanagement tools such as Symmetrix Management Console1 orSolutions Enabler CLI.

The following subsections respectively outline the steps that shouldbe observed when creating consistency groups and activatingconsistency for Concurrent or Cascaded SRDF.

1. Unisphere for VMAX does not yet support composite groups soSymmetrix Management Console or SYMCLI must be used



Concurrent SRDF consistency groups1. Identify devices. Identify the Concurrent devices that are in use

in the VMware environment through the use of the VSI StorageViewer feature.

2. Create RDF1 group. Create a RDF consistency-enabled RDF1composite group with the Symmetrix Management Console orSolutions Enabler CLI on the protection side SYMAPI server andadd the devices. Examples of both methods are shown in Figure40 and Figure 41 respectively.

Figure 40 Creating composite group for Concurrent RDF devices with SMC


102


Figure 41 Creating composite group for Concurrent RDF devices with SE

3. Create RDF2 group. After creating the RDF1 group, the RDF2group must be created. Using one of the same mechanisms asdiscussed above, create an RDF consistency-enabled RDF2composite group on the recovery side SYMAPI server and addthe devices. The recovery side must be the Asynchronous site ofthe Concurrent SRDF setup.

4. Enable consistency for both legs of the Concurrent setup. TheSRDF SRA does not necessarily require the Synchronous leg to beconsistent, but it is a best practice to do so.

Since the SRDF SRA requires the Concurrent SRDF topology tohave its two mirrors of the R1 devices operate in different RDFmodes, SRDF consistency protection cannot be enabled at thecomposite group level as with other configurations. Instead,consistency must be individually enabled for each RDF grouprepresenting the device mirrors by its group name. EnablingSRDF consistency at the group name causes the RDF daemon tomonitor the SRDF groups separately so that if a concurrent R1device is unable to propagate its data to one of its remote R2partners, the daemon suspends the SRDF links for only the grouprepresenting that R2 mirror.

The RDF group of the leg operating in asynchronous mode isenabled with MSC consistency protection and the RDF group ofthe leg operating in synchronous mode is enabled with RDF-ECAprotection.

RDF group-level consistency can only be enabled with SolutionsEnabler CLI. The ability to do so is not yet included in Unispherefor VMAX or the Symmetrix Management Console. Thefollowing syntax should be used to enable consistency:



Use the symcg command to define a name to represent an SRDFgroup:

symcg -cg <CG name> set -name <name> -rdfg <SymmetrixSN>:<RDF Group #>

Then use the symcg command to enable consistency protection forthat SRDF group:

symcg -cg <CG name> enable -rdfg name:<name>

This must be executed twice, once for each leg of the ConcurrentSRDF setup. An example of doing so with Solutions Enabler CLI isdisplayed in Figure 42. The Synchronous target is enabled firstfollowed by the Asynchronous target. The Symmetrix serial numberin the example is 1253 and the Synchronous RDF Group number is 16and the Asynchronous RDF Group number is 16.

Figure 42 Enabling consistency for a Concurrent SRDF setup with SolutionsEnabler

Cascaded SRDF consistency groups1. Identify devices. Identify the Cascaded devices that are in use in

the VMware environment through the use of the Virtual StorageIntegrator Storage Viewer feature.


104


2. Create RDF1 group. Create a RDF consistency-enabled RDF1composite group with the Symmetrix Management Console orSolutions Enabler CLI on the protection side SYMAPI server andadd the devices. Examples of both methods are shown in Figure43 and Figure 44 respectively.

Figure 43 Creating composite group for Cascaded RDF devices with SMC



Figure 44 Creating composite group for Cascaded RDF devices with SE

3. Create RDF2 group. After creating the RDF1 group, the RDF2group must be created. Using one of the same managementapplications discussed above, create an RDF consistency-enabledRDF2 composite group on the recovery side SYMAPI server andadd the devices. The recovery side must be the Asynchronous siteof the Cascaded SRDF setup.

4. Enable consistency for both legs of the Cascaded setup. UnlikeConcurrent SRDF, the SRDF SRA does in fact require theSynchronous leg, in addition to the Asynchronous leg, to beconsistent.

The RDF group of the leg operating in asynchronous mode isenabled with MSC consistency protection and the RDF group ofthe leg operating in synchronous mode is enabled with RDF-ECAprotection. Also, unlike Concurrent SRDF, users can enable SRDFconsistency protection for cascaded devices using a compositegroup or an SRDF group name. It should also be noted that onemethod or the other should be chosen but not both. For example,users cannot enable the first hop using a composite group andthen enable the second hop using its SRDF group name.

Composite group-level consistency can be enabled with SolutionsEnabler CLI or the Symmetrix Management Console. Thefollowing Solutions Enabler CLI syntax is to be used to enableconsistency:

Use the symcg command to enable consistency protection for acomposite group Synchronous leg:

symcg -cg <CG name> enable


106


Use the symcg command to enable consistency protection for acomposite group Asynchronous leg:

symcg -cg <CG name> enable -hop2

Examples of enabling consistency with the SymmetrixManagement Console and Solutions Enabler CLI are respectivelydisplayed in Figure 45 and Figure 46.

Figure 45 Enabling consistency for a Cascaded SRDF setup with SMC



Figure 46 Enabling consistency for a Cascaded SRDF setup with SE

Assisted troubleshooting of SRDF configurations with VSIPrior to VSI, identifying misconfiguration of SRDF for VMwarevCenter Site Recovery Manager could be a difficult task dependingon the size, scope and complexity of the implementation. VSIprovides a mechanism through the consistency group creationprocess that allows the user to quickly and easily identify theseproblems. VSI identifies each device, datastore, and virtual machinepresent in the consistency group and if any errors are discovered oneor more of the flags are set and a corresponding alert is shown in theVSI consistency group panel. There are four types of flags: deviceflags, datastore flags, virtual machine flags, and consistency groupflags. These flags are as follows:.


108


Device flagsTable 5 shows the flags that can be set depending on the deviceconfiguration. Each device present in the consistency group isanalyzed and one or more of the flags below are set to true.

Table 5 Device flags (page 1 of 2)

Flag Name Description

NON_VM_DEVICE This flag is set on a device when a device becomespart of the CG due to the presence of an SRDF/Adevice and not all of the devices present in the RAgroup are contained in the corresponding datastore.CG creation will be successful when any of theseflags are set.

NON_RDF_DEVICE This flag is set on a NON RDF Symmetrix device.This device will be displayed in the CG but will beignored during CG creation.

NON_SYMM_DEVICE This flag is set on a NON Symmetrix device. Thisdevice will be displayed in the CG but will beignored during CG creation.

NOT_VISIBLE_DEVICE This flag is set on a Symmetrix device that is notvisible to the ESX host. This device will be displayedin the CG but CG creation is not allowed due to thepresence of such devices.

REMOTE_DEVICE_NOT_VISIBLE This flag is set on a valid RDF Symmetrix devicethat has a remote device which is not visible to theremote ESX host. This device will be displayed inthe CG but CG creation is not allowed due to thepresence of such devices.

REMOTE_DEVICE_NOT_FOUND This flag is set on a valid RDF Symmetrix devicethat has a remote device attached but couldn't bequeried for properties. CG creation is not alloweddue to the presence of such devices.

NON_VM_NON_VISIBLE_DEVICE This flag is set on a device when a device becomespart of the CG due to the presence of an SRDF/Adevice and not all of the devices present in the RAgroup are contained in the corresponding datastoreand if such devices are not visible to thecorresponding ESX host. CG creation is not alloweddue to the presence of such devices.



Datastore flagsTable 6 shows the flags that are set depending on the devices presentin the datastore. Each datastore attached to the consistency group isanalyzed and one or more of the flags below are set.

RDF_TYPE_NOT_SUPPORTED This flag is set on RDF devices which are neitherR1 nor R2. This device will be displayed in the CGbut will be ignored during CG creation.

RDF_MODE_NOT_SUPPORTED This flag is set on RDF devices which are neitherAsync nor Sync. This device will be displayed in theCG but will be ignored during CG creation.

MULTI_SITE_NOT_SUPPORTED This flag is set on RDF devices which areconfigured in three site mode. Even though the SRAsupports three sites, VSI does not support creatinggroups for three site devices.

Table 5 Device flags (page 2 of 2)


Table 6 Datastore flags


MIXED_RDF_MODE_DEVICES This flag is set on a datastore that has mixed modeRDF devices (SRDF/A and SRDF/S devices). Ifsuch devices are present in a datastore, thedatastore and the corresponding CG will be badgedwith an alert icon within the VSI Plug-In UI. CGcreation is not allowed when datastores containsuch devices.

DEVICES_FROM_MULTIPLE_SYMMS

This flag is set on a datastore that has RDF devicesfrom multiple Symmetrix arrays. If such devices arepresent in a datastore, the datastore and thecorresponding CG will be badged with an alert iconwithin the VSI Plug-In UI. CG creation is not allowedwhen datastores contain such devices.


110


Virtual machine flags

Table 7 shows the flags that are set depending on the devices presentin the VM. Each VM attached to the consistency group is analyzedand one or more of the flags below are set.

Consistency group flags

Table 8 shows the flags that are set depending on the devices presentin the Consistency Group. Each consistency group is analyzed andone or more of the flags below are set.

Table 7 Virtual Machine flags


MIXED_RDF_MODE_DEVICES This flag is set on a VM that has mixed mode RDFdevices (SRDF/A and SRDF/S devices). If suchdevices are present in a VM, the VM and thecorresponding CG will be badged with an alert iconwithin the VSI Plug-In UI. CG creation is not allowedwhen VMs contain such devices.


This flag is set on a VM that has RDF devices frommultiple Symmetrix arrays. If such devices arepresent in a VM, the VM and the corresponding CGwill be badged with an alert icon within the VSIPlug-In UI. CG creation is not allowed when VMscontain such devices.

Table 8 Consistency Group flags


MIXED_RDF_MODE_DEVICES This flag is set on a CG that has mixed mode RDFdevices (SRDF/A and SRDF/S devices). If suchdevices are present in a CG, the CG will be badgedwith an alert icon within the VSI Plug-In UI. CGcreation is not allowed when it contains suchdevices.


This flag is set on a CG that has RDF devices frommultiple Symmetrix arrays. If such devices arepresent in a CG, the CG will be badged with an alerticon within the VSI Plug-In UI. CG creation is notallowed when it contains such devices.



When the CG is first displayed, VSI marks any offending objects witha general warning icon. If the object is far down in the CG hierarchyall of the superior objects to the offending object are marked with thewarning sign to signal that there is an issue. For instance if a devicecontaining a VMFS volume has a problem, the consistency group ismarked with a warning label, the virtual machine is marked, theVMFS volume is marked and finally the actual offending device ismarked. The specific error caused by one of these flags being set canbe seen by simply mousing over the device with the warning.Figure 47 shows an example of a device used as a Raw DeviceMapping that has the NON_RDF_DEVICE flag set.

Figure 47 Troubleshooting SRDF configuration with VSI

Recovery site device masking requirementsSome users prefer to not mask their recovery site target devices (R2 orR21 devices as the case may be) to the recovery hosts until the time ofrecovery as they prefer to do so only when necessary1. Regardless,EMC still highly recommends to always have the recovery devicesmasked to the recovery site ESXi servers due to the fact that many

1. While the recovery and protected devices should be masked, with threesite SRDF configurations (Star or non-Star) the SRDF SRA never requiresthat the bunker devices be masked to any compute resources.


112


VSI and SRDF SRA operations will not function if the recoverydevices are not present. Nevertheless, there are a few operations thatdo not have this requirement. The following operations will functionif the recovery devices are not masked at the time of execution:

◆ Test failover with TimeFinder

◆ Device discovery if the global option FilterNonVmwareDevices isdisabled

The following operations will FAIL if the recovery devices are notmasked and presented to the recovery hosts:

◆ Recovery (disaster recovery or planned migration)

◆ Test Failover with the option TestFailoverWithoutLocalSnapshotsenabled

◆ Device discovery if the global option FilterNonVmwareDevices isenabled

◆ VSI Test Failover replica configuration

◆ VSI Gold Copy replica configuration

◆ VSI Consistency group creation feature

If users opt to leave their recovery target devices unmasked it isstrongly advised to put scripts in their recovery plans that willautomatically mask the respective R2 devices to the recovery hostsprior to the SRM recovery step “Change Recovery Site Storage toWriteable”.



Configuring VMware vCenter SRM protection groupsThe basic unit of VMware vCenter Site Recovery Manager failoverprocedure is called a “datastore group”. A datastore group is a logicalgrouping of devices and is the smallest unit of storage that can befailed over or tested independently. There are several rules thatcontrol how datastore groups are calculated:

◆ If a replicated device is used by a datastore containing virtualmachines, all devices used by that datastore are combined.

◆ If a device is a part of a consistency group, all devices in theconsistency group are combined.

◆ If a virtual machine spans multiple datastores, all devicesbelonging to all such datastores are combined.

Protection groups can include one or more datastore groups and arethe building block of recovery plans (which are discussed in the nextsection). Protection groups include the datastores groups which are tobe failed over simultaneously. As such, failover is absolute—eitherthe entire protection group fails over or none of it.

Protection groups can be created by selecting on the ProtectionGroups button on the left-hand side menu and then right-clicking onthe “All Protection Groups” folder and choosing “Create ProtectionGroup” as displayed in Figure 48. This process can be initiated fromeither SRM server.

Configuring VMware vCenter SRM protection groups 113

114


Figure 48 Creating protection groups with VMware vCenter SRM

At this point the “Create Protection Group” wizard will appear. Thewizard consists solely of choosing the target SRM server and whatdatastore group(s) will be in the protection group and a name for thegroup. The wizard is shown in Figure 49.

It should be noted that while the process can be initiated from eitherSRM server, datastore groups will only appear for the SRM serverthat has access to the R1 devices in that group. So if no datastoregroups appear in the create protection group wizard it is likely thatthe incorrect SRM server was previously selected in the wizard.



Figure 49 Protection group creation wizard

Configuring VMware vCenter SRM protection groups 115

116


Configuring VMware vCenter SRM recovery plansVMware vCenter Site Recovery Manager (SRM) provides extensivefunctionality to configure flexible disaster recovery plans. Detailedinformation on the creation and configuration of these plans arebeyond the scope of this TechBook. Readers should consult VMwaredocumentation available on the VMware website for further details.However, to explain the functionality provided by EMC SRDFAdapter for VMware vCenter Site Recovery Manager during thetesting and execution of the recovery plans, sample protection groupsand recovery plans were created for the environments used in thisTechBook.

In this case, one recovery plan, “SRDF-Star-Cascaded-RP” wascreated. This plan utilizes the corresponding protection groupSRDF-Star-Cascaded. The recovery plan “SRDF-Star-Cascaded-RP” isshown in Figure 50.

Figure 50 Display of the recovery plan “SRDF-Star-Cascaded-RP”



Configuring advanced SRM optionsVMware SRM offers a wide variety of advanced options and settingsthat can be altered by the SRM administrator. The vast majority ofthese options are well beyond the scope of this book but this sectionwill briefly discuss a couple of key settings involved in storageconfiguration and behavior of SRM.

If these settings are changed from the default it is recommended tomake the change on both the protected and recovery SRM servers tomaintain configuration consistency between the two environments.Changes of an option on one SRM server does not propagate to theother SRM server therefore the onus is on the user to do so.

Below are descriptions and recommendations for a list of six of themost common storage-related advanced SRM settings:

1. storage.commandTimeout—This option controls the amount oftime SRM allows the SRA to execute an operation. In small tomedium environments the default of 300 seconds is sufficient butin larger or more complex environments (especially large Starenvironments) or in situations where the SRA must wait for datato synchronize, increasing this setting may be advisable. It isrecommended to not adjust this setting unless it is deemednecessary. This option affects both recovery and test recoveryoperations.

2. storage.storagePingInterval—This option controls the timeinterval between SRM-initiated automatic SRA rediscovery of thestorage arrays and replication information. The default setting isto rediscover every 24 hours (configured in seconds in SRM). Ingeneral, it is not recommended to change this setting as it isimportant that SRM and the SRA update regularly to make surethat the environment is in a valid state. However, there are certaintimes that a user might need to temporarily reconfigure storage toan unsupported (by the SRDF SRA) state such as Adaptive Copy.If a rediscover occurs during this state any protection groupscontaining those devices will be invalidated until they arereverted to a supported state. In these cases it maybeadvantageous to increase the interval to a time longer than theexpected temporary reconfiguration. It is important to revert theinterval back to its original time of 24 hours when completed.

Configuring advanced SRM options 117

118


3. storageProvider.fixRecoveredDatastoreNames—Duringrecovery or test recovery, a prefix is added to datastore nameswhen the ESXi servers re-signature and mount recovereddatastores. This option is a check-box setting that, when enabled,will remove the snap-xxxxxx prefixes that are automaticallyadded to recovered datastore names. SRM will remove theseprefixes after mounting the volume but before registering anyvirtual machines. By default this setting is disabled anddatastores will keep the prefixes unless manually removed. Ingeneral it is recommended to enable this option to provide forcases such as user-generated scripts that may run againstdatastores and are sensitive to their exact names. This optionaffects both recovery and test recovery operations.

4. storageProvider.hostRescanRepeatCount— (SRM 5.0.1 andlater) This option controls the number of times per host that arescan is issued during a recovery operation. It is only in rareoccasions (usually very large environments) that this settingshould be increased from the default of 1. While there is nopractical upper limit to this setting, it should not be necessary toever increase the value to more than 2. This option affects bothrecovery and test recovery operations.

5. storageProvider.autoResignatureMode—(SRM 5.1 only) Thisoption controls the process of resignaturing during recovery ofdatastores. Prior to SRM 5.1, SRM set the host-wide optionLVM/EnableResignature to enabled which would blanketresignature all unresolved snapshots. This behavior has beenchanged and enhanced in SRM 5.1 to create far more granular andintelligent control. This option affects both recovery and testrecovery operations. There are three different modes for this newoption:

• Disable—New behavior in SRM 5.1 which will use a selectiveresignature API to query and resignature only the snapshotVMFS volumes relevant to SRM. This is the default andrecommended mode. SRM will check before every HBArescan (during every test and failover operation) to make surethat the LVM.EnableResignature option on all ESX hosts isdisabled, and if it is not, SRM will disable it and leave it assuch. This mode is selected by setting the option value to 0.

• Enable—Same behavior as in SRM 5.0.1 and before. Turns onthe LVM/EnableResignature flag in the respective ESX hoststo enable automatic resignature of ALL unresolved VMFS



volumes. SRM will check before every HBA rescan (duringevery test and failover operation) to make sure that theLVM.EnableResignature option on all ESX hosts is enabled,and if it is not, SRM will enable it and leave it as such. Nosnapshot is left out regardless of its relationship to SRM. EMCstrongly discourages the use of this mode for the reason notedin the below warning. This mode is selected by setting theoption value to 1.

• Ignore—New behavior in SRM 5.1 which will not change theLVM/EnableResignature flag on the respective ESX hosts.Whatever the user has pre-configured in ESX forLVM/EnableResignature, SRM will use it. SRM will look ifvolumes are auto-resignatured, if not it will query andselectively resignature them. This mode is selected by settingthe option value to 2.

During a recovery or test recovery with SRM 5.0.1 and earlier or SRM5.1 with the Auto-Resignature Mode set to Enabled, SRMautomatically sets LVM.enableResignature to 1 if the flag is notalready set in order to resignature snapshot volumes and mount themon ESX hosts. The flag is NOT reset to 0. Setting theLVM.enableResignature flag on ESX hosts is a host-wide operationand, if set, all snapshot LUNs that can be resignatured areresignatured during the subsequent host rescan. If snapshot volumesunrelated to SRM are currently forcefully mounted to ESX hosts onthe recovery site, these LUNs are resignatured as part of a hostrescan. Accordingly, all of the virtual machines on these volumeswill become inaccessible and will require re-registration. To preventoutages, ensure that no forcefully-mounted snapshot LUNs arevisible to ESX hosts on the recovery site or set Auto-Resignaturemode to disabled. For more information refer to VMware KB article2010051

6. storageProvider.autoDetachLUNsWithDuplicateVolume—(SRM 5.1 only) When multiple unmounted unresolved snapshotvolumes are found, SRM will use its API to detach theseunwanted and unused snapshot LUNs. SRM will thenresignature the volume(s) that are to be recovered and thenre-attach the previously detached LUNs. This setting is onlyapplicable when autoResignatureMode is either set to disable or


120


ignore and is only applicable on ESXi 5 and later (prior versionsof ESX do not include the detach mechanism). The default for thisoption is enabled.

This option is targeted for environments that have multi-extentdatastores with multiple copies of them presented to the recoveryenvironment. Resignaturing in these situations can be complexand the functionality introduced by enabling this optioneliminates the possibility of errors in regards to the resignaturingprocess. EMC’s recommendation is to leave this option enabled.

It is important to note that SRM will only detach detectedsnapshots of the devices to be recovered in the recovery plan.Unmounted, unresolved devices unrelated to the recovery plan(not in the recovery plan or snapshots of devices in it) will not bedetached/re-attached. This option affects both recovery and testrecovery operations.

Figure 51 shows the advanced options editor within SRM and thelocation of all of the aforementioned options.



Figure 51 SRM advanced options editor


122



3


◆ Introduction ...................................................................................... 124◆ Test failover workflow in VMware vCenter SRM....................... 125◆ EMC VSI Symmetrix SRA Utilities with Test Failover ............... 135◆ Device Write Pacing and TimeFinder-based Test Failover......... 149◆ Testing recovery plans using EMC TimeFinder/Clone.............. 154◆ Testing recovery plans using EMC TimeFinder/Snap ............... 160◆ Testing recovery plans using EMC TimeFinder/VP Snap......... 166◆ Test Failover Advanced Options.................................................... 174◆ Test Failover without TimeFinder.................................................. 183◆ Testing recovery plans using GNS-enabled groups.................... 206◆ Test Failover with non-VMware devices ...................................... 210

Testing Recovery Plans

Testing Recovery Plans 123

124


IntroductionVMware vCenter Site Recovery Manager provides users the ability totest their recovery plans in an isolated environment. Testing recoveryplans allow customers to test their disaster recovery ability withoutimpacting production applications and replication. The testing of therecovery plans is achieved by leveraging local replication technologyprovided by the storage arrays.

The storage adapter provided by the vendor has to support thecapabilities offered by VMware vCenter Site Recovery Manager. TheEMC SRDF Adapter for VMware vCenter Site Recovery Managersupports the testing of recovery plans through the use ofTimeFinder/Clone, TimeFinder/Snap1, or TimeFinder/VP Snaptechnologies. Additionally, the EMC SRDF Adapter offers the abilityto perform a test failover without the use of local replicationtechnologies by running the test on the R2 device itself.

This chapter will discuss the following topics related to test failover:

◆ Test failover SRM workflow

◆ Configuring test failover with the VSI Symmetrix SRA Utilities

◆ Test failover using TimeFinder/Clone

◆ Test failover using TimeFinder/Snap

◆ Test failover using TimeFinder/VP Snap

◆ Test failover Advanced Options

◆ Test failover without TimeFinder

◆ Test failover with GNS-enabled groups

◆ Test failover with non-VMware devices

Note: TimeFinder/Mirror is no longer supported with the 5.1 release of theSRDF SRA. For customers using older versions of the SRDF SRA,TimeFinder/Mirror operations and requirements are discussed in theappendix.

1. VMAX 20K/40K arrays only. TimeFinder/Snap is not supported onVMAX 10K



Test failover workflow in VMware vCenter SRMIn general, the workflow of test failover with VMware vCenter SiteRecovery Manager 5 with the EMC SRDF SRA has not significantlychanged since previous versions. However, the process has been splitup into discrete operations in SRM 5 to allow for more granularcontrol. Test failover is now a two part process:

◆ Test—Initiates test failover, mounts replica devices and powerson virtual machines according to the selected recovery plan.

◆ Cleanup—Powers down and removes test environment fromrecovery site, unmounts and discards replica devices and returnsstatus to original pre-test state.

This section will cover the general workflow of the test failoveroperation regardless of replica mode (i.e., TimeFinder type or R2method). The subsequent sections will deal specifically with theconfiguration and execution of the different test failover modes.

Requirements Before a test failover can be executed the following requirementsmust be met:

◆ A recovery plan must be configured and associated with one ormore protection groups.

◆ Inventory mappings should be configured so virtual machinescan be properly recovered.

◆ A replica method for test failover, either TimeFinder or R2recovery, must be chosen and configured. In the case ofTimeFinder, device pairings (target devices to be replicas of theR2 devices) must be identified. Specific requirements for this andhow to configure these pairings is discussed in detail in theupcoming sections.

◆ If consistency protection is not enabled, the test failover operationwill fail1. For example, if consistency has been disabled by theuser or the RDF link is in “Transmit Idle” state, the adaptercannot perform consistent split on the BCVs or consistent activateon clones and snaps. Refer to the section, “Consistency groups”on page 86 for more information on consistency.

1. Consistency does not need to be enabled on SRDF/S device groupslimited to only one RDF group but it is recommended.

Test failover workflow in VMware vCenter SRM 125

126


◆ Test failover, by default, will fail when the involved RDF links arein the “Suspended” or “Split” state. The SRA global option,“TestFailoverForce” can be enabled to allow test failover in thesesituations.

IMPORTANT

In Cascaded SRDF/Star environments that have been recovered tothe Asynchronous target site, test failover, with or withoutTimeFinder, ("without" referring to enabling the advanced settingTestFailoverWithoutLocalSnapshots) is not supported. Only fullrecovery back to the original workload site is supported with theSRDF SRA.

Test Once all required configurations are complete and the user is ready toperform a Site Recovery Manager test failover operation the usermust select the desired Recovery Plan and click on the Test link asshown in Figure 52.

Figure 52 Initiating a Recovery Plan test failover operation with VMware SRM



Once clicked, a short series of windows pop-up to confirm the testfailover operation and that this is in fact what the user wants to do.There is a checkbox option offered in the first window called“Replicate Recent changes to recovery site” and this option, whatenables/disables the “SyncOnce” operation, is not required norneeded for SRDF operations. This option is a generic SRM operationand may be relevant for other vendors and their respectivereplication technologies. This option can be seen enclosed in a redbox in Figure 53.

Figure 53 Test failover confirmation wizard

SRDF and consistency technology (as well as additional SRArequirements) make sure that an up-to-date and consistent copy isavailable on the R2 side. Users can opt to uncheck this option as itwill instruct SRM to skip this step consequently shortening the testfailover time by a small amount.

At a high level, a recovery plan test involves the following:

◆ Creation of a “bubble” IP network1 for the virtual machines torun in so production networks are not interfered with.

◆ A temporary snapshot of the R2 device is resignatured andmounted at the recovery site.


128


◆ Shadow VMs are unregistered and are replaced with the replicaVMs which are then, by default, powered on.

◆ If the plan requests suspension of local virtual machines at therecovery site, they are suspended during the test.

◆ Finally, it is important to remember that no operations aredisrupted at the protected site and that the replication of theSymmetrix devices containing the VMFS datastores is notaffected or changed1.

1. If a special network is not created beforehand by the user, SRM will createone automatically. The problem with the SRM-created networks is thatthey will not allow virtual machines on different hosts to communicate asthese networks are internal to each host and do not have external uplinks.If inter-host communication is required, the user should create insulatednetworks across ESXi hosts and configure it for use in the recovery plan.

1. With the exception of when the TestFailoverWithoutLocalSnapshotsadvanced option is enabled. More information on this option is foundlater in this chapter.



Once the user has confirmed the test failover operation can proceed,the Recovery Plan will be initiated in test mode. A completed testrecovery can be seen in Figure 54. The test environment will remainoperational until a Cleanup operation has been executed.

Figure 54 Completed Test Recovery Plan

Cleanup Once a recovery plan has been tested the test environment can bediscarded and reset through the use of the “Cleanup” operationoffered by SRM. The Cleanup operation automatically reverts allchanges incurred by the recovery plan test and allows for subsequentfailover operations.

The Cleanup operation performs the following operations:

◆ Power off and unregister test virtual machines

◆ Unmount and detach replica VMFS volumes


130


◆ Replace recovered virtual machines with original placeholders(shadow VMs), preserving their identity and configurationinformation.

◆ Clean up replicated storage snapshots that were used by therecovered virtual machines during the test.

Before resetting the environment after a test failover, ensure that therecovery plan worked as desired. Verify the success of any customscripts, application functionality, networking, etc. Once all facets ofthe test have been verified by the involved stakeholders a Cleanupoperation can be executed.

IMPORTANT

After a test failover has been executed, an actual failover or anothertest failover cannot be run until a Cleanup operation has occured. Itis advisable to run a Cleanup operation as soon as the testenvironment is no longer needed to allow for any subsequentoperations to be run without delay.

As can be seen in Figure 55, a Cleanup can only be run against arecovery plan if the recovery plan status is in “Test Complete”.Furthermore, even if a test failover was not entirely successful, aCleanup operation will still need to be executed before another testfailover can be run. The “Test Complete” status will be assigned tothe Recovery Plan regardless of the level of success reached by thetest failover.

Figure 55 Recovery plan in “Test Complete” status after successful test failover



The Cleanup process is initiated, in a similar fashion to the testfailover process, by clicking on the Cleanup link after selecting theappropriate Recovery Plan. This can be seen in Figure 56.

Figure 56 Initiating a Cleanup after a test recovery

The Cleanup link launches a similar set of windows that the originaltest operation brought up to confirm the reset activities that it willexecute. The first attempt at running this Cleanup after a particularfailover offers no configurable parameters and simply displaysdetails for confirmation. This set of screens are shown in Figure 57.

Figure 57 Cleanup operation confirmation wizard


132


Figure 58 shows the steps taken by the Cleanup process itself.

Figure 58 Cleanup operation steps

Due to a variety of reasons, the first attempt at a Cleanup operationmay fail. Typical causes include:

◆ Test failover did not complete entirely successfully

• Storage replica creation failure

• Virtual machine inventory mappings incorrect

◆ Environment change after the test failover but before Cleanup

• Manual change to storage outside of SRA

• Significant protection group change

• VMware environment failure

• Manual change to VMware environment outside of SRM

In cases such as these, the first Cleanup operation will fail. This is dueto the fact that, on the first run the Cleanup operation does nottolerate any failures with any step of the Cleanup process. Therefore,if the Cleanup process encounters an error it will immediately fail.Once the Cleanup process has failed for the first time the ability toforce the Cleanup becomes available. The Cleanup confirmationwizard, when run subsequent to a failure, will now offer a checkboxto force the Cleanup as seen in Figure 59. This will alter the behavior



of the Cleanup process to ride through any error encountered. Anyoperation it can complete successfully will be completed and, unlikebefore, any operation that encounters an error will be skipped.

Figure 59 Executing a Cleanup operation with the force option

In general, it is not advisable to resort to the force Cleanup unless anactual failover operation needs to be run immediately and the time totroubleshoot any issues encountered in the Cleanup cannot beafforded. Otherwise, before using the force option, attempt to resolveany issues first and then retry a non-forced Cleanup again. If a forceCleanup is used in haste, it may require additional manualintervention afterwards because the SRA and SRM may not be able torecover and ready themselves for another test failover or failoverwithout user intervention.


134


When a force Cleanup is run, as seen in Figure 60, users shouldexecute another test failover as soon as possible to verify theenvironment is functioning correctly.

Figure 60 Force Cleanup operation with failures



EMC VSI Symmetrix SRA Utilities with Test FailoverThe EMC SRDF Adapter requires the definition of device pairs in theoptions file when recovery plans are tested. Although standard texteditors can be used for this task, defining a large number of pairs canbe cumbersome and error-prone. For this reason, EMC highlyrecommends the use of the EMC VSI Symmetrix SRA Utilities to editthe configuration file. Doing so greatly reduces the possibility of errorand drastically simplifies the configuration process, especially inlarge environments.

This section will explain:

◆ The test failover options file

◆ How to use VSI to configure the options file

Test Failover options fileThe TimeFinder device pairings for test failover are stored in anoptions file (EmcSrdfSraTestFailoverConfig.xml) on the recovery site.Even though it is not recommended to manually edit and most userswill never see the file itself, it is helpful to know how the test failoveroptions file works. An example options file is shown in Figure 61.

There are a few different blocks within the XML file:

◆ CopyInfo—Information for each test failover operation.

◆ ArrayID—This option takes the array ID. Users can specifymultiple CopyInfo blocks within TestFailoverInfo with differentarray IDs.

◆ CopyType—This option is used to select the type of replicationtechnology for test failover operation. Possible values are VSE(VP Snap), CLONE and SNAP.

◆ SavePoolName—If the CopyType selected is SNAP, it is requiredto input save pool name.

◆ DeviceList—To perform a test failover operation, it is required toprovide the device pair information. Each device pair representssource and target device pairs. For all copy types, the sourcedevice is the R2 device on the recovery site. For VSE (VP Snap)the target devices must be TDEVs and for SNAP the target

EMC VSI Symmetrix SRA Utilities with Test Failover 135

136


devices must always be VDEVs. CLONE does not have suchdevice type limits so the targets can be a wide variety of devicetypes.

Figure 61 Test failover options file example

There are some important points to remember when configuring theoptions file:

1. There can be multiple <CopyInfo> blocks each with differentarray IDs. This is the most common scenario.

2. Each <CopyInfo> block should contain every device contained inthe corresponding protection group.

3. There can be multiple <CopyInfo> blocks with the same array ID.However, any two <CopyInfo> blocks should not contain thedevices from the same protection group. In such case, the adapterpicks the first <CopyInfo> block defined in the options file.

4. If two different <CopyInfo> blocks have the same array ID and ifa subset of the devices present in a protection group is present inthe first <CopyInfo> block and if the remaining set of devices iscontained in the other <CopyInfo> block, the adapter doesn't



combine them and instead the adapter fails the request becausethe adapter expects all of the devices present in a protectiongroup to be contained in a single <CopyInfo> block.

Now that the details of the options file have been discussed, it isimportant to note that the user should never have to deal with theseintricacies. The EMC VSI Symmetrix SRA Utilities masks anycomplexity by adhering to these rules and enforcing proper behaviorfor the user making configuration of the pairings file extremely easy.

EMC VSI Symmetrix SRA Utilities for vSphere ClientThe EMC VSI Symmetrix SRA Utilities for vSphere Clientautomatically recognizes the presence of the EMC SRDF Adapter andactivates the appropriate extensions required for supporting edits tothe EMC SRDF Adapter pairings file. Figure 62 shows the additionaltab provided by VSI when it is installed on the VMware vCenter SiteRecovery Manager at the recovery site. The tab is activated when acluster object or ESX(i) host object is selected in the left pane in theInventory view. The figure shows the relevant information that isdisplayed when a ESX(i) host is selected. A union of all informationfrom the ESX(i) hosts belonging to a cluster is displayed if the clusterobject is selected instead of an ESX(i) host object. For this reason, it isrecommended to edit the pairings from a cluster object whenapplicable.

Figure 62 VSI functionality for defining test failover device pairings

Figure 62 also shows that the plug-in provides a yellow status box inwhich succinct messages concerning the operation of the plug-in aredisplayed. These status boxes can easily be removed bydouble-clicking on the yellow box. The plug-in also provides a box to


138


select the task for device pairing configuration: either for creation ofgold copies during execution of the recovery plans, or for creation ofTimeFinder pairs used in the testing of recovery plans.

Figure 63 shows an error that can be encountered by users wheninitiating the test failover configuration process in VSI.

Figure 63 Error in VSI test failover pairings screen

The error in Figure 63 usually occurs for the one of the followingreasons:

1. The user is logged in to the wrong SRM server. To configurepairings the user must be operating the SRA Utilities installed onthe site which is presented with the R2/R21 devices (the recoverysite).

2. The recovery site target devices (R2 or R21 as the case may be) arenot masked to the recovery hosts. VSI verifies maskinginformation and the recovery site target devices must be maskedto the recovery hosts. See section “Recovery site device maskingrequirements” on page 111 in chapter 2 for more information onmasking requirements.

3. The SYMAPI server is not configured within VSI. See the section“VSI configuration” on page 61 in chapter 2 for more instructionson doing so.

4. The local and/or remote array(s) have not been discovered by theconfigured SYMAPI server.

5. Symmetrix Authorizations and/or Symmetrix Access ControlLists are preventing VSI from having appropriate access to thearray information. Refer to Chapter 10, “Symmetrix Security withEMC SRDF Adapter,” on page 375 for more information.



6. The SYMAPI server is blocking user access to array informationdue to an enabled, but improperly configured nethost file. Referto the section, “Nethost configuration” on page 42 in chapter 2.

In the case of bidirectional protection, SRDF relationships will befound on both sides. To configure pairings for a given set of devices,use the SRA Utilities on the recovery site for that set of devices.

The EMC VSI Symmetrix SRA Utilities offers two configurationoptions for the test failover pairings screen. The option “Only showreplicas with sessions” offers the ability to have VSI only display testfailover candidate replicas that are currently involved in a replicationsession with the listed R2 device. This option is disabled by defaultbut can be useful in large environments as it vastly increases thespeed of which the adapter displays replica candidates. If a userdesires to configure replica candidates that are not currently involvedin an active TimeFinder replication session with a R2 they shouldleave this option unchecked. The second option called “Star devices”transforms (when enabled) the replica listing columns to show SRDFStar device information which is markedly different than two-siteinformation. These options are shown in Figure 64.

Figure 64 VSI Symmetrix SRA Utilities test failover options

When the “Star Devices” option is unchecked, the configurationscreen will only report two-site SRDF relationships (SRDF/S orSRDF/A). Table 9 shows the columns and their descriptions fortwo-site SRDF:


140


Table 9 VSI Symmetrix SRA Utilities test failover configuration panel (page 1 of2)

Column heading DescriptionConfigurable fromwithin VSI?

Source Array The array ID number of the hosting thelocal source (R1) device. This columnalso displays the status of the SRDFreplication and is denoted by a diamondthat can be either filled green, red, orunfilled. Green indicates that the status ofthe SRDF link for the device pair isnormal whereas an unfilled diamond orred diamond indicates an abnormal statusthat may require the attention of thestorage administrator.

No

Source Device The Symmetrix device number of thelocal source (R1) device

No

Group The SRDF Group number of the remotetarget (R2) device. The SRDF groupdefines the logical relationship betweenSRDF devices and SRDF directors onboth sides of the SRDF links

No

SRDF Mode The mode of the given device pair, eitherSynchronous or Asynchronous

No

SRDF State This current state of the SRDFrelationship (Consistent, SyncInProg,Synchronized, Split etc...). Possiblevalues depend on the mode of the SRDFrelationship.

No

Target Array The array ID number of the hosting theremote target (R2) device

No

Target Type The device type for the remote targetdevice. It will always be some form of anR2 device (R2+TDEV, R2+R-5 etc...).

No

Target Device The Symmetrix device number of theremote target (R2) device

No

Replica Device The Symmetrix device ID of thesuggested replica device for the given R2.This field is a dropdown list offeringpossibly many device options.

Yes



Figure 65 shows the configuration screen for test failover when “StarDevices” remains unchecked.

Figure 65 VSI Symmetrix SRA Utilities test failover configuration for two-site SRDF

When the “Star Devices” option is checked, the configuration screenwill only report SRDF/Star relationships (Concurrent or Cascaded).The VSI Symmetrix SRA Utilities observe the setting of the SRDFSRA advanced option, FailoverToAsyncSite when populating the

Replica Type The device type for the selected replicadevice.

N/A—depends onreplica device

Replica State The TimeFinder session state for theselected replica device and the remotetarget (R2) device if a session exists.


Snap Pool The name of the Snap pool that iscurrently chosen for the selected replica.This column will only appear when theTimeFinder/Snap replication type ischosen. If multiple Snap pools exist, thenthis field will be converted into adrop-down box.

Yes

Table 9 VSI Symmetrix SRA Utilities test failover configuration panel (page 2 of2) (continued)



142


SRDF relationship information into the columns. If the setting isdisabled (meaning recovery is intended for the Synchronous site) thesynchronous site device information will be displayed. Whenenabled the asynchronous site device information will be displayed.Table 9 shows the columns and their descriptions for SRDF/Star:

Table 10 VSI Symmetrix SRA Utilities test failover configuration panel forSRDF/Star (page 1 of 3)



No


No

Star CG Name The name of the composite groupassociated with the Star configuration.

No

Star Mode The mode of the given Star configuration,either Cascaded or Concurrent.

No

Star State This current state of the Starconfiguration. Can either be Protected orUnprotected.

No

Target Array The array ID number of the hosting theremote target device. This will either bethe Synchronous site array or theAsynchronous site array depending onhow the SRDF SRA setting,FailoverToAsyncSite, is configured.

No



Target Type The device type for the remote targetdevice. It will always be some form of anR2 or R21 device (R2+TDEV, R21+R-5etc...). This will either be a type of R21 ora type of R2, depending on how theSRDF SRA setting, FailoverToAsyncSite,is configured and whether or not the Starconfiguration is Concurrent or Cascaded.Concurrent devices will always be R2’sregardless of whether recovery is at theasynchronous or synchronous site.

No

Target Device The Symmetrix device number of theremote target device This will either be aSynchronous site device or aAsynchronous site device depending onhow the SRDF SRA setting,FailoverToAsyncSite, is configured.

No

Replica Device The Symmetrix device ID of thesuggested replica device for the givenR2/R21. This field is a dropdown listoffering possibly many device options.

Yes



Replica State The TimeFinder session state for theselected replica device and the remotetarget (R2/R21) device if a session exists.



Yes

Table 10 VSI Symmetrix SRA Utilities test failover configuration panel forSRDF/Star (page 2 of 3) (continued)



144


Figure 66 shows the configuration screen for test failover when “StarDevices” is checked.

IMPORTANT

The EMC VSI Symmetrix SRA Utilities does not support theconfiguration of test failover pairings for non-StarConcurrent/Cascaded SRDF. For test failover with those SRDFconfigurations, users must edit the test failover pairings XML filemanually.

Workload Site Name The name assigned in the Starconfiguration for the current workloadarray.

No

Target Site Name The name assigned in the Starconfiguration for the target arrayconfigured for recovery. This will either bethe Synchronous site or theAsynchronous site depending on how theSRDF SRA setting, FailoverToAsyncSiteis configured.

No

Target Site State The state of the link to the configuredtarget site. This can be a variety of states,such as Protected, Connected,Disconnected, Halted, Isolated, etc...

No

Table 10 VSI Symmetrix SRA Utilities test failover configuration panel forSRDF/Star (page 3 of 3) (continued)




Figure 66 VSI Symmetrix SRA Utilities test failover configuration for SRDF/Star

Figure 65 and Figure 66 show the detailed information displayed byVSI to simplify the task of defining device pairs. The figures show therelevant information required for configuring test failover pairings.The type of TimeFinder technology (Clone, Snap or VP Snap) can beselected via the drop-down menu located above the pairingconfiguration table. The plug-in automatically filters the SRDF pairsconfigured on the Symmetrix storage array to display only thosepairs that are relevant to the object selected on the left pane of theInventory view.

The potential device pairings displayed by VSI are determined byusing the following logic:


146


◆ If the options file has existing device pairs for the selectedTimeFinder technology, by default, VSI automatically displaysthe pairs that are defined in the options file and suggests theexisting pairing if they are valid.

◆ If the options file does not define any valid device pairs or if theuser chooses a TimeFinder technology that is not configured inthe options file, VSI determines the device possibilities bychecking that they:

• If the mode of replication is asynchronous is the write pacingenabled on the R1 or R21 (depending on where the recoverysite is) if the TimeFinder method selected is Snap or VP Snap.TimeFinder/Clone does not have write pacing requirements.

• Have the same configuration as the R2 (for example,metavolume configuration, size).

• Have the appropriate device type for the TimeFindertechnology in use.

• Are presented to the recovery side--replica devices must bemasked to the recovery side hosts prior to invoking the SRAUtilities. VSI filters the candidate list to display only thosedevices that are visible to the object selected in the left pane ofthe Inventory view. The additional check prevents users fromdefining device pairs that would result in a failed test of arecovery plan.

VSI Symmetrix SRA Utilities is unaware of VMware/Symmetrixdevice relationship status. Therefore, it does not filter out replicadevice candidates containing VMFS volumes or ones that are in useas an RDM. So it is important to note that if a device isinadvertently selected and used in test failover, any data on it will beoverwritten by the TimeFinder copy operation. Consequently, it isimportant to verify using VSI Storage Viewer or through other meansthat the device is not in use by the VMware environment. It is alsoimportant to ensure that is not in use by hosts external to theVMware environment. The SRDF SRA version 5.1, however, doeshave an advanced option “CheckforVirtualDisks” that will filterVMware devices out during test failover. This option can be enabledto provide further protection.



The plug-in uses three ellipses as a candidate for a replica device if itis unable to find a viable candidate for a SRDF pair1. If the type ofTimeFinder technology that is selected does not apply to a SRDF pair,for reasons discussed in the appropriate sections, the replica devicecolumn is populated with “NA”. The plug-in also provides theopportunity to change the save pool if TimeFinder/Snap technologyis selected for defining device pairs.

The recommendation given by the plug-in can be overridden byselecting a different device from the drop-down menu provided inthe Replica Device column. The Replica Device can be changed to“Ignore” if a SRDF pair is not involved in either the test of a recoveryplan or creation of a gold copy during the execution of a recovery.

The information displayed by the plug-in can be refreshed to reflectany changes in the configuration by clicking the Refresh button.Finally, after making the appropriate selections, the user can save thedevice pair definitions to the options file by clicking the SaveOptions File button next to the Refresh button. The plug-in ensureseach device pair definition is unique before saving it to the optionsfile. After saving the file, it is advisable to use the “Export...” link tosave a copy of the configured pairings in case the options file isoverwritten.

IMPORTANT

The options file must be configured properly prior to the “Test”operation and the “Cleanup” operation. The SRA does notmaintain a memory of device pairings used in a “Test” operationfor the eventual “Cleanup” operation. The SRA will reference theoptions file both times for pairing information. If a long time haspassed since the “Test” operation was run and before the“Cleanup” is run it is possible that the options file has beenreconfigured and may no longer include the valid pairs forCleanup. Therefore it is important to verify the pairings are correctimmediately before a “Test” and a “Cleanup” or those operations

1. The plug-in also displays three ellipses if there are insufficient replicadevices for creating device pairings. For example, if there are eight SRDFdevice pairs but only four possible replica devices, the plug-in assigns thefour devices to the first four SRDF device pairs (or if they are already in areplication session they will be associated with the respective R2s). Thereplica device column for the remainder of the SRDF pairs would bedenoted by three ellipses.


148


may fail. It is recommended to use the VSI “Export” functionalityto save a copy of the configured device pairings so they can beeasily and properly reconfigured at a later date.

VMware vCenter Site Recovery Manager version 5 and EMC SRDFAdapter version 5 support recovery plan definitions that includemultiple protection groups that involve more than one Symmetrixstorage array at the recovery site. VSI supports this feature byimplementing two things:

◆ Grouping of the information on the recovery site per targetSymmetrix when displaying device pairings in vSphere client.

◆ Accommodating the changes to the CopyInfo stanza and theaddition of the ArrayID tag in the EmcSrdfSraOptions.xml fileused by the EMC SRDF Adapter.

The change to the visual representation of the data is shown inFigure 67.

Figure 67 VSI plug-in for defining device pairings used by EMC SRDF Adapter



Device Write Pacing and TimeFinder-based Test FailoverThe EMC SRDF Adapter for VMware vCenter Site Recovery Managersupports a variety of EMC TimeFinder modes to test recovery plans.The use of TimeFinder/Snap or VP Snap with an R2 device in anSRDF/A session is only allowed when SRDF/A device-level writepacing is configured on the RA group(s) on the R1/R21 side of theSRDF replication. SRDF/A device-level write pacing extendsSRDF/A group-level write pacing to address conditions that lead tocache overflow. Without some level of throttling, a TimeFinder/Snapor VP Snap copy off of a R2 can possibly cause the SRDF/A session todrop due to the R2 lagging too far behind the R1/R21. This istypically due to certain high-intensity copy-on-write workloads onthe TimeFinder copy. This situation can overwhelm the R2 device andcause cache overflow on the R1/R21 side as writes build up.

Device write pacing was introduced to prevent these scenarios in5875 Enginuity. Therefore, support of a TimeFinder/Snap or VP Snapcopy of an active two-site SRDF/A R2 is limited to VMAXenvironments running Enginuity 5875 and higher.

For Cascaded SRDF or Cascaded SRDF/Star environments, of whichthe Asynchronous target is the recovery site, device write pacingmust be enabled on the R21 devices. Write pacing of an R21 device isonly supported with the Enginuity 5876.159 and higher.

This section describes configuring write pacing in order to useTimeFinder/Snap or VP Snap for test failover with SRDF/A devices.

SRDF/A and device-level write pacingThe SRDF/A write pacing feature helps secure the availability of anSRDF/A session by preventing conditions that cause this cacheoverflow on both the R1 and R2 sides. SRDF/A detects when SRDFI/O service rates are lower than the host I/O rates, and then takescorrective actions to slow down the host I/O rates to match the SRDFI/O service rates. It can work in conjunction with the SRDF/A DeltaSet Extensions and transmit idle functionality. SRDF/A write pacingmonitors and responds to spikes in the host write I/O rates andslowdowns in data transmittal to the R2 and in R2 restore rates. Bymonitoring and pacing the host write I/Os, the SRDF/A write pacingfeature controls the amount of cache used by SRDF/A. This preventscache overflow on both R1 and R2, and helps the SRDF/A session tostay up and running.

Device Write Pacing and TimeFinder-based Test Failover 149

150


If a host application cannot tolerate the I/O throttling caused whendevice level write pacing is triggered, do not enable this feature.Instead use TimeFinder/Clone for test failover operations.

To enable device-level write pacing follow these steps:

1. Identify the RA group(s) in use by the R1 or R21 SRDF/A devices.Write pacing will need to be enabled on every RA group thatcontains the SRDF/A devices that will be used withTimeFinder/Snap or VP Snap.

2. When setting up device-level write pacing, a variety of optionscan be configured to control the behavior. In-depth discussions onhow to best configure these options are beyond the scope of thisbook. Default values are acceptable in most situations. ConsultEMC SRDF documentation or your local EMC SymmetrixPerformance Consultant for further details.

3. Users must also configure the device write pacing autostartoption to “on” to best protect the resiliency of a user’senvironment. Device-level write pacing autostart specifieswhether to automatically enable the device-level write pacingwhen the SRDF/A session is activated. If autostart is not set,write pacing will need to be activated manually any time theSRDF/A link is resumed. Figure 68 shows how to enableautostart and device-level write pacing using Unisphere forVMAX for two-site SRDF or three-site (non-Star) SRDF.



Figure 68 Activating device-level write pacing and enabling Autostart withUnisphere for VMAX

Note: When the Autostart feature is enabled on the R1 or R21 side ofSRDF/A session, the Device-level Pacing feature will automatically beactivated by Enginuity when the SRDF/A session is activated. The SRDF/Asession is activated by Solutions Enabler when a RDF Control to bring thedevices RW on the link is issued (RESUME, ESTABLISH, etc.)

4. Once device write pacing and autostart is enabled on the R1 orR21 RA group, the same must be done for the respective R2 RAGroup. Using the same process as shown in Figure 68 (but insteadof the local array use the remote array), enable autostart anddevice write pacing on the R2 RA group.


152


5. Device write pacing for SRDF/Star configurations cannot yet beconfigured with Unisphere for VMAX. Instead, Solutions EnablerCLI must be used. The following is the syntax for enabling devicewrite pacing:

symrdf -sid <Symmetrix SN> -rdfg <RDF Group #>activate -rdfa_devpace -star

The previous command must be run for the R1 or R21 RA group(depending on whether the configuration is ConcurrentSRDF/Star or Cascaded SRDF/Star) and the R2 RA group.Running it on both will activate device write pacing on both sidesof the SRDF relationship for the given devices/RA group. Nextautostart must be enabled for both sides as well. The followingcommand will enable autostart on both sides of the SRDFrelationship, so it will only need to be run once. It can be executedon either the R1/R21 RA group or the R2 RA group—it does notmatter which.

symrdf -sid <Symmetrix SN> -rdfg <RA Group #> setrdfa_pace -dp_autostart on -both_sides -star

The -both_sides option will enable autostart on both sides ofthe SRDF relationship. If that option is missed, the command willneed to be executed again but with the remote RA groupspecified instead.

Before configuring TimeFinder/Snap or VP Snap pairs for two-site orConcurrent SRDF/A replicated devices with the VSI SRA Utilities, (inaddition to the usual requirements such as proper sizing, etc.) a fewconditions must be met to allow this functionality:

◆ The Enginuity version of both arrays must be 5875 or higher

◆ Device-level write pacing must be activated with autostartenabled on both the R1 (protected site) RA group and R2(recovery site) RA group

Three-site Cascaded replication, where the recovery is at the R2 sitenot the R21 site, requires the following conditions to be met:

◆ The Enginuity version of the R21 array must be 5876.159 orhigher. The R2 array must be running at least 5875.

◆ Device-level write pacing must be activated with autostartenabled on both the R21 (protected site) RA group and R2(recovery site) RA group



If these conditions are not met, the VSI Symmetrix SRA Utilities willnot allow the configuration of TimeFinder/Snap or VP Snap pairingsfor SRDF/A devices. Instead of giving the option to pick a device,“NA” will be displayed in the replica device column and noalterations will be allowed until these conditions have been satisfied.Once these requirements are met, the user can refresh the “testfailover Configuration” screen as shown in Figure 69. Fortroubleshooting purposes, if one or both of first two of the aboveconditions are not met, VSI will log one or more of the followingerrors in the SRA Utilities log file:

◆ “The current array configuration does not support TF/<Snap orVP Snap> with Asynchronous SRDF. The R2 (or R21), <deviceID>, will be skipped.”

◆ “This version of Symmetrix Enginuity does not supportTimeFinder/Snap with SRDF/A target devices, so no replicaswill be listed for those target devices.”

Figure 69 TimeFinder/Snap with SRDF/A in VSI


154


Testing recovery plans using EMC TimeFinder/CloneEMC SRDF Adapter for VMware vCenter Site Recovery Managersupports the use of EMC TimeFinder/Clone technology to testrecovery plans. The following steps should be adhered to whentesting recovery plans with TimeFinder/Clone technology:

1. Ensure the target of the cloning operation is presented (masked)to the appropriate VMware ESX hosts at the recovery site.

2. Using VSI in Figure 70 one can check for the presence of localreplication candidates. For TimeFinder/Clone operations, it isimportant to check that the correct amount of properly sizeddevices are present on the recovery cluster. This can be checkedusing the test failover configuration feature of VSI SymmetrixSRA Utilities as shown in Figure 70.

3. To configure test failover replica pairs for two site SRDF, leave thecheckbox called “Star Devices” unchecked. For SRDF/Star testfailover replica pairings, check the box so that VSI will instead listall discovered and valid SRDF/Star devices.

4. VSI discovers all of the replicated devices and all of the possiblecandidates that can be used for TimeFinder/Clone replicadevices. VSI then makes suggestions for device pairing based onTimeFinder/Clone rules for valid target devices1 (assuming thereare appropriate candidates available). VSI can discover multiplevalid candidate devices for a given R2 device and will list them inthe Replica Device column. This cell is a drop down list thatdisplays all possibilities and from which one can be selected asseen in Figure 70. VSI will display what it decides to be the mostappropriate device in the top of the column. Priority is given todevice pairings that are already configured for that R2 in theoptions file. Second in priority are pairs that have existingTimeFinder replication sessions. Therefore, if no valid pairs arealready configured in the options file, VSI will next opt to list anycurrently active TimeFinder target devices and their replica state(PreCopy, Copied etc.). Users may choose any of the deviceslisted in the dropdown regardless of the priority assigned to it byVSI (these are merely suggestions).

1. Thin devices (TDEVs) must be bound to a thin pool or they will be filteredout as replica candidates.



Figure 70 Determining devices pairings with VSI for Test failover TimeFinderoperations

5. Once a user has selected the appropriate pairs in VSI, click “SaveOptions File” as shown in Figure 70. This will save the devicepairs to the EmcSrdfSraTestFailoverConfig.xml file. The adapteruses the pairings defined in the options file to determine thedevices to use when a recovery plan is tested. The options file alsoreflects the TimeFinder technology to be used when a recoveryplan is tested.

Figure 71 shows the options file containing the pairings requiredto test a recovery plan. The figure shows the organization of theoptions file and the tag, ArrayID, which enables the adapter tosupport the feature that allows definition of a recovery plan thatinclude multiple storage arrays. Thus, with the SRDF Adapterwhen a recovery plan involves multiple Symmetrix arrays at thetarget, a CopyInfo stanza with appropriate TimeFindertechnology, Symmetrix serial number, and device pair definitionsshould be created for each Symmetrix array.

Testing recovery plans using EMC TimeFinder/Clone 155

156


Figure 71 Test failover options XML file for TimeFinder/Clone

6. Click the Test link after selecting the recovery plan that is to betested. The EMC SRDF Adapter for VMware vCenter SiteRecovery Manager uses the information provided in the optionsfile and performs appropriate TimeFinder/Clone operations.

The testing of a recovery plan is shown in Figure 72. It can be seenfrom the figure that the EMC SRDF Adapter has successfully createda local copy of the devices defined in its protection group usingTimeFinder/Clone technology. VMware vCenter Site RecoveryManager has also successfully used the copy of the datastore on theVMware ESX servers at the recovery site to power on the replicavirtual machines hosted on those devices.



Figure 72 Running a test of a recovery plan using EMC TimeFinder/Clonetechnology

The state of the Clone pairs can be verified by utilizing the SolutionsEnabler command, symclone or Unisphere for VMAX. An examplescreenshot of Unisphere is shown in Figure 73. The state of thedevices should either be Copied or CopyInProg.


158


Figure 73 Determining the state of the clone pairs during the test of a recoveryplan

It is important to note that the EMC SRDF Adapter creates andactivates the TimeFinder/Clone sessions only if the state of the SRDFrelationship for the devices involved in the recovery plan is either“Synchronized” or “Consistent.” Without the additional check, it isimpossible for the adapter to determine if the source devices for theTimeFinder/Clone sessions contain valid data for testing recoveryplans. However, there are situations (for example, enterpriseconsistency configurations that contain data not within a VMwareenvironment) in which users may nevertheless desire to test therecovery plan even though the SRDF link is in either a “Suspended”or “Split” state. To accommodate this, EMC SRDF Adapter providesthe option, TestFailoverForce. When this option is changed from thedefault value of “No” to “Yes,” the EMC SRDF Adapter ignores thestate of the SRDF links and creates a TimeFinder/Clone copy. Theonus in this case is on the user to ensure that the source devices (R2)in the TimeFinder/Clone sessions contain a consistent and valid copyof the data. The option, TestFailoverForce, needs to be enabled in thefile EmcSrdfSraGlobalOptions.xml using the EMC VSI SymmetrixSRA Utilities.



The termination of the recovery plan test is accomplished by clickingthe Cleanup link as discussed previously in this chapter. As shown inFigure 74, this step automatically invokes the EMC SRDF Adapter tore-create the cloning relationship. This optimizes the performance offuture tests of recovery plans while minimizing the impact on thestorage arrays when copying data to the target clone devices.

Figure 74 Clone relationship after the Cleanup operation after a test failover

EMC SRDF Adapter includes the global option,TerminateCopySessions, in the file EmcSrdfSraGlobalOptions.xml.By default the value for this option is set to “No” to allow forre-creation of the clone relationship. However, if the value for theoption is changed to “Yes,” the EMC SRDF Adapter automaticallyterminates the clone relationship when the execution of the recoveryplan is completed. This setting, like other advanced settings, can beenabled/disabled using the EMC VSI Symmetrix SRA Utilities.


160


Testing recovery plans using EMC TimeFinder/SnapThe EMC SRDF Adapter for VMware vCenter Site Recovery Managersupports EMC TimeFinder/Snap technology to test recovery plans.For Symmetrix arrays running Enginuity 5874 this functionality islimited to devices that are replicated using the synchronous mode ofSRDF. For VMAX arrays running Enginuity 5875 and laterTimeFinder/Snap with Asynchronous replication is permitted whenwrite pacing is enabled. Write pacing is discussed earlier in thischapter in the section, “Device Write Pacing and TimeFinder-basedTest Failover” on page 149.

The process for using TimeFinder/Snap replicas is explained in thissection. The following steps should be adhered to when testingrecovery plans with TimeFinder/Snap technology:

1. Ensure the target of the Snap operation is presented (masked) tothe appropriate VMware ESX hosts at the recovery site.

2. Using VSI in Figure 75 one can check for the presence of localreplication candidates. For TimeFinder/Snap operations, it isimportant to check that the correct amount of properly sizeddevices are present on the recovery cluster. This can be verifiedusing the test failover configuration feature of VSI SymmetrixSRA Utilities as shown in Figure 75.

3. To configure test failover replica pairs for two site SRDF, leave thecheckbox called “Star Devices” unchecked. For SRDF/Star testfailover replica pairings, check the box so that VSI will instead listall discovered and valid SRDF/Star devices.

4. VSI discovers all of the replicated devices and all of the possiblecandidates that can be used for TimeFinder/Snap replica devices(VDEVs). VSI then makes suggestions for device pairing based onTimeFinder/Snap rules for valid target devices (assuming thereare appropriate candidates available). VSI can discover multiplevalid candidate devices for a given R2 device and will list them inthe Replica Device column. For SRDF/A devices VSI will check tomake sure write pacing is enabled, if write pacing is not enabledthe column replica options will be marked as N/A. The replicadevice cell is a drop down list that displays all possibilities andfrom which one can be selected as seen in Figure 75. VSI willdisplay what it decides to be the most appropriate device in thetop of the column. Priority is given to device pairings that arealready configured for that R2 in the options file. Second in



priority are pairs that have existing TimeFinder replicationsessions. Therefore, if no valid pairs are already configured in theoptions file, VSI will next opt to list any currently activeTimeFinder target devices and their replica state (Recreated,CopyOnWrite etc...). Users may choose any of the devices listedin the dropdown regardless of the priority assigned to it by VSI(these are merely suggestions). Additionally, whenTimeFinder/Snap technology is leveraged, a non-default savepool can be specified. It is important to note that only one poolper array can be chosen for all of the devices in a protectiongroup.

Figure 75 Configuring TimeFinder/Snap replica pairs for test failover

5. Once the pairs and the Save (Snap) Pool have been chosen in VSI,the user must click “Save Options File” shown in Figure 75. Thiswill save the device pairs and the save pool to theEmcSrdfSraTestFailoverConfig.xmlfile on the recovery site. Thedevice pairs will define the relationship between the SRDF deviceat the recovery site (R2) and the Snap target. The adapter uses thepairings defined in the options file to determine the devices to usewhen a recovery plan is tested.

Testing recovery plans using EMC TimeFinder/Snap 161

162


Figure 76 shows the options file containing the pairings requiredto test a recovery plan. The figure shows the organization of theoptions file and the tag, ArrayID, which enables the adapter tosupport the feature that allows definition of a recovery plan thatinclude multiple storage arrays. Thus, with the SRDF Adapterwhen a recovery plan involves multiple Symmetrix arrays at thetarget, a CopyInfo stanza with appropriate TimeFindertechnology, Symmetrix serial number, and device pair definitionsshould be created for each Symmetrix array.

Figure 76 Defining device pairs and TimeFinder technology when testingrecovery plans

6. Click the Test link after selecting the recovery plan that is to betested. The EMC SRDF Adapter for VMware vCenter SiteRecovery Manager uses the information provided in the optionsfile and performs appropriate TimeFinder/Snap operations.

7. The testing of a recovery plan is shown in Figure 77. It can be seenfrom the figure that the EMC SRDF Adapter has successfullycreated a local copy of the devices defined in its protection groupusing TimeFinder/Snap technology. VMware vCenter Site



Recovery Manager has also successfully used the copy of thedatastore on the VMware ESX servers at the recovery site topower on the replica virtual machines hosted on those devices.

Figure 77 Running a test of a recovery plan using EMC TimeFinder/Snaptechnology

Note: If a test recovery fails with the following error in the SRDF SRA log:[SYMAPI_C_MIXED_SNAP_TGT: Copy sessions do not allowmixing virtual and non-virtual targets] this means that there isat least one VP Snap session associated with the given R2 device.TimeFinder/Snap and TimeFinder/VP Snap sessions can not co-exist withthe same source. If this error is encountered, an administrator must eitherterminate the VP Snap session(s) or the SRM admin must choose analternative method of test failover to TimeFinder/Snap.

The state of the Snap pairs when the test environment is running canbe verified by utilizing the Solutions Enabler command, symsnap orUnisphere for VMAX. An example screenshot of Unisphere is shownin Figure 78.


164


Figure 78 Determining state of VDEV device during the testing of a recovery plan

The EMC SRDF Adapter creates and activates the TimeFinder/Snapsessions only if the state of the SRDF relationship for the devicesinvolved in the recovery plan is “Synchronized” without theadditional check, it is impossible for the adapter to determine if thesource devices for the TimeFinder/Snap sessions contain valid datafor testing recovery plans. However, there are situations (for example,enterprise consistency configurations that contain data not within aVMware environment) in which users may nevertheless desire to testthe recovery plan even though the SRDF link is in either a“Suspended” or “Split” state. To accommodate this, EMC SRDFAdapter provides the option, TestFailoverForce. When this option ischanged from the default value of “No” to “Yes,” the EMC SRDFAdapter ignores the state of the SRDF links and creates aTimeFinder/Snap copy. The user needs to ensure that the sourcedevice in the TimeFinder/Snap sessions contain a consistent andvalid copy of the data. This setting, like all advanced settings can beenabled/disabled using the EMC VSI Symmetrix SRA Utilities.

The termination of the recovery plan test is accomplished by clickingthe Cleanup link as discussed previously in this chapter.



The EMC SRDF Adapter automatically re-creates theTimeFinder/Snap sessions as seen in Figure 79. The behavior can bechanged to terminate all sessions by setting the global option,TerminateCopySessions to “Yes”.

Figure 79 Terminating TimeFinder/Snap sessions at the end of the test of arecovery plan


166


Testing recovery plans using EMC TimeFinder/VP SnapThe EMC SRDF Adapter 5.1 and higher for VMware vCenter SiteRecovery Manager supports EMC TimeFinder/VP Snap technologyto test recovery plans.

TimeFinder VP Snap provides space-efficient snaps for virtual pooldevices (Symmetrix Virtual Provisioning/Thin devices), whileoffering the efficiency of TimeFinder/Snap technology withimproved cache utilization and simplified pool management.TimeFinder VP Snap is available for all Symmetrix VMAX Familystorage arrays running Enginuity level of 5876 and higher, andSolutions Enabler V7.4.

TimeFinder/VP Snap with Asynchronous replication is permittedwhen write pacing is enabled. Write pacing is discussed earlier in thischapter in the section, “Device Write Pacing and TimeFinder-basedTest Failover” on page 149.

Note: It is important to note that VP Snap is a feature extension ofTimeFinder/Clone NOT TimeFinder/Snap and therefore requiresTimeFinder/Clone licensing.

TimeFinder/VP Snap OverviewThis new TimeFinder technology adds the ability to share capacityallocation for track groups originating from the same source volume.TimeFinder VP Snap is easy to manage and combines the benefits ofboth full copy clone and space saving snap technologies.

TimeFinder VP Snap provides the ability for multiple clone sessionsto target thin devices and share extent allocations within the thinpool. This reduces the space needed for the storage of saved tracks.

VP Snap sessions copy data from the source device to the targetdevice only if triggered by a host write operation. Read I/Os toprotected tracks on the target device do not result in data beingcopied.

For a single activated VP Snap session on a source device, the targetrepresents a single point-in-time copy of the source. Copied dataresides on allocations in the thin pool. For example, if tracks 100, 200,and 300 are written on the source device, the point-in-time data foreach track resides in a unique allocation in the thin pool.



When there is a second VP Snap session from the same source deviceto a different target, the allocations can be shared. For example, ifthere is a write I/O to tracks 1100, 1200, and 1300 on the sourcedevice, the data is new to both targets' point-in-time, and thepoint-in-time data can be saved in a single set of allocations that isshared by both target devices.

However, if there is another write I/O to tracks 100, 200, or 300 on thesource device, since the data is new for only the second session'spoint-in-time, when the tracks are copied to the second target device,the point-in-time data is put into a set of allocations that are uniquelyowned by the second target device. In other words, the allocations forthese tracks cannot be shared.

If more VP Snap sessions are added to the same source device, data iscopied to the targets based on whether the source data is new withrespect to the point-in-time of each copy. When data is copied to morethan one target, only a single shared copy resides in the thin pool.

If there is a write I/O to one or more of the tracks stored in a sharedallocation, the affected allocation for that target will be split off fromthe shared group because the data is now different than the data forthe other targets that are using that allocation. The new data of thewritten target will be stored in a separate allocation while the sharedallocation will still contain the data of the other targets.

When VP Snap sessions are terminated, the target device is removedfrom any shared allocations that were part of the session, and anynon-shared allocations for that device are deallocated. When all butone of the VP Snap sessions are terminated, the last remaining sessionuses the same space in the thin pool, but it is no longer a sharedallocation. Upon termination of the last session, the space isdeallocated.

Configuring TimeFinder/VP Snap for test failoverThe process for using TimeFinder/VP Snap replicas of R2 devicesusing SRDF Synchronous mode is explained in this section. Thefollowing steps should be adhered to when testing recovery planswith TimeFinder/VP Snap technology:

1. Ensure the target of the VP Snap operation is presented (masked)to the appropriate VMware ESX hosts at the recovery site.

Testing recovery plans using EMC TimeFinder/VP Snap 167

168


2. Using VSI in Figure 80 one can check for the presence of localreplication candidates. For TimeFinder/VP Snap operations, it isimportant to check that the correct amount of properly sizeddevices are present on the recovery cluster. This can beautomatically verified using the test failover configurationfeature of VSI Symmetrix SRA Utilities as shown in Figure 80.

3. The VSI SRA Utilities version 5.4 introduces support forTimeFinder/VP Snap for test failover. To configure the pairs, theTimeFinder/VP Snap option should be chosen in the drop-downmenu.VSI discovers all of the replicated devices and all of thepossible candidates that can be used for TimeFinder/VP Snapreplica devices (thin device, non-RDF device, bound to a pool,etc.). VSI then makes suggestions for device pairing based onTimeFinder/VP Snap rules for valid target devices (assumingthere are appropriate candidates available). VSI can discovermultiple valid candidate devices for a given R2 device and willlist them in the Replica Device column. This cell is a drop downlist that displays all possibilities and from which one can beselected as seen in Figure 80.

VSI will display what it decides to be the most appropriate devicein the top of the column. Priority is given to device pairings thatare already configured for that R2 in the XML file. Second inpriority are pairs that have existing TimeFinder replicationsessions. Therefore, if no valid pairs are already configured in theXML file, VSI will next opt to list any currently active TimeFindertarget devices and their replica state (Created, CopyOnWrite etc.).Users may choose any of the devices listed in the dropdownregardless of the priority assigned to it by VSI (these are merelysuggestions).



Figure 80 Determining devices presented to VMware ESX on the recovery site

4. Once the pairs have been chosen in VSI, the user must click “SaveOptions File” shown in Figure 80. This will save the device pairsto the EmcSrdfSraTestFailoverConfig.xml file on the recovery site.The device pairs will define the relationship between the SRDFdevice at the recovery site (R2) and the VP Snap target. Theadapter uses the pairings defined in the options file to determinethe devices to use when a recovery plan is tested.

Figure 81 shows the options file containing the pairings requiredto test a recovery plan. The figure shows the organization of theoptions file and the tag, ArrayID, which enables the adapter tosupport the feature that allows definition of a recovery plan thatinclude multiple storage arrays. Thus, with the SRDF Adapterwhen a recovery plan involves multiple Symmetrix arrays at the


170


target, a CopyInfo stanza with appropriate TimeFindertechnology, Symmetrix serial number, and device pair definitionsshould be created for each Symmetrix array.

Figure 81 Defining device pairs and TimeFinder technology when testingrecovery plans

5. Click the Test link after selecting the recovery plan that is to betested. The EMC SRDF Adapter for VMware vCenter SiteRecovery Manager uses the information provided in the optionsfile and performs appropriate TimeFinder/VP Snap operations.

6. The testing of a recovery plan is shown in Figure 82 on page 171.It can be seen from the figure that the EMC SRDF Adapter hassuccessfully created a local copy of the devices defined in itsprotection group using TimeFinder/VP Snap technology.VMware vCenter Site Recovery Manager has also successfullyused the copy of the datastore on the VMware ESX servers at therecovery site to power on the replica virtual machines hosted onthose devices.



Figure 82 Running a test of a recovery plan using EMC TimeFinder/VP Snaptechnology

Note: If a test recovery fails with the following error in the SRDF SRA log:[SYMAPI_C_MIXED_SNAP_TGT: Copy sessions do not allowmixing virtual and non-virtual targets] this means that there isat least one TimeFinder/Snap session associated with the given R2 device.TimeFinder/Snap and TimeFinder/VP Snap sessions can not co-exist withthe same source. If this error is encountered, an administrator must eitherterminate the Snap session(s) or the SRM admin must choose an alternativemethod of test failover to TimeFinder/VP Snap.

The state of the VP Snap pairs when the test environment is runningcan be verified by utilizing the Solutions Enabler command,symclone or Unisphere for VMAX. An example screenshot ofUnisphere is shown in Figure 78.


172


Figure 83 Determining state of VP Snap pair during the testing of a recovery plan

The termination of the recovery plan test is accomplished by clickingthe Cleanup link as discussed previously in this chapter. The EMCSRDF Adapter automatically re-creates the TimeFinder/VP Snapsessions during Cleanup as seen in Figure 79. The behavior can bechanged to terminate all sessions by setting the global option,TerminateCopySessions to “Yes”.



Figure 84 Recreating TimeFinder/VP Snap sessions at the end of the test of arecovery plan


174


Test Failover Advanced OptionsIn the majority of cases, the default behavior of the SRDF SRA meetsthe needs of most users. Nevertheless, there are situations where thebehavior of the adapter must be altered to satisfy differing needs orless common configurations. To allow for this tighter control, testfailover offers the following advanced options:

1. IgnoreActivatedSnapshots—allows the user to manually createand activate replica devices before the test failover operation.DEFAULT=DISABLED.

2. TestFailoverForce—allows the test failover operation to continueeven if the RDF link is “Split” or “Suspended”.DEFAULT=DISABLED

3. CheckForVirtualDisks—adds an additional check into the SRDFSRA test failover that ensures chosen replica devices are not inuse by the recovery VMware environment.DEFAULT=DISABLED

4. TerminateCopySessions—during cleanup of the test failover theSRDF SRA terminates the TimeFinder session instead ofautomatically re-creating it. DEFAULT=DISABLED

5. TestFailoverWithoutLocalSnapshots—allows the user to forgothe use of TimeFinder during test failover and instead split theRDF link and run the test off of the R2 devices. This option isdiscussed separately in the next main section in this chaptercalled, “Test Failover without TimeFinder” on page 183.DEFAULT=DISABLED

The following four subsections will discuss each option andwhen/how they should be used. All advanced options can bechanged by using the SRDF SRA Global Options editor in the EMCSymmetrix SRA Utilities for VSI.



Figure 85 VSI Symmetrix SRA Utilities Global Options editor

IgnoreActivatedSnapshotsEven as pervasive as server virtualization has become, manyenvironments still host a mixture of virtual and physical servers.Furthermore, these servers have dependencies between their hostedapplications without regard to underlying server architecture therebycreating relationships that traverse the boundaries between thevirtual and physical realms. Consequently, during a disaster recoverytest, enterprise consistent copies of application data across physicalservers and virtual machines may be required in order for the test tobe a true success.

The default behavior of the SRA does not allow for this to happen. Ifthe customers have consistently activated the R2 and replica devicesfor the physical and virtual environment themselves the SRA willincrementally re-create the VMware-related devices and then activate

Test Failover Advanced Options 175

176


them again during test failover. Therefore, the VMware replicadevices, while still internally consistent, will no longer be consistentwith the devices external to VMware.

Therefore, version 5.1 and later of the SRDF SRA introduces newbehavior to accept the already activated state of the VMware replicadevices. In order to address this need, a new global option"IgnoreActivatedSnapshots" has been added to the config file"EmcSrdfSraGlobalOptions.xml". The default value of this option willbe "No". If set to "Yes", the adapter will be tuned to satisfy thisscenario. This behavior enables the user to consistently activate theVMware-related replicas (TimeFinder/Clone, Snap or VP Snap)along with any external devices at the appropriate time beforeexecuting a test failover operation within SRM. The SRA will ignorethe already activated devices and report success.

The SRDF SRA still requires a correctly configured"EmcSrdfSraTestFailoverConfig.xml" file in order to perform the testoperation. This configuration file contains information such as"ArrayId", "Copy Type" and "Device Pairs". The device pair sourcecorresponds to each of the "R2" devices contained in the consistencygroup that is being test failed over. The device pair target representsthe device on which the user wants to take the snapshots of the R2devices.

The information in the aforementioned options file is critical for SRAin order to perform the test operation. SRA returns results to SRMwith these snapshot device names along with their respectiveWWNs. This enables SRM to mount the datastores and boot VMsfrom the target devices reported in the SRA response.

In order to have the SRDF SRA work in this specific scenario, one ofthe two requirements must be met.

◆ After the consistent activate of the TimeFinder device pairs, aseparate consistency group should be created for VMware relatedRDF devices. The XML configuration file"EmcSrdfSraTestFailoverConfig.xml" should be appropriatelyconfigured to contain all the target device pairs for the R2 devicescontained in the consistency group. In this approach, EMC VSIcan be used to populate the XML config file.

◆ If a separate consistency group cannot be created for VMwarerelated RDF devices, the XML configuration file"EmcSrdfSraTestFailoverConfig.xml" should be appropriatelyconfigured to contain all the target device pairs for the R2 devices



contained in the consistency group. This means that the R2devices presented to the VMware environment and the R2devices presented to servers external to the VMwareenvironment (and their respective TF target devices) must beadded to the XML file. Since VSI does not have the capability tomanipulate and configure the XML pairings file with devices notpresented to the VMware environment, the user must manuallyenter the device pairs to match the device group. The SRM testfailover request contains the name of the consistency group onwhich a specified operation needs to be performed and as aresult, the SRA performs the requested operation on the entireconsistency group as a unit.

In the first scenario, the user is responsible to create/activate theTimeFinder session for the TimeFinder pairs prior to performing the"Test" operation. If they do not, the SRA will only perform acreate/activate on the VMware devices and not the external onessince they are not in the device group. If the user re-creates theTimeFinder targets before the “Cleanup” operation (and theIgnoreActivatedSnapshots is still enabled) the SRA will ignore thealready re-created devices and report success. In the second scenariothe SRA will create/activate the VMware and non-VMware devicesduring the test operation so user intervention is not required in thiscircumstance.

Note: IgnoreActivatedSnapshots is also now supported withTestFailoverWithoutLocalSnapshots. Discussion on the interaction of thesetwo features will be discussed later in this chapter in the section entitled,“Test Failover without TimeFinder” on page 183. This section discusses theadvanced option TestFailoverWithoutLocalSnapshots in detail and isrequired knowledge to fully understand how this interaction behaves.

TestFailoverForceThe EMC SRDF Adapter creates and activates the TimeFindersessions only if the state of the SRDF relationship for the devicesinvolved in the recovery plan is “Synchronized”. Without theadditional check, it is impossible for the adapter to determine if thesource devices for the TimeFinder sessions contain valid data fortesting recovery plans. However, there are situations (for example,enterprise consistency configurations that contain data not within aVMware environment) in which users may nevertheless desire to testthe recovery plan even though the SRDF link is in either a


178


“Suspended” or “Split” state. To accommodate this, EMC SRDFAdapter provides the option, TestFailoverForce. When this option ischanged from the default value of “No” to “Yes,” the EMC SRDFAdapter ignores the state of the SRDF links and creates a TimeFindercopy of whatever data is on the R2. In this case, the onus is on theuser to ensure that the source devices in the TimeFinder sessionscontain a consistent and valid copy of the data.

CheckForVirtualDisksIntroduced in version 5.1 of the SRDF SRA is an additional securitycheck during test failover operations called CheckForVirtualDisks.This advanced option will query the input candidate TimeFinderreplica target devices for whether or not they are in use as an RDM ora VMFS. If any of the input devices are detected to be in use in therecovery VMware environment, the SRDF SRA will fail the testfailover operation before creating a TimeFinder session for thatdevice pair. This will prevent the accidental overwriting of RDMs orVMFS volumes by the copied data. It is important to note that theSRA will only check for the use of the device in the recovery sidevCenter associated with the recovery side SRM server. If the device isin use in another environment, VMware or otherwise, the SRA will beunable to detect that.

This option:

◆ Will mark any device containing a VMFS volume as in-use.

◆ Will mark devices as in-use if they are a raw device mapping(RDM). The virtual machine(s) using the RDM do not have to becurrently powered on for the device to be recognized as in-use.

◆ If a virtual machine has raw device mappings and that virtualmachine is not currently registered to the vCenter server, theSRDF SRA will not be able to identify that device as in use as anRDM and it will be seen as a valid candidate device for testfailover.

This option can be enabled using the VSI Symmetrix SRA Utilities asseen in Figure 86.



Figure 86 Enabling CheckForVirtualDisks with VSI Symmetrix SRA Utilities

While this option can be enabled via VSI in the global options file,there is, however, further configuration required to provide the SRDFSRA access to the information it requires. The SRDF SRA connects tothe local vCenter and queries the vCenter inventory to see if any ofthe input replica devices are in use.

It order to enable this behavior, an administrator must grant vCenteraccess for the Solutions Enabler install local to the recovery SRMserver that will execute the test recovery operation. It is important tonote that this must be done on the local installation of SolutionsEnabler on the SRM server and not the remote SYMAPI server. This isregardless of whether or not a remote SYMAPI server is configuredfor use by SRDF SRA control operations.

Note: As readers may note, this is the same process required for the advanceddiscovery option, FilterNonVmwareDevices. If authorizations were createdfor use with that option, no further configuration is required and thefollowing process can be skipped.


180


If the credentials are missing or incorrect the SRDF SRA will not failthe test failover process and will continue to create replicas withoutchecking for in-use devices. Therefore, it is important to ensure thatvalid credentials are added to the SYMAPI database prior to testrecovery.

For example:

In this environment, the FQDN of the recovery site vCenter server isch-vcenter4.ebc.emc.local. The vCenter server can be accessed via adomain account “hostec”.

Note: The user account must have at least Read Only permissions (assignedto the user account within vCenter) to the recovery site vCenter server objectand to the ESXi server objects1.

Authorization is to be added using the Solutions Enabler CLIcommand shown below (bolded portions are unique variables andare to be changed for each given environment):

symcfg add authorization -host <FQDN of vCenterserver2> -username <username> -password <password>-vmware -namespace “vmware/vc”

In order to run Solutions Enabler commands from the SRM serverWindows Command Prompt, the user must either be issuingcommands from the C:\Program Files (x86)\EMC\SYMCLI\bin\directory or have the local Windows “Path” variable configured withthis location to automatically search this location when SYMCLIcommands are issued. Furthermore, the environmental variable“SYMCLI_CONNECT_TYPE” on the SRM server must either be setto LOCAL or unset. This will ensure that the authorizationcommands are issued to the local installation of Solutions Enablerand not to a remote SYMAPI host.





An authorization must be added for the recovery site SRDF SRA. Anadministrator must log in to the recovery site SRM server and run thefollowing command:


Once this authorization is added, no further configuration is requiredand the recovery SRDF SRAs will use this authorization to query thevCenter server to verify whether or not the chosen replicas are in use.

If the SRDF SRA identifies a device that is in use, but configured inthe pairings file to be a replica device, the test recovery process willfail with the following message in the SRDF SRA log file:

SraCloneGroup::CreateCopySessions ] [ERROR]: Targetdevice [<device ID>] is already used in a Datastore oras RDM. Using it to create a snapshot will delete allthe existing data

If the credentials were input correctly or missing, furthertroubleshooting information on can be found in the viclient logcreated by the SRDF SRA. This log is located on the recovery SRMserver in the below location:




1. Optionally the password can be omitted and once the command has beenexecuted a prompt will appear asking for a password in which input willbe obfuscated with asterisks.


182


This option could be beneficial to larger VMware environmentswhere the control of each and every device is tougher due to thehigher number of simultaneous users and/or devices.

TerminateCopySessionsThe EMC SRDF Adapter automatically re-creates the TimeFindersessions during Cleanup by default. This allows for quicker re-testingof recovery plans by avoiding the step of recreation TimeFinder pairs.The behavior can be changed to instead terminate all sessionsregardless by setting the global option, TerminateCopySessions to“Yes”. If the adapter is set to terminate the copy session, it will alsoremove the target devices from the device/composite group. If it isnot set to terminate the sessions, the target devices will remain in thegroup.

This option only is relevant to test failovers that use TimeFinder. Ithas no effect on test failovers executed with the optionTestFailoverWithoutLocalSnapshots enabled.















Test Failover without TimeFinderCertain customers require all disaster recovery testing to be true testsof their ability to fail over. This means the steps taken to test thefailover process must be absolutely identical to the steps of the actualfailover process. This requirement, therefore, assures no variables aremissed in test failovers and consequently the test failover is a true testof the disaster recovery process. In the typical test failover scenario,local replication technologies are used on the remote site to make acopy of the R2 devices. The copy is then presented to the recoveryESX cluster and the environment is powered on from those replicadevices. In an actual VMware vCenter SRM failover, the replication ishalted from the protected site and the remote R2 device is presentedto the host and the production VMs are brought up on the remote R2device. These scenarios differ due to the fact that the test failover usesa copy of the SRDF replicated target and the actual failover uses theSRDF replicated target. In a true test the SRDF replicated targetwould be used to perform a test failover.

Fortunately, the EMC SRDF Adapter version 5 allows for the abilityto do exactly that. This feature enables the user to perform testfailover operations directly off of the R2 devices at the remote site bysplitting the link and write enabling the R2’s. The R2 is thenpresented to the remote ESX cluster for test recovery of the replicatedvirtual environment.

EMC does not generally recommend using this method for testfailovers. Since this process temporarily splits the link, remote dataprotection is consequently suspended and could lead to data loss orunavailability in the event of a simultaneous protected site failure.Before using this method of test failover, carefully consider thepossible ramifications of doing so.

This section is divided into eight separate sub-sections:

1. Overview and configuration

2. Two-site SRDF

3. Concurrent SRDF/Star

4. Cascaded SRDF/Star

5. Concurrent SRDF (non-Star)

Test Failover without TimeFinder 183

184


6. Cascaded SRDF (non-Star)

7. TestFailoverWithoutLocalSnapshots withIgnoreActivatedSnapshots

8. Gold copy protection

TestFailoverWithoutLocalSnapshots overview and configurationFor a quick comparison, the traditional form of test failover usingTimeFinder is schematically drawn out in Figure 87 on page 184 andthis form of test failover that does not use TimeFinder is laid out inFigure 88 on page 185.

Figure 87 Traditional test failover with TimeFinder replication



Figure 88 Test failover using SRDF without local replication

In order for the EMC SRDF Adapter to perform a test failover in thismanner (directly off the R2) an advanced option in theEmcSrdfSraGlobalOptions file must be altered. Using the VSISymmetrix SRA Utilities, as shown in Figure 89, the option<TestFailoverWithoutLocalSnapshots>1 must be set to

1. TestFailoverWithoutLocalSnapshots and TestFailoverForce are notsupported for simultaneous use.


186


”Enabled”. This will force the adapter to ignore any TimeFinderconfigurations in the options file and will then perform subsequenttest failovers off the R2.

Figure 89 Changing global options file to allow failover without TimeFinder

TestFailoverWithoutLocalSnapshots with two-site SRDFWith two-site SRDF, the replication state should be either“Synchronized” or “Consistent” (depending on the replicationmethod) before test failover. Figure 90 shows the replication and thedevice states before the test failover operation.

Figure 90 Replication and device state before test failover



When the test failover is initiated, the EMC SRDF Adapter performsan “RDF Split” operation on the RDF link. This makes the R2 devices“Ready” as shown in Figure 91.

Figure 91 Split state of SRDF replication after test failover is initiated

The EMC SRDF Adapter then initiates the test virtual machinerecovery from the R2 devices directly. Using the Storage Viewerfeature of VSI, it can be seen in Figure 92 that the recovered virtualmachines are running off of the R2 devices.

Figure 92 Recovered test virtual machine running on an R2 as seen with theStorage Viewer feature of VSI


188


When the user executes the “Cleanup” operation to complete the testfailover process, the RDF link is incrementally re-established, makingthe R2 devices once again “Write Disabled” and replicationre-initiates.

TestFailoverWithoutLocalSnapshots with Concurrent SRDF/StarThe 5.1 release of the SRDF SRA added support for the advancedoption, TestFailoverWithoutLocalSnapshots, with Concurrent Star.Since Star configurations are inherently more complex than two-siteSRDF the process requires different actions from the SRA.

Whether the recovery SRM server is on the Synchronous site or theAsynchronous site of the Star configuration, the process is essentiallythe same. What differs in the test failover workflow between the twodifferent setups is which leg of the SRDF/Star setup is isolated andrecovered.

Figure 93 and Figure 94 show a diagram of the state of the ConcurrentSRDF/Star environment when a test failover operation has executedwith the TestFailoverWithoutLocalSnapshots option enabled. Figure93 shows an environment with failover set to the Sync site(FailoverToAsyncSite is disabled) and Figure 94 shows anenvironment with failover set to the Async site(FailoverToAsyncSite3 is enabled).



Figure 93 TestFailoverWithoutLocalSnapshots enabled for Concurrent SRDF/Star tothe Sync target site


190


Figure 94 TestFailoverWithoutLocalSnapshots enabled for Concurrent SRDF/Star tothe Async target site

For the purposes of demonstration, the following example will be atest failover run off of the Synchronous site devices.

Before a test can be executed, both active links of the ConcurrentSRDF/Star configuration must be in the “Protected” state. If eitherlink is in a state besides “Protected” the operation will fail.

Note: The overall Star state can be “Protected” or “Unprotected” for testfailover but it is highly recommended that the Star state be in a “Protected”state. It should also be noted that if the Star state is “Unprotected” before thetest failover, the SRA will fully protect Star during the cleanup stageregardless of the initial Star state.

An example of a Concurrent SRDF/Star configuration prior to testfailover can be seen in a screenshot from Symmetrix ManagementConsole in Figure 95. It should be noted that both active links are inthe “Protected” state1 and the overall Star state is also “Protected”.



Figure 95 Concurrent SRDF/Star before test failover

Once the test failover begins the SRDF SRA will disable Star (if notalready in the “Unprotected” state). Then, depending on whether therecovery VMware environment is at the asynchronous target site orthe synchronous target site the SRDF SRA will perform a Star“isolate” operation on the target devices. The “isolate” operationsplits the RDF pairs and sets the R2 devices to Read/Write enabledallowing them to be used by the recovery environment.

From this point, the recovery operation proceeds normally. Theexpected state of a Concurrent SRDF/Star enivironment after a testoperation but before a cleanup can be seen in Figure 96.

1. The target sites being in a protected state is a hard requirement.


192


Figure 96 Concurrent SRDF/Star before test failover cleanup

The screenshot from Symmetrix Management Console shows threethings:

1. An “Unprotected” overall Star state

2. An isolated Synchronous target site

3. A protected Asynchronous target site

This shows that the VMware environment is at the Synchronous siteand that the devices in that location are R/W enabled while theasynchronous target site devices are unaffected.

Once the test failover has completed and all desired application testsare verified, the test failover can be ended and reverted by clickingthe Cleanup operation button. During the cleanup operation, theSRDF SRA will execute the following Star operations:

1. Disconnect the appropriate target site. This is required becausethe subsequent connect operation requires the site to be in the“Disconnected” state.

2. Connect the target site. This will resume incremental replicationto the target site and overwrite any changes made to the R2devices with the data from the R11.

3. Protect the target site. This will re-enable consistency to the targetsite.

4. Enable/protect Star. The last operation will be to fully enable Starto revert the environment back to the protected state.



TestFailoverWithoutLocalSnapshots with Cascaded SRDF/StarThe 5.1 release of the SRDF SRA added support for the advancedoption, TestFailoverWithoutLocalSnapshots, with CascadedSRDF/Star. Since Star configurations are inherently more complexthan two-site SRDF the process requires different actions from theSRA.

Due to the nature of cascaded replication, the process forTestFailoverWithoutLocalSnapshots with Cascaded SRDF/Star isslightly different in workflow depending on whether the target site isthe Synchronous site or the Asynchronous site. Each workflow willbe discussed separately in the two subsequent sub-sections.

Cascaded SRDF/Star with recovery environment at the Sync siteThe SRDF SRA 5.1 supports performing test failover withTestFailoverWithoutLocalSnapshots enabled with CascadedSRDF/Star with the recovery site at the Synchronous target site. Thispresumes that the FailoverToAsyncSite option is set to disabled forboth instances of the SRDF SRA.

Before a test with either configuration can be executed, both activelinks of the Cascaded SRDF/Star configuration must be in the“Protected” state. If either link is in a state besides “Protected” theoperation will fail.

The overall Star state can be “Protected” or “Unprotected” for testfailover but it is highly recommended that the Star state is initially ina “Protected” state. It should also be noted that if the Star state is“Unprotected” before the test failover, the SRA will fully protect Starduring the cleanup stage regardless of the initial Star state.

Table 12 chronologically depicts the steps automatically taken by theSRDF SRA for test failover to the Synchronous target site of aCascaded SRDF/Star configuration1.

1. In the SRDF SRA log, operations involving the different sites are referredto as site1 or site2. These refer to the sync site and the async siterespectively.


194


When testing Cascaded SRDF/Star with theTestFailoverWithoutLocalSnapshots mechanism on the Synchronoustarget site, BOTH links are suspended for the duration of the test.This means that there is NO continuous remote protection until thetest has completed and the devices have been re-synchronized. Sincethe test is running on the Synchronous site, that site will no longerprovide a valid copy of the production data until aresynchronization due to changes incurred during the test. TheAsynchronous site will contain a valid point-in-time copy but if theworkload site is lost during the test any updates committed to theproduction data during the test will be lost. It is very important tounderstand this and the possible repercussions before choosing thismethod of test failover.

Figure 93 shows a diagram of the final state of a Cascaded SRDF/Starenvironment when a test failover operation has completed and beforea SRM Cleanup operation is run.

Table 12 TestFailoverWithoutLocalSnapshots to Synchronous target site withCascaded SRDF/Star

Step Star action Description

1 Disable Star Before any site can be manipulated the overallStar state must be “Unprotected”. If Star isprotected at the start of the test failover, theSRDF SRA will disable Star.

2 UnprotectAsync site

Disables consistency at the async site. Due tothe nature of a Cascaded setup, the async sitemust be unprotected and disconnected beforethe Sync site can be isolated. An unprotectoperation must occur before the site can bedisconnected.

3 DisconnectAsync site

Suspends RDF replication from the R21 (sync)to the R2 (async) devices. If this action was notperformed, any changes to the sync site duringthe test would be propagated to the async siteleaving no valid point-in-time copies of the R1device.

4 Isolate Syncsite

Disables consistency and suspends RDFreplication from the R1 (workload) devices tothe R21 (Sync) and sets the R21 devices toread/write enabled.



Figure 97 TestFailoverWithoutLocalSnapshots enabled for Cascaded SRDF/Star tothe Sync target site

Figure 97 shows that the recovered test VMware environment is atthe Synchronous site and consequently the devices in that locationare read/write enabled. The asynchronous target site devices are alsodisconnected but remain in the write disabled state assuring at leastthere are valid point-in-time copies of the R1 workload devices thatpersist throughout the test.

Once the test failover has completed and all desired application testsare verified, the test failover can be terminated and reverted byclicking the Cleanup operation button.


196


Table 13 chronologically depicts the steps taken by the SRDF SRA forthe test failover cleanup operation for the Synchronous target site of aCascaded SRDF/Star configuration.

Cascaded SRDF/Star with recovery environment at the Async siteThe SRDF SRA 5.1 supports performing test failover with theTestFailoverWithoutLocalSnapshots behavior with CascadedSRDF/Star with the recovery site at the Asynchronous target site.This presumes that the FailoverToAsyncSite option is set to enabledfor both instances of the SRDF SRA.

Before a test can be executed, both active links of the CascadedSRDF/Star configuration must be in the “Protected” state. If eitherlink is in a state besides “Protected” the operation will fail.

The overall Star state can be “Protected” or “Unprotected” for testfailover but it is highly recommended that the Star state be in a“Protected” state prior to the test failover. It should also be noted thatif the Star state is “Unprotected” before the test failover, the SRA willfully protect Star during the cleanup stage regardless of the initialStar state.

Table 13 TestFailoverWithoutLocalSnapshots Cleanup operation with theSynchronous target site with Cascaded SRDF/Star

Step Star Action Description

1 DisconnectSync site

Star requires that a site is in the disconnectedstate before it can be reconnected andreplication be re-established. Therefore adisconnect operation must be run to transitionthe sync site from a state of isolated.

2 Connect Syncsite

Incrementally re-establish replication to Syncsite from the workload site

3 Connect Asyncsite

Incrementally re-establish replication from theSync site to the Async site.

4 Protect Syncsite

Re-activate consistency protection to the Syncsite

5 Protect Asyncsite

Re-activate consistency protection to the Asyncsite

6 Enable Star Re-enables consistency protection andthree-site recoverability across the three sites.



Figure 98 shows a diagram of the final state of a Cascaded SRDF/Starenvironment when a test failover operation has completed and beforea SRM Cleanup operation is run.

Figure 98 TestFailoverWithoutLocalSnapshots enabled for Cascaded SRDF/Star tothe Async target site

Table 14 TestFailoverWithoutLocalSnapshots to Asynchronous target site withCascaded SRDF/Star


1 Disable Star Before any site can be manipulated the overallStar state must be “Unprotected”. If Star isprotected at the start of the test failover, theSRDF SRA will disable Star.

2 Isolate Asyncsite

Disables consistency protection and suspendsRDF replication from the R21 (sync) devices tothe R2 (async) and sets the R2 devices toread/write enabled.


198


Figure 98 shows that the recovered test VMware environment is atthe Asynchronous site and that the devices in that location areread/write enabled. The Synchronous target site devices remainsynchronized and consistent with the workload site withoutinterruption throughout the test procedure.


Table 15 chronologically depicts the steps taken by the SRDF SRA forthe test failover cleanup operation for the Asynchronous target site ofa Cascaded SRDF/Star configuration.

TestFailoverWithoutLocalSnapshots with Concurrent SRDF (Non-Star)The SRDF SRA 5.1 introduces support for non-Star three-siteConcurrent SRDF. Along with general support, the ability to executea test failover using TestFailoverWithoutLocalSnapshots enabled isalso supported.

As previously mentioned in chapter one, SRDF Concurrent is onlysupported in a mixed mode configuration. This means that one leg ofthe concurrent topology must be operating in Synchronous mode andthe other leg must be in Asynchronous mode. In other words, the twolegs cannot be both Async or both Sync—they must be a mix.Futhermore, the SRDF SRA only supports failover to theAsynchronous site of SRDF Concurrent. If a users wishes to failover

Table 15 TestFailoverWithoutLocalSnapshots Cleanup operation with theAsynchronous target site with Cascaded SRDF/Star


1 DisconnectAsync site

Star requires that a site is in the disconnectedstate before it can be reconnected andreplication be re-established. Therefore adisconnect operation must be run to transitionthe async site from a state of isolated.

2 Connect Asyncsite

Incrementally re-establish replication from theSync site to the Async site.

3 Protect Asyncsite

Re-activate consistency protection to the Asyncsite

4 Enable Star Re-enables consistency protection andthree-site recoverability across the three sites.



to the Synchronous site, it must be done outside of SRM. As a result,TestFailoverWithoutLocalSnapshots is only supported for testrecovery on the Asynchronous site.

TestFailoverWithoutLocalSnapshots has the same requirements astest failover with TimeFinder:

1. Concurrent must be in a mixed mode configuration

2. The asynchronous leg of the Concurrent SRDF setup must haveconsistency enabled

3. The Asynchronous devices of the Concurrent setup must eitherbe in a state of “Consistent” or “SyncInProg”. Although“SyncInProg” is an acceptable state, the SRA must wait forcomplete synchronization before continuing which can lead toSRM timeout scenarios. It is therefore recommended to ensure thedevices are all “Consistent”.

If the Concurrent SRDF configuration is valid, the SRDF SRA willproceed with the RDF Split operation on the replication between theWorkload site and the Asynchronous site. The Split operation willsuspend replication and make the Asynchronous devices Read/Writeenabled to the hosts. An example of a Concurrent SRDF environmentafter the test failover operation but before the Cleanup operation isdepicted in the diagram in Figure 100.


200


Figure 99 TestFailoverWithoutLocalSnapshots and Concurrent SRDF


The Cleanup operation consists of performing an incrementalestablish on the Asynchronous devices which will revert them totheir write-disabled state and overwrite any invalid tracks on the R2devices.

TestFailoverWithoutLocalSnapshots with Cascaded SRDF (Non-Star)The SRDF SRA 5.1 introduces support for non-Star three-siteCascaded SRDF1. Along with general support, the ability to execute atest failover using TestFailoverWithoutLocalSnapshots is alsosupported.

1. SRDF Extended Distance Protection (EDP) also known as DisklessCascaded is not support with the SRDF SRA.



As previously mentioned in chapter one, Cascaded SRDF is onlysupported in a mixed mode configuration. This means that the firstleg (also known as a “hop”) of the cascaded topology must beoperating in Synchronous mode and the second leg must be inAsynchronous mode. Futhermore, the SRDF SRA only supportsfailover to the tertiary/Asynchronous site of Cascaded SRDF. If ausers wishes to failover to the Synchronous site, it must be doneoutside of SRM. As a result, TestFailoverWithoutLocalSnapshots isonly supported for test recovery on the Asynchronous site.

TestFailoverWithoutLocalSnapshots has the same requirements astest failover with TimeFinder:

1. The Synchronous devices must be in a “Synchronized” state andthe Asynchronous devices must be in the “Consistent” state1.“SyncInProg” is also an acceptable state for devices but the SRAmust wait for complete Synchronization before continuing whichcan lead to SRM timeout scenarios.

2. Both the first and second hop must have consistency enabled.

If the Cascaded SRDF configuration is valid, the SRDF SRA willproceed with the RDF Split operation on the replication between theSynchronous site and the Asynchronous site. The Split operation willsuspend replication and make the Asynchronous devices read/writeenabled to the hosts. An example of a Cascaded SRDF environmentafter the test failover operation but before the Cleanup operation isdepicted in the diagram in Figure 100.

1. The second leg can be split beforehand if IgnoreActivatedSnapshots isenabled.


202


Figure 100 TestFailoverWithoutLocalSnapshots and Cascaded SRDF


The Cleanup operation consists of performing an incrementalestablish on the Asynchronous devices which will revert them totheir write-disabled state and overwrite any invalid tracks on the R2devices.

TestFailoverWithoutLocalSnapshots and IgnoreActivatedSnapshotsVersion 5.1 of the SRDF SRA allows the options,TestFailoverWithoutLocalSnapshots and IgnoreActivatedSnapshotsto be simultaneously enabled. IgnoreActivatedSnapshots is discussedin detail in the previous section in this chapter,“IgnoreActivatedSnapshots” on page 175. IgnoreActivatedSnapshotsallows the user to prepare the recovery devices manually instead ofhaving the SRA do it. In the case of



TestFailoverWithoutLocalSnapshots, this means the user can split theRDF links before the test failover operation begins and, if desired,resume the replication before the SRM cleanup operation.

When both of these options are enabled, the SRA will examine the R2devices to ensure that they have been split beforehand. If the deviceshave been split, then the test failover operation will skip the RDF splitoperation and move on to recovering the devices. If only some of thedevices in the recovery plan are split and other R2 devices are stillwrite disabled and the RDF pair states are still Synchronized orConsistent, the SRA will skip over the devices that are split and onlysplit the ones that have not been split.

In the case of two-site RDF configurations or three-site non-Starconfigurations the RDF command “symrdf split” is the only requiredoperation on the relevant RDF pairs. In the case of SRDF/Star theoperation is slightly different. For SRDF/Star, the SRA expects thestate of the Star configuration to be disabled and that the target site bein the “isolated” state. Refer to the previous relevant section for agiven SRDF configuration for the particular required states. The stepsissued by the SRDF SRA should be exactly followed by the user.

After the test failover has been completed and before the cleanupoperation is initiated, the user can either resume replicationthemselves or let the SRA resume replication automatically duringcleanup. It is generally recommended to let the SRA resumereplication as it will ensure all of the proper steps are followed forcleaning up the environment. If manual resynchronization is desiredbefore the cleanup, two-site RDF pairs or three-site non-Starconfigurations should be returned to the SyncInProg, Synchronized,or Consistent state. SRDF/Star requires that the target site bedisconnected, reconnected, protected and Star re-enabled so that Staris back in the overall protected state.

Protection during test failover with R2 devicesBefore executing a test failover withTestFailoverWithoutLocalSnapshots, EMC recommends additionalprotection to guard against the possibility of a protected site failureduring a test failover operation. Since the SRDF link is split, remotereplication is halted for the entirety of the test and the R1/productiondata is no longer protected. This is due to the fact that changesincurred on the R2 device during the test failover operation willinvalidate the data on it as a usable copy of the R1 production data.


204


The longer the test is run, the possibility for protected site failureincreases. For this reason, EMC recommends configuring additionalprotection of R1 devices that are being tested with the advancedoption TestFailoverWithoutLocalSnapshots enabled.

Users should manually make a TimeFinder copy (Clone or Snap) ofthe R2 device to maintain a valid copy of the R1 data on the remoteSymmetrix array during a test failover procedure. This method doeshave an inherent weakness though; the gold copy will not protect anychanges made to the R1 device after the gold copy has beensplit/activated. However, it will still provide a valid point-in-timecopy of the R1 device in case of a site failure of the protected siteduring the test failover.

Figure 101 Creating a gold copy before test failover

To minimize differences between the gold copy and the R1, the goldcopy should be split/activated (term varies depending on the type ofTimeFinder technology used) immediately prior to executing the testfailover operation as shown in Figure 101.

Currently, there is no mechanism within the SRA or VSI toautomatically create a TimeFinder gold copy on the R2 (recovery site)during a test failover. Therefore, users must manually create the copy



using Symmetrix management applications such as Unisphere orSolutions Enabler command line utilities. For information on creatinga TimeFinder copy please refer to the TimeFinder Product Guideavailable at support.emc.com.

While this option does at least offer some level of protection againstfailure, it will be insufficient for applications that mandate anaggressive RPO. It is recommended in those situations to use thetraditional TimeFinder method for test failover.


206


Testing recovery plans using GNS-enabled groupsGroup name services (GNS) provides a common repository to storeand maintain SYMAPI device group (DG) and composite group (CG)definitions across Symmetrix arrays that are visible to all locallyattached hosts. By default, with GNS disabled, group definitions arestored in the local SYMAPI configuration database file on the hostthat created the group. Enabling GNS enables group definitions to bestored on the Symmetrix array in a shared GNS repository. Thisshared GNS repository is visible to any GNS-enabled locally-attachedhost, enabling these hosts to perform control operations, regardless ofwhich host initially defined the group. In addition, if one host goesdown, you can still perform SYMCLI control operation from anotherlocal host in your Symmetrix environment.

In the GNS-enabled environment, each host performing managementoperations must run an instance of the GNS daemon (storgnsd). Inthis case, the Solutions Enabler SYMAPI and SYMCLI do not directlyaccess the GNS shared repository. Instead, requests are forwarded tothe GNS daemon, which processes all GNS operations. This daemonis the only entity that directly accesses the GNS shared repository andis responsible for ensuring that each host has access to the mostcurrent GNS definitions

From each host, a GNS daemon listens for GNS requests from localclients (same host) and carries them out on the locally attachedSymmetrix array. In addition, the GNS daemon monitors the GNSrepositories on all locally-attached Symmetrix arrays, at auser-configured polling interval, for changes made to the shared GNSrepository by other daemons (on other hosts). When a change isidentified, the GNS daemon will update the host to ensure that allGNS-enabled hosts refer to the same group definitions.

In non-GNS enabled environments, the EMC SRDF SRA deletes andrecreates device group definitions as part of the test failover orfailover functionality. The SRM passes a set of devices to SRA thenthe SRA deletes and recreates the device group definitions if any ofthose devices already exist in device group definitions. The SRA doesthis to make sure the devices are not part of multiple groups or if onlya subset of devices are part of the existing device groups.

In GNS enabled environments, since the device group definitions aremanaged by GNS service the SRA does not delete and recreate thegroup, but instead only adds the target TimeFinder devices to the



existing group. The SRA includes the ability to detect whether or nota target device group is managed by GNS or not and will report assuch in the SRDF SRA log:

[11/29 14:11:56 10096 0599 SymmUtil::IsGnsGroupDevice]Checking if Symm Dev [013D], Symm [000192603603] ispart of any GNS enabled DG

[11/29 14:11:56 10096 0257SraDevicePairings::GetCopyInfo] [WARNING]: The SymmDev [013D], Symm [000192603603] is part of a GNSenabled DG

If the respective R2 devices are included in a GNS-enabled group theSRA test failover behavior will change automatically with noadditional configuration from the user. All other operations andrequirements remain unchanged as the user must still fully configurethe options file with correct device pairings and TimeFinder methodusing VSI Symmetrix SRA Utilities before executing a test failoveroperation.

Mirroring remotedevice groupdefinitions

The option gns_remote_mirror in the GNS daemon’s options filedetermines whether GNS should attempt to remotely mirror a devicegroup or a composite group RDF definition contained in the sharedGNS repository for a given Symmetrix array.

When enabled, GNS will maintain a remote mirrored groupdefinition with the same name as the local one creating a usablegroup to hosts (including GNS daemons) directly connected to theremote Symmetrix array(s). The remote mirrored group has the samename as the local one, and has a mirror image of its contents; in otherwords, the data reflects the perspective of the local array. This done toensure that the mirrored group is a legal, usable group to hostsdirectly connected to the remote Symmetrix arrays. This would meanif a user created a RDF1 group on the protected site SYMAPI server acorresponding RDF2 group would be created within GNSautomatically (by default no longer than 60 seconds after the RDF1group creation) and be available for use on the recovery site SYMAPIserver.

A device group that has been created by this mirroring mechanism isflagged internally by GNS as being a mirror. By default, thesemirrored groups are read-only and cannot be modified or renamedby administrators. Only GNS is permitted to continue to update themas the local groups upon which they are based are changed.

Testing recovery plans using GNS-enabled groups 207

208


Consequently, GNS Remote Mirroring alters how the SRA musthandle adding replica devices to target RDF2 groups for test failover.When the remote mirroring option is set and the RDF2 group thatcontains the devices in the protection group(s) is in fact a remotemirror the SRA cannot add the TimeFinder replica devices to theRDF2 group in the typical fashion. The SRA must instead add them tothe RDF1 group as remote target devices. The GNS daemon willrecognize this change and will propagate the remote targets to theremote group. In the remote group, the devices that were labeled asremote targets in the RDF1 group will appropriately become localtargets in the RDF1 group. This will allow the TimeFinder operationsexecuted by the SRDF SRA to succeed while adhering to GNS remotemirroring rules.

This behavior is newly supported with the SRDF SRA 5.1. As anexample, during test failover, the SRA will ascertain whether or notthe target local device/composite group is GNS enabled and, mostimportantly, a remote mirror. Figure 102 shows a printout of a devicegroup that is remotely mirrored in GNS.

Figure 102 GNS Remote Mirror group

During the test failover operation the SRA will log the followingmessages (among others) when detecting a GNS Remotely Mirroredgroup:

SymmUtil::IsGnsGroupDevice] The DG [SRDFS] is beingmanaged by GNS. The DG has flags: 0x1800

SymmUtil::IsGnsMirrorGroupDevice] The DG [SRDFS] ismirrored GNS group. The DG has flags: 0x1800

When the device/composite group on the recovery side isdetermined to be a remote mirror, the SRDF SRA will automaticallyalter its behavior for this group and add the replica devices as remotetargets to the RDF1 group on the protection-side SYMAPI server.



It is important to note though that if a mirrored group is directlymodified, the connection between it and the local group, on which itwas based, is broken. At that point, it is no longer a mirror andchanges to its base group (the RDF1 group in this case) may no longerbe propagated to it. This may defeat the purpose for originallydeciding to use GNS remote mirroring. Consequently, it is notrecommended to use mirrored groups with the SRA. Rather, as in allcases, use the VSI Symmetrix SRA Utilities to create the groups. Evenwith mirroring enabled on the target SYMAPI servers, mirroredgroups will not be created but instead standalone RDF1 and RDF2groups will be created (the groups will still be managed by GNSthough).

The process to configure Group Name Services is beyond the scope ofthis book, but for more information please refer to the SolutionsEnabler Array Management Product Guide on support.emc.com orUnisphere Online Help for further details.

Testing recovery plans using GNS-enabled groups 209

210


Test Failover with non-VMware devicesIn most environments, not all applications have been virtualized.Consequently, there are environments that have a mixture of virtualand physical servers, many of which have dependencies across theirhosted applications regardless of the fact that they have differentunderlying server architectures. Consequently, during a disasterrecovery test, enterprise consistent copies of application data acrossphysical servers and virtual machines may be required in order forthe test to be a true success. SRM does not have a built in mechanismfor including replicated devices that are not presented to the VMwareenvironments. SRM will exclude them in datastore groups if they donot host VMFS volumes or are in use as Raw Device Mappings.Nevertheless, there is a way to bypass SRM to have the SRA controldevices that are not presented to or in use by the VMwareenvironment. This way, during test failover, the SRA can create andsplit TimeFinder copies of R2 devices that are not specified explicitlyin the protection group. In order for this to work there are a fewrequirements:

◆ The replication mode of the VMware and non-VMware devicesmust be the same. In other words, if the VMware devices arereplicated in a Synchronous fashion, so must the non-VMwaredevices.

◆ If the devices are configured in a SRDF/Star topology, thenon-VMware devices and the VMware devices must be in thesame SRDF/Star group.

◆ The R1 non-VMware devices must manually be added to theRDF1 device/composite group on the local SYMAPI server andthe R2 devices must be added to the RDF2 device/compositegroup on the remote SYMAPI server1.

◆ The non-VMware R1 and R2 devices must reside on the samerespective arrays as the R1 and R2 devices that are presented tothe VMware environment in the target protection group.

1. If the user recreates the groups with VSI at a later time, the non-VMwaredevices will not be included and will need to be re-added manually.



The first step of the process is to configure the test failover optionsfile. This should be accomplished with VSI just as described in theearlier sections of this chapter for whatever TimeFinder technologiesare used in the environment. Refer to those sections for instructionsand details on how to use VSI.

In this example, R2 device number 28C will be configured to bepaired with a TimeFinder/Clone target device 222. This can be seenin Figure 103.

Figure 103 Configuring VMware R2 device pairs with VSI for test failover

Test Failover with non-VMware devices 211

212


The resulting options file that is created is shown in Figure 104. Ascan be seen, a single device pair for the R2 (device 28C) andTimeFinder target device (device 222) is configured for use withTimeFinder/Clone.

Figure 104 Test failover configuration file

Note: If the option TestFailoverWithoutLocalSnapshots is enabled the testfailover configuration file does not need to be populated with the VMware ornon-VMware devices. Nevertheless, the R2 devices do need to be in theremote device/composite group.

Since the VSI Symmetrix SRA Utilities does not allow users toconfigure pairings for devices not presented to the VMwareenvironment, they must be manually entered into the test failoverconfiguration file through the use of a text editor. For this example, anexternal SRDF device pair is required for proper functioning of theVMware environment during production (for example a physicaldatabase that has not yet been virtualized) and therefore a consistentcopy must be present during the test as well. The R2 device of thispair is Symmetrix device 28D. Using the details of this device the usermust manually find a valid TimeFinder target device or create a newone. Once the device has been identified or created, that devicenumber must be added along with the non-VMware R2 to the test



failover configuration file. In this example, device 223 is a valid andavailable TimeFinder/Clone target and will be paired with the R2device 28D. Using a text editor this pair will be added within the<DeviceList> stanza into a new <DevicePair> stanza. The syntaxshould be identical to the pairing configured by VSI.

Figure 105 shows the test failover options file after the non-VMwarepair has been added manually. Contrast this file with the previousversion shown in Figure 104.

Figure 105 Test failover configuration file with manually added device pair

For every non-VMware device, a new <DevicePair> stanza should becreated and populated. Once the configuration file is complete itshould be saved. At this point, the test recovery can be executednormally. The SRA will add the TimeFinder target devices inconfiguration file to the device/composite group and perform thenecessary TimeFinder operations to allow the targets to becomeuseable to the host.

Note: If devices are present in the R2 device/composite group and the userdoes not wish for the SRA to create TimeFinder copies of them, the devicesmust be removed from the group. If they remain in the group and they arenot listed in the options file, the test failover operation will fail.

It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are read/write enabled.Since they are not in the VMware environment, the SRA has no

Test Failover with non-VMware devices 213

214


method of presenting them to a host. Therefore, the user mustperform whatever functions are necessary for the non-VMware hoststo be able to use the TimeFinder copies that the SRA created. If thevirtual machines depend on the non-VMware applications to berunning before they can power on, it would be advisable to add apause into the recovery plan before SRM powers on the virtualmachines so steps can be taken by the administrator to prepare theapplications external to the VMware environment.


4


◆ Introduction ...................................................................................... 216◆ Gold Copy considerations and requirements .............................. 217◆ Configuring recovery site gold copies .......................................... 228◆ Configuring protected site gold copies......................................... 240

Gold Copy ProtectionDuring Failover

Gold Copy Protection During Failover 215

216

Gold Copy Protection During Failover

IntroductionDuring the execution of a recovery plan, the EMC SRDF Adaptersupports the creation of gold copies of the data on the recovery siteand/or protected site before permitting the recovery VMware ESXihosts to access the replicated data on the R2. The creation of goldcopies provides users with dependent-write consistent copies of thedata. This allows the ability to manually revert to these copies in caseof any problems during the execution of recovery plans.

This chapter will discuss the following topics:

◆ Gold copy considerations and requirements

◆ Configuration of recovery site gold copies

◆ Configuration of protected site gold copies



Gold Copy considerations and requirementsGold copies, intrinsically, are no different than the replicas used fortest failover. They have, for the most part, the same requirements andare created in a similar way. The main differences revolve aroundwhen they are created and how they are used. For this reason, thesame devices can be used for test failover and then re-purposed forgold copy replicas during a recovery plan. Different devices can ofcourse also be dedicated for gold copies if desired. For a descriptionof the gold copy creation workflow, refer to Figure 106.

Similar to test failover pairings, the gold copy pairings are saved inXML files. There are two separate files for gold copy pairings for therecovery and protected site and they are respectively:

◆ EmcSrdfSraRecoverySiteGoldcopyConfig.xml--For recovery sidegold copy creation, users should edit the instance of this file onthe recovery SRM server (R2 side).

◆ EmcSrdfSraProtectionSiteGoldcopyConfig.xml--For protectionside gold copy creation, users should edit the instance of this fileon the protection SRM server (R2 side).

Gold Copy considerations and requirements 217

218


Figure 106 Gold copy creation workflow



An example of a gold copy XML file is shown in Figure 107. It can beseen by this figure that the XML file is designed almost identical tothe test failover pairings configuration file.

Figure 107 Gold copy configuration options file

The XML file requires the following information:

◆ CopyInfo—Information for each gold copy operation.

◆ ArrayID—This option takes the array ID. Users can specifymultiple CopyInfo blocks within RecoverySiteGoldcopyInfo (orProtectedSiteGoldcopyInfo as the case may be) with differentarray IDs.

◆ CopyType—This option is used to select the type of replicationtechnology for gold copy creation operation. Possible values areVSE, CLONE and SNAP.

◆ SavePoolName—If the CopyType selected is SNAP, it is requiredto input save pool name. This is required to create a snap session.

◆ DeviceList—To perform a gold copy operation, it is required toprovide the device pair information. Each device pair representssource and target device pairs. For the recovery site gold copy


220


file, the source device is the R2 device on the recovery site. Forthe protected site gold copy file, the source device is the R1device on the protected site.

There are some important points to remember when configuring theoptions file(s):

◆ There can be multiple <CopyInfo> blocks each with differentarray IDs. This is the most common scenario.

◆ Each <CopyInfo> block should contain every device contained inthe corresponding protection group.

◆ There can be multiple <CopyInfo> blocks with the same array ID.However, any two <CopyInfo> blocks should not contain thedevices from the same protection group. In such case, the adapterpicks the first <CopyInfo> block defined in the options file.

◆ Having multiple <CopyInfo> blocks for the same array id withdifferent sets of devices and different copy type is supported bythe adapter.

◆ If two different <CopyInfo> blocks have the same array ID and ifa subset of the devices present in a protection group is present inthe first <CopyInfo> block and if the remaining set of devices iscontained in the other <CopyInfo> block, the adapter doesn'tcombine them and instead the adapter fails the request becausethe adapter expects all of the devices present in a protectiongroup to be contained in a single <CopyInfo> block.

Furthermore, most requirements for valid gold copy replicas areidentical to test failover:

◆ Devices must be of proper size and configuration to be validreplicas.

◆ For SRDF/A configurations in which recovery site gold copiesthat utilize TimeFinder/Snap or VP Snap, device write pacingmust be enabled on the R1 or R21. Information on how this isenabled can be found on page 239.

◆ Thin devices must be bound to a thin pool to be viewed as validcandidate replicas by VSI.

◆ TimeFinder/VP Snap must use devices of type “TDEV” or“BCV+TDEV” and TimeFinder/Snap must uses devices of type“VDEV”.

◆ Clone targets cannot be RDF devices



It should be noted that there are indeed a few important differencesbetween gold copy requirements and test failover replicarequirements:

◆ For test failover replicas, the target devices must be masked to theESX hosts. For gold copy replicas, this is not a requirement. VSIoffers all valid devices regardless of whether or not they aremasked to the ESX hosts.

◆ Consistency protection is not required for gold copy creationunlike test failover. Test failover performs a "consistent" split oractivate whereas gold copy doesn't need to. For example, if theRDF link is in "Transmit Idle" state, the adapter cannot perform aconsistent activate on clones and snaps. Therefore test failoverfails whereas gold copy operation detects this scenario andperforms a normal split.

Advanced option: FailoverIfGoldCopyFailsBy default, the SRA allows the continuation of the failover processeven if the attempt to create a gold copy of the replicated data fails. Ifthe creation of a gold copy of the replicated data is a criticalrequirement, the value of the option, FailoverIfGoldCopyFails,should be changed to “No” before the execution of the recovery plan.This option is located in the EmcSrdfSraGlobalOptions.xml file andcan be edited using the VSI Symmetrix SRA Utilities as shown inFigure 108. This option should be edited on the recovery site SRMserver to control the recovery side gold copies and on the protectedsite SRM server to control the protected site gold copies.


222


Figure 108 Altering gold copy behavior with VSI Symmetrix SRA Utilities

Advanced option: CheckForVirtualDisksIntroduced in version 5.1 of the SRDF SRA is an additional securitycheck (disabled by default) during test failover and gold copyoperations. The CheckForVirtualDisks advanced option will querythe input candidate TimeFinder replica devices for whether or notthey are currently in use as an RDM or VMFS. If any of the inputdevices are detected to be one of the two, the SRDF SRA will fail thegold copy operation and no gold copies will be activated. This willprevent an accidental overwrite of RDMs or VMFS volumes by theTimeFinder gold copy operation.

Note: The SRDF SRA will only check for the use of the device in the recoveryside vCenter associated with the recovery side SRM server. If the device is inuse in a separate environment, VMware or otherwise, the SRA will be unableto detect that and that data will be overwritten.

This option:

◆ Will mark any device containing a VMFS volume as in-use.

◆ Will mark devices as in-use if they are a raw device mapping(RDM). The virtual machine(s) using the RDM do not have to becurrently powered on for the device to be recognized as in-use.



◆ If a virtual machine has raw device mappings and that virtualmachine is not currently registered to the vCenter server, theSRDF SRA will not be able to identify that device as in use as anRDM and it will be seen as a valid candidate device for a goldcopy.

This option can be enabled using the VSI Symmetrix SRA Utilities asseen in Figure 109.

Figure 109 Enabling CheckForVirtualDisks with VSI Symmetrix SRA Utilities

While this option can be enabled via VSI in the global options file,there is, however, further configuration required to provide the SRDFSRA access to the information it requires. The SRDF SRA connects tothe local vCenter and queries the vCenter inventory to see if any ofthe input replica devices are in use. If this check is desired for bothprotected and recovery side gold copies for a given recoveryoperation, the CheckForVirtualDisks advanced option must beenabled for both the recovery and protected site SRDF SRAs.


224


It order to enable this behavior for recovery site gold copy creation,an administrator must grant recovery site vCenter access for theSolutions Enabler install local to the recovery SRM server. Similarly,in order to enable this behavior for protected site gold copy creation,an administrator must grant protected site vCenter access for theSolutions Enabler install local to the protected site SRM server.

If the credentials are missing or incorrect the SRDF SRA will not failthe gold copy process and will continue to create gold copies withoutchecking for in-use devices. Therefore, it is important to ensure thatvalid credentials are added to the SYMAPI database prior torecovery.

For example:

In this environment, the FQDN of the protected site vCenter server isch-vcenter3.ebc.emc.local. The protected vCenter server can beaccessed via a domain account “hostec”.The FQDN of the recoverysite vCenter server is ch-vcenter4.ebc.emc.local. The recoveryvCenter server can also be accessed via a domain account “hostec”1.

Note: The user account(s) must have at least Read Only permissions(assigned to the user account within vCenter) to the vCenter server objectand to the ESXi2 server objects that host (or will host in the case of therecovery side) the virtual machines using the SRDF devices to be managed bySRM.

It is important to note that this must be done on the localinstallations of Solutions Enabler on the SRM servers and not theremote SYMAPI servers. This is regardless of whether or not a remoteSYMAPI server is configured for use by the SRDF SRA controloperations.

Authorization must be added using the Solutions Enabler CLIcommand shown below (bolded portions are variables unique toeach environment):

1. As readers may note, this is the same process required for the advanceddiscovery option, FilterNonVmwareDevices. If authorizations werecreated for use with that option, no further configuration is required andthe following process can be skipped.




symcfg add authorization -host <FQDN of vCenterserver1> -username <username> -password <password>-vmware -namespace “vmware/vc”

In order to run Solutions Enabler commands from the SRM serverWindows Command Prompt, the user must either be issuingcommands from the C:\Program Files (x86)\EMC\SYMCLI\bin\directory or have the local Windows “Path” variable configured withthis location to automatically search this location when SYMCLIcommands are issued. Furthermore, the environmental variable“SYMCLI_CONNECT_TYPE” on the SRM server must either be setto LOCAL or unset. This will ensure that the authorizationcommands are issued to the local installation of Solutions Enablerand not to a remote SYMAPI host.

If protected site gold copies are configured for use, an authorizationmust be added for the protected site SYMAPI server. Anadministrator must log in to the protected site SRM server and runthe following command:


If recovery site gold copies are configured for use, an authorizationmust be added for the recovery site SYMAPI server. An administratormust log in to the recovery site SRM server and run the followingcommand:

symcfg add authorization -hostch-vcenter4.ebc.emc.local -username hostec-password password -vmware -namespace “vmware/vc”

Once these authorizations are added, no further configuration isrequired and the protected and/or recovery SRDF SRAs will usethese authorizations to query the vCenter servers to verify whether ornot the chosen replicas are in use.


2. Optionally the password can be omitted from the initial command andonce it has been executed a prompt will appear asking for the password.This method will mask the entered password from view for secure entry.


226


If the SRDF SRA identifies a device that is in use, the gold copyprocess will fail with the following message in the SRDF SRA log file:

SraCloneGroup::CreateCopySessions ] [ERROR]: Targetdevice [<device ID>] is already used in a Datastore oras RDM. Using it to create a snapshot will delete allthe existing data

FailoverIfGoldCopyFails and CheckForVirtualDisks concurrenceAs previously mentioned, one device that is deemed to be in-use willprevent the activation of any of the gold copy devices for a givenprotection group as this occurrence will fail the gold copy operationimmediately1. While the gold copy TimeFinder sessions are createdone at a time, the sessions are not activated until all of the gold copyreplica devices are validated and their sessions created (essentiallythe activate is an all-or-nothing operation). This is because the SRDFSRA requires that there be copies of all the R1 or R2 devices to allowfor a scenario that might require a full recovery to the original data.To properly achieve this, there needs to be a copy of all of the devicesand they need to be consistent point-in-time copies with each other.Consequently the sessions do not need to be created simultaneously,but they do need to be activated simultaneously. Therefore, if onedevice is invalid, the activate operation will not be attempted as theSRDF SRA will be unable to create gold copies for all of the devices. Ifgold copy protection is an important requirement, it is recommendedto disable the advanced option FailoverIfGoldCopyFails (enabled bydefault) on both the recovery and protected site whenCheckForVirtualDisks is enabled.

Disabling FailoverIfGoldCopyFails will then cause the recoveryoperation to fail if one of the gold copies replicas is found to beinvalid. As a result, the source/replica pairing can be reconfiguredand the recovery can be attempted again. This will ensure that fullyconsistent copies of all of the R1 or R2 devices are created andactivated during failover.

1. Note that if both recovery side and protected side gold copies areconfigured for creation, a failure on the protected side gold copy creationwill only cause the other protected side gold copies to not be created. Therecovery side gold copies will be unaffected and vice versa.



Troubleshooting CheckForVirtualDisksIf the credentials were input correctly or missing, furthertroubleshooting information on can be found in the viclient logcreated by the SRDF SRA. This log is located on the respective SRMserver in the below location:




This option could be beneficial to larger VMware environmentswhere the control of each and every device is tougher due to thehigher number of simultaneous users and/or devices.














228


Configuring recovery site gold copiesThe configuration and behavior of recovery site gold copy pairs ismanaged using the Gold Copy feature in the Virtual StorageIntegrator Symmetrix SRA Utilities.

The EMC VSI Symmetrix SRA Utilities for vSphere Clientautomatically recognizes the presence of the EMC SRDF Adapter andactivates the appropriate extensions required for supporting edits tothe EMC SRDF Adapter gold copy pairings file. Figure 110 shows theadditional tab provided by VSI when it is installed on the VMwarevCenter Site Recovery Manager at the recovery site. The tab isactivated when a cluster object or ESX(i) host object is selected in theleft pane in the Inventory view. The figure shows the relevantinformation that is displayed when a ESX(i) host is selected. A unionof all information from the ESX(i) hosts belonging to a cluster isdisplayed if the cluster object is selected instead of an ESX(i) hostobject. For this reason, it is recommended to edit the pairings from acluster object when applicable.

Figure 110 VSI functionality for defining gold copy device pairings

Figure 110 also shows that the plug-in provides a yellow status box inwhich succinct messages concerning the operation of the plug-in aredisplayed. These status boxes can easily be removed bydouble-clicking on the yellow box. The plug-in also provides a box toselect the task for device pairing configuration for creation of goldcopies during execution of the recovery plans.

Figure 111 shows an error that can be encountered by users wheninitiating the gold copy configuration process in VSI.



Figure 111 Error in VSI gold copy pairings screen

The error in Figure 111 usually occurs for the one of the followingreasons:

1. The user is logged in to the wrong SRM server. To configurepairings the user must be operating the SRA Utilities installed onthe site which is presented with the R2/R21 devices (the recoverysite).

2. The recovery site target devices (R2 or R21 as the case may be) arenot masked to the recovery hosts. VSI verifies maskinginformation and the recovery site target devices must be maskedto the recovery hosts. See section “Recovery site device maskingrequirements” on page 111 in chapter 2 for more information onmasking requirements.

3. The SYMAPI server is not configured within VSI. See the section“VSI configuration” on page 61 in chapter 2 for more instructionson doing so.

4. The local and/or remote array(s) have not been discovered by theconfigured SYMAPI server.

5. Symmetrix Authorizations and/or Symmetrix Access ControlLists are preventing VSI from having appropriate access to thearray information. Refer to Chapter 10, “Symmetrix Security withEMC SRDF Adapter,” on page 375 for more information.

6. The SYMAPI server is blocking user access to array informationdue to an enabled, but improperly configured nethost file. Referto the section, “Nethost configuration” on page 42 in chapter 2.

Configuring recovery site gold copies 229

230


In the case of bidirectional protection, SRDF relationships will befound on both sides. To configure pairings for a given set of devices,use the SRA Utilities on the recovery site for that set of devices.

The EMC VSI Symmetrix SRA Utilities offers two configurationoptions for the gold copy pairings screen. The option “Only showreplicas with sessions” offers the ability to have VSI only display goldcopy candidate replicas that are currently involved in a replicationsession with the listed R2 device. This option is disabled by defaultbut can be useful in large environments as it vastly increases howquickly the adapter displays replica candidates. If a user desires toconfigure replica candidates that are not currently involved in anactive TimeFinder replication session with a R2 they should leave thisoption unchecked. The second option called “Star devices”transforms (when enabled) the replica listing columns to show SRDFStar device information which is markedly different than two-siteinformation. These options are shown in Figure 112.

Figure 112 VSI Symmetrix SRA Utilities gold copy options

When the “Star Devices” option is unchecked, the configurationscreen will only report two-site SRDF relationships (SRDF/S orSRDF/A). Table 17 shows the columns and their descriptions fortwo-site SRDF:



Table 17 VSI Symmetrix SRA Utilities gold copy configuration panel (page 1 of2)



No


No

Group The SRDF Group number of the remotetarget (R2) device. The SRDF groupdefines the logical relationship betweenSRDF devices and SRDF directors onboth sides of the SRDF links

No

SRDF Mode The mode of the given device pair, eitherSynchronous or Asynchronous

No

SRDF State This current state of the SRDFrelationship (Consistent, SyncInProg,Synchronized, Split etc...). Possiblevalues depend on the mode of the SRDFrelationship.

No

Target Array The array ID number of the hosting theremote target (R2) device

No

Target Type The device type for the remote targetdevice. It will always be some form of anR2 device (R2+TDEV, R2+R-5 etc...).

No

Target Device The Symmetrix device number of theremote target (R2) device

No

Replica Device The Symmetrix device ID of thesuggested replica device for the given R2.This field is a dropdown list offeringpossibly many device options.

Yes


232


Figure 113 shows the configuration screen for gold copy when “StarDevices” remains unchecked.

Figure 113 VSI Symmetrix SRA Utilities gold copy configuration for two-site SRDF

When the “Star Devices” option is checked, the configuration screenwill only report SRDF/Star relationships (Concurrent or Cascaded).The VSI Symmetrix SRA Utilities observe the setting of the SRDFSRA advanced option, FailoverToAsyncSite when populating the



Replica State The TimeFinder session state for theselected replica device and the remotetarget (R2) device if a session exists.



Yes

Table 17 VSI Symmetrix SRA Utilities gold copy configuration panel (page 2 of 2)(continued)




SRDF relationship information into the columns. If the setting isdisabled (meaning recovery is intended for the Synchronous site) thesynchronous site device information will be displayed. Whenenabled the asynchronous site device information will be displayed.Table 17 shows the columns and their descriptions for SRDF/Star:

Table 18 VSI Symmetrix SRA Utilities gold copy configuration panel for SRDF/Star(page 1 of 3)



No


No

Star CG Name The name of the composite groupassociated with the Star configuration.

No

Star Mode The mode of the given Star configuration,either Cascaded or Concurrent.

No

Star State This current state of the Starconfiguration. Can either be Protected orUnprotected.

No

Target Array The array ID number of the hosting theremote target device. This will either bethe Synchronous site array or theAsynchronous site array depending onhow the SRDF SRA setting,FailoverToAsyncSite, is configured.

No


234


Target Type The device type for the remote targetdevice. It will always be some form of anR2 or R21 device (R2+TDEV, R21+R-5etc...). This will either be a type of R21 ora type of R2, depending on how theSRDF SRA setting, FailoverToAsyncSite,is configured and whether or not the Starconfiguration is Concurrent or Cascaded.Concurrent devices will always be R2’sregardless of whether recovery is at theasynchronous or synchronous site.

No

Target Device The Symmetrix device number of theremote target device This will either be aSynchronous site device or aAsynchronous site device depending onhow the SRDF SRA setting,FailoverToAsyncSite, is configured.

No

Replica Device The Symmetrix device ID of thesuggested replica device for the givenR2/R21. This field is a dropdown listoffering possibly many device options.

Yes



Replica State The TimeFinder session state for theselected replica device and the remotetarget (R2/R21) device if a session exists.



Yes

Table 18 VSI Symmetrix SRA Utilities gold copy configuration panel for SRDF/Star(page 2 of 3) (continued)




Figure 114 shows the configuration screen for gold copy when “StarDevices” is checked.

Figure 114 VSI Symmetrix SRA Utilities gold copy configuration for SRDF/Star

Workload Site Name The name assigned in the Starconfiguration for the current workloadarray.

No

Target Site Name The name assigned in the Starconfiguration for the target arrayconfigured for recovery. This will either bethe Synchronous site or theAsynchronous site depending on how theSRDF SRA setting, FailoverToAsyncSiteis configured.

No

Target Site State The state of the link to the configuredtarget site. This can be a variety of states,such as Protected, Connected,Disconnected, Halted, Isolated, etc...

No

Table 18 VSI Symmetrix SRA Utilities gold copy configuration panel for SRDF/Star(page 3 of 3) (continued)



236


IMPORTANT

The EMC VSI Symmetrix SRA Utilities version 5.4 does notsupport the configuration of gold copy pairings for non-StarConcurrent/Cascaded SRDF. For recovery site gold copyconfiguration with those SRDF configurations, users must edit thegold copy pairings XML file manually.

Figure 113 and Figure 114 show the detailed information displayedby VSI to simplify the task of defining device pairs. The figures showthe relevant information required for configuring recovery site goldcopy pairings. The type of TimeFinder technology (Clone, Snap or VPSnap) can be selected via the drop-down menu located above thepairing configuration table. The plug-in automatically filters theSRDF pairs configured on the Symmetrix storage array to displayonly those pairs that are relevant to the object selected on the leftpane of the Inventory view.

The potential device pairings displayed by VSI are determined byusing the following logic:

◆ If the options file has existing device pairs for the selectedTimeFinder technology, by default, VSI automatically displaysthe pairs that are defined in the options file and suggests theexisting pairing if they are valid.

◆ If the options file does not define any valid device pairs or if theuser chooses a TimeFinder technology that is not configured inthe options file, VSI determines the device possibilities bychecking that they:

• If the mode of replication is asynchronous is the write pacingenabled on the R1 or R21 (depending on where the recoverysite is) if the TimeFinder method selected is Snap or VP Snap.TimeFinder/Clone does not have write pacing requirements.

• Have the same configuration as the R2 (for example,metavolume configuration, size).

• Have the appropriate device type for the TimeFindertechnology in use.

• Are presented to the recovery side--replica devices must bemasked to the recovery side hosts prior to invoking the SRAUtilities. VSI filters the candidate list to display only thosedevices that are visible to the object selected in the left pane of



the Inventory view. The additional check prevents users fromdefining device pairs that would result in a failed test of arecovery plan.

VSI Symmetrix SRA Utilities is unaware of VMware/Symmetrixdevice relationship status. Therefore, it does not filter out replicadevice candidates containing VMFS volumes or ones that are in useas an RDM. So it is important to note that if a device isinadvertently selected and used in gold copy creation, any data on itwill be overwritten by the TimeFinder copy operation. Consequently,it is important to verify using VSI Storage Viewer or through othermeans that the device is not in use by the VMware environment. It isalso important to ensure that is not in use by hosts external to theVMware environment. The SRDF SRA version 5.1, however, doeshave an advanced option “CheckforVirtualDisks” that will filterVMware devices out during gold copy creation. This option can beenabled to provide further protection.

The plug-in uses three ellipses as a candidate for a replica device if itis unable to find a viable candidate for a SRDF pair1. If the type ofTimeFinder technology that is selected does not apply to a SRDF pair,for reasons discussed in the appropriate sections, the replica devicecolumn is populated with “NA”. The plug-in also provides theopportunity to change the save pool if TimeFinder/Snap technologyis selected for defining device pairs.

The recommendation given by the plug-in can be overridden byselecting a different device from the drop-down menu provided inthe Replica Device column. The Replica Device can be changed to“Ignore” if a SRDF pair is not involved in the creation of a gold copyduring the execution of a recovery plan.

The information displayed by the plug-in can be refreshed to reflectany changes in the configuration by selecting the Refresh button.Finally, after making the appropriate selections, the user can save the

1. The plug-in also displays three ellipses if there are insufficient replicadevices for creating device pairings. For example, if there are eight SRDFdevice pairs but only four possible replica devices, the plug-in assigns thefour devices to the first four SRDF device pairs (or if they are already in areplication session they will be associated with the respective R2s). Thereplica device column for the remainder of the SRDF pairs would bedenoted by three ellipses.


238


device pair definitions to the options file by selecting the SaveOptions File button next to the Refresh button. The plug-in ensureseach device pair definition is unique before saving it to the optionsfile.

VMware vCenter Site Recovery Manager version 5 and EMC SRDFAdapter version 5 support recovery plan definitions that includemultiple protection groups that involve more than one Symmetrixstorage array at the recovery site. VSI supports this feature byimplementing two things:

◆ Grouping of the information on the recovery site per targetSymmetrix when displaying device pairings in vSphere client.

◆ Accommodating the changes to the CopyInfo stanza and theaddition of the ArrayID tag in the EmcSrdfSraOptions.xml fileused by the EMC SRDF Adapter.

The change to the visual representation of the data is shown inFigure 115.

Figure 115 VSI plug-in for defining device pairings used by EMC SRDF Adapter

The configuration of the gold copies can be exported from VSI byselecting the “Export” link. Once the pairings have been configuredwithin VSI, select the “Save Options File” link to save the pairings tothe options file name EmcSrdfSraRecoverySiteGoldcopyConfig.xml.

Furthermore, the recovery site gold copy options file has beenconfigured to create TimeFinder/Clone replicas of the R2 data for useas gold copies. Figure 116 shows a list of the clones for the RDF2



device group containing the R2 devices. This figure shows thatTimeFinder/VP Snap sessions have been created and activated (notethe CopyOnWrite state).

Figure 116 Creation of a gold copy of replicated data during execution of arecovery plan

Device Write Pacing and recovery site gold copiesThe EMC SRDF Adapter for VMware vCenter Site Recovery Managersupports a variety of EMC TimeFinder modes to create gold copies.The use of TimeFinder/Snap or VP Snap with an R2 device in anSRDF/A session is only allowed when SRDF/A device-level writepacing is configured on the RA group(s) on the R1/R21 side of theSRDF replication. The process to configure write pacing is identical tothe write pacing setup process described for test failover. If devicewrite pacing has not already been setup for use in test failoveroperations, refer to the section, “SRDF/A and device-level writepacing” on page 149 in Chapter 3 to configure it for gold copy use.


240


Configuring protected site gold copiesIntroduced in SRM 5 is the ability to have the SRA create a gold copyof the R1 devices on the protected site during a failover. Thisadditional gold copy adds a further layer of protection to a VMwareenvironment during failover.

Like recovery site gold copies, the SRDF SRA supports the use of allTimeFinder technologies (Snap, VP Snap, and Clone) to create thereplica. Unlike recovery site gold copies, the ability to create the goldcopy pairings XML file for the protected site is not yet included inVSI. Therefore, if a user wants the SRDF SRA to create protected sitegold copies during failover they must manually discover validreplica devices and configure the XML options file.

IMPORTANT

There is a protected site gold copy XML file on both SRM servers.For a particular failover, edit the file on the source/R1/protectionSRM server. Even though the recovery site SRA is performing thefailover, the protected site SRA creates the protected site gold copy.

Consequently, the configuration of protected site gold copies issomewhat more complex as the onus is on the user to correctly selectthe candidate devices instead of relying on VSI to filter out incorrectones.

As previously discussed, the XML file requires five things:

◆ Symmetrix array serial number

◆ TimeFinder technology (VSE, Snap, Clone)

◆ Snap pool name only if TimeFinder/Snap is used

◆ R1 Symmetrix device numbers

◆ Target replica Symmetrix device numbers

Note: If the protected site is down during the failover or there areconnectivity issues to the array, the protected side gold copies may failregardless of proper configuration.



The following process should be followed to configure the protectedsite gold copy XML file:

1. Identify the recovery plan and included protection groups. Theuser must first decide for which recovery plan(s) they want goldcopy protection. Navigate through the SRM interface to select thedesired recovery plan and identify the protection groups. Thiscan be seen in Figure 117. Also take note of the direction of theprotection groups. In the next step, you will need to select thearray manager of the source SRM server noted in the “Direction”column. In the example in Figure 117 there is only one protectiongroup named “SRDFS” and the local or source SRM server is“ch-vcenter4.ebc.emc.local”.

Figure 117 Identifying protection groups for gold copy configuration

Configuring protected site gold copies 241

242


2. Identify Symmetrix serial number. Once the correct protectiongroups have been identified, navigate to the “Array Managers”and select the array manager from the SRM server that isindicated as the source or local SRM server in Figure 117 onpage 241. From the selected SRDF SRA array manager panel thatappears, select the “Devices” tab shown in Figure 118. The “LocalArray” listed in Figure 118 is the array ID that should be enteredinto the gold copy XML file ArrayID tag.1

Figure 118 Select array manager panel

3. Identify R1 devices. From the “Devices” screen scan down the“Protection Group” column. Every device that is in the “LocalDevice” column that is in one of the corresponding protectionsgroups identified in the previous step should be entered into itsown DevicePair stanza in the gold copy XML file. In the exampleused in Figure 119 on page 243, each device in the protectiongroup “SRDFS” has been entered into the Source tag in aDevicePair stanza under the ArrayID of 000192603603. Usually bydefault there is only one DevicePair stanza under the DeviceList

1. If the recovery plan has protection groups that use different arrays youwill need a new CopyInfo stanza for each Symmetrix array.



stanza, so the user will need to manually add a new DevicePairstanza for each R1 device. All DevicePair stanzas must beenveloped in the DeviceList stanza. In other words, for everyCopyInfo stanza there is only one ArrayID, one CopyType andone DeviceList. A DeviceList may contain many DevicePairs. ACopyInfo stanza should equal a protection group.

Figure 119 Identifying R1 devices and adding them to the gold copy config XML


244


4. Choose TimeFinder method. Once all of the R1 devices havebeen entered into the options file as well as the array ID, aTimeFinder method must be chosen for the gold copies. This isindicated in the CopyType line by entering either “VSE” forTimeFinder/VP Snap, “Clone” for TimeFinder/Clone and“Snap” for TimeFinder/Snap.

5. Identify replica candidate(s). Once a TimeFinder method hasbeen decided upon, suitable replica devices must be identified (orcreated) and added to the options file. Gold copy devices, unliketest failover replicas, do not have the requirement of beingpresented to hosts. Therefore the pool of devices that can be usedmay be larger. If users just want to use devices that are presented,they can use the EMC VSI Storage Viewer to identify devices thatare presented to the VMware environment. By selecting on acluster object in the vCenter hierarchy in the protected sitevCenter the EMC VSI tab appears on the right as seen inFigure 120. From there, select the “LUNs” link to see the list ofSymmetrix devices presented and their details.

Figure 120 Identifying protected site gold copy candidates with VSI StorageViewer

If the user does not want to use a device presented to the VMwareenvironment for the gold copy replica or there are no validcandidates, they can create their own or find an unpresentedpre-existing device.



When using pre-existing devices it is important to follow thesetwo guidelines:

• Ensure the candidate replica device is not involved in anyother TimeFinder (or another replication technology) session.

• It is HIGHLY recommended that the device not be masked toany other environments. If it is, there is a significant chancethat the device contains data which will be overwritten by theTimeFinder gold copy operation leading to data loss.Therefore, ensure the device does not belong to another hostby examining masking information.

• Enable CheckForVirtualDisks on the protected side SRDFSRA. This will enable the SRA to make sure the chosen replicais not in use in the protected VMware environment.

6. Enter replica Symmetrix device numbers into options file. Oncethe replicas have been identified, enter each device into a Targetline in a DevicePair stanza until each source R1 device has a validpair as shown in Figure 121. Save the file and initiate the failover.

Figure 121 Completed protection gold copy XML configuration file


246



5


◆ Introduction ...................................................................................... 248◆ Recovery ............................................................................................ 249◆ Failback.............................................................................................. 273◆ Recovery with non-VMware devices ............................................ 274

Recovery Operationswith Two-site SRDF

Recovery Operations with Two-site SRDF 247

248

Recovery Operations with Two-site SRDF

IntroductionThe cornerstone of VMware vCenter Site Recovery Manager is theability to failover a virtual infrastructure from one datacenter toanother. Understanding how this process works with the Symmetrixstorage array and Symmetrix Remote Data Facility (SRDF) is anintegral part of datacenter migration and disaster recovery.

The SRDF SRA supports two modes of SRDF replication; three-siteand two-site replication. This chapter will deal with recovery usingtwo-site replication and the subsequent chapter will discuss failoverwith three-site SRDF replication.

Specifically, this chapter will discuss the following topics as theypertain to two-site SRDF configurations:

◆ Planned Migration

◆ Disaster recovery

• Failure of the compute environment

• Complete failure of protected site

◆ Reprotection and Failback

◆ Recovery with non-VMware devices



RecoveryPrior to Site Recovery Manager 5, workflow capabilities includedboth the testing and execution of a recovery plan. With version 5,VMware has enhanced these capabilities by introducing a newworkflow designed to deliver migration from a protected site to arecovery site through execution of a planned migration workflow.Planned migration ensures an orderly and pre-tested transition froma protected site to a recovery site while minimizing the risk of dataloss.

With Site Recovery Manager 5, all recovery plans, whether they arefor migration or recovery, run as part of a planned workflow thatensures that systems are properly shut down and that data issynchronized with the recovery site prior to migration of theworkloads. This ensures that systems are properly quiesced and thatall data changes have been completely replicated prior to starting thevirtual machines at the recovery site. If, however, an error isencountered during the recovery plan execution, planned migrationwill stop the workflow, providing an opportunity to fix the problemthat caused the error before attempting to continue.

As previously noted, all recovery plans in Site Recovery Manager 5now include an initial attempt to synchronize data between theprotection and recovery sites, even during a disaster recoveryscenario. During a disaster recovery event, an initial attempt will bemade to shut down the protection group’s virtual machines andestablish a final synchronization between sites. This is designed toensure that virtual machines are static and quiescent before runningthe recovery plan, to minimize data loss where possible during adisaster. If the protected site is no longer available, the recovery planwill continue to execute and will run to completion even if errors areencountered. This new attribute minimizes the possibility of data losswhile still enabling disaster recovery to continue, balancing therequirement for virtual machine consistency with the ability toachieve aggressive recovery-point/time objectives.

Figure 122 shows the recovery plan execution wizard.

Recovery 249

250


Figure 122 Recovery plan execution wizard

Planned Migration—Two-site replicationThe recovery option “Planned Migration” assures a gracefulmigration of virtual machines from a local vCenter to a remotevCenter. Any errors that the recovery plan encounters willimmediately fail the operation and require the user to remediatethese errors and restart the migration process. Therefore, a PlannedMigration assumes the following things (among other minor details):

◆ The protected and recovery VMware environments are up andrunning (including ESX hosts, vCenter, virtual machines, SRMserver etc.) without issues.

◆ The storage environment is stable and configured properly. Thisincludes the array(s), the fabric (SAN) and the SYMAPI serversconfigured in the array managers.

◆ No network connectivity issues.

Before executing a recovery plan failover, it is highly recommendedto test the recovery plan first (preferably multiple times) using the“Test” feature offered by SRM. Information on configuring and



running a recovery test is discussed in detail in Chapter 3, “TestingRecovery Plans.”

The first step is to ensure that RDF Pair is in a proper state. In order torun a successful recovery the RDF Pair state must be either,“Synchronized”, “SyncInProg”, “Consistent”, or “Suspended”. If theRDF pair is not in one of these states, it must either be changed to aproper state using Symmetrix management applications or a DisasterRecovery operation may need to be run to allow the SRA to ignoreinvalid RDF pair states. Generally, a good indicator of valid RDF pairstatus is shown in the “Devices” tab in a given array manager. If the“Direction” column shows a blue directional arrow, the RDF PairState is valid for Planned Migration. If the “Direction” column showsa broken gray bar either manual intervention is required or theDisaster Recovery option needs to be selected. An example of bothicons can be seen in Figure 123.

Figure 123 Checking replication status in SRM

While the “Direction” column is, in general, a good indicator of RDFpair states, it is inadequate to cover the many diverse possibilities ofRDF pair states. Therefore, it is advisable to use Unisphere for VMAXor Solutions Enabler to determine the exact status. Figure 124 showsan example of a device group containing four SRDF/A devices whichall are “Consistent”.

Recovery 251

252


Figure 124 Querying for SRDF status with Unisphere for VMAX

At this point, a planned migration can be initiated by selecting theappropriate recovery plan and selecting the “Recovery” link as seenin Figure 125.

Figure 125 Initiating a planned migration with SRM

Once the Recovery link has been selected, a short confirmationwizard appears asking to confirm the initiation of the recoveryoperation and in which mode the recovery plan should be run. Thisscreen is shown in Figure 126.



Figure 126 Recovery operation confirmation wizard

As soon as the wizard completes the recovery operation willcommence. When a recovery plans reaches the “Change RecoverySite Storage to Writeable” step, as shown in Figure 127, the SRAperforms an RDF failover operation on the devices in the protectiongroup.

Recovery 253

254


Re

Figure 127 Steps of a recovery plan in VMware Site Recovery Manager

In situations where the link is still in “SyncInProg” the SRA will waitfor the remote devices to fully synchronize. They will need to reachstates of either “Consistent” or “Synchronized” depending onwhether or not they are SRDF/A or SRDF/S respectively. As soon asthe RDF pairs synchronize, the RDF failover operation will be run.Failover will behave similarly when the RDF Pair state is“Suspended”. The SRDF SRA will detect this state and perform an“RDF Resume” to resume replication. The adapter will intermittentlyquery the state and wait until the pairs synchronize. In both of thesesituations (depending on the amount of data that needs tosynchronize) this process could take a long time to complete and mayexceed the default timeout for storage operations of five minutes.Therefore, it is advisable to ensure the storage is synchronized beforea planned migration is attempted. The other option is to raise thestorage operation timeout setting to a sufficiently high enough valueto allow for a full synchronization. If increasing the timeout ispreferred, the setting needs to be changed on the recovery site SRMserver.



When the RDF failover is issued, the RDF pair state changes from“Synchronized” (if SRDF/S) or “Consistent” (if SRDF/A) to “FailedOver”. A “before and after” example of this changed state forSRDF/S devices is shown in Figure 128 using Unisphere for VMAX.

The RDF Failover operation includes the following steps that areautomatically executed in a planned migration:

1. SRDF links are suspended.

2. Source (R1) devices are write disabled to protected site ESX(i)servers.

3. The target (R2) devices are read/write enabled to recovery siteESX(i) servers.

Figure 128 RDF pair states as seen from the recovery site before and after failover

Once the R2 devices are write-enabled, the devices can be mountedand the virtual machines can be registered and powered-on. Inaddition to the R2 devices being mounted on the recovery-side ESXhosts, the R1 volumes will be unmounted and detached from theprotection-side ESX hosts.

Recovery 255

256


When the VMFS volumes on the R2 devices are mounted, the ESXkernel must resignature the VMFS first because it is seen as a copydue to its invalid signature. The reason for the invalid signature, andtherefore the subsequent resignaturing, is due to the fact that the R1and R2 devices have different world wide names (WWNs) but anidentical VMFS volume signature. The VMFS volume was originallycreated on the R1 device and the signature of the VMFS is based, inpart, on the WWN of the underlying device. Since the WWN changesbetween the R1 and the R2 but the signature is copied over, the ESXkernel will identify a WWN/VMFS signature mismatch and require aresignature before use1.

Once mounted, the VMware kernel automatically renames VMFSvolumes that have been resignatured by adding a “SNAP-XXXXXX”prefix to the original name to denote that it is a copied file system.VMware vCenter Site Recovery Manager provides an advancedsetting (disabled by default),storageProvider.fixRecoveredDatastoreNames, that will cause thisprefix to be automatically removed during the recovery plan. Checkthis option on the recovery site to enable this behavior.

Disaster Recovery—Two-site replicationThe new recovery option “Disaster Recovery2” should be selected forrecovery when there are issues with the infrastructure that willprevent a graceful recovery of virtual machines from a local vCenterto a remote vCenter. Unlike the “Planned Migration” option, mosterrors that the recovery plan encounters will be ignored by SRM. Theonly errors that will prevent a recovery in disaster recovery mode arefailures in the recovery site infrastructure. Anything from minorerrors to a complete a failure of the protected site infrastructure willnot prevent a recovery operation run in disaster recovery mode.

If possible, the “Planned Migration” is preferable as it will morelikely allow for a clean subsequent reprotection and/or failback.Therefore, if errors are encountered an earnest attempt to remediate

1. While ESX allows for the ability to force mount VMFS volumes withinvalid signatures without resignaturing the datastore, SRM does not.

2. SRM 5.0.1 introduces an advanced Disaster Recovery option called ForcedRecovery. Forced Recovery allows SRM to recover virtual machines incases where storage arrays fail at the protected site and, as a result,protected virtual machines are unmanageable and cannot be shut down,powered off, or unregistered. This option is disabled by default.



them should be made. If these errors cannot be fixed due toequipment failure or if time is of the essence and the virtualenvironment must be recovered as quickly as possible, the “DisasterRecovery” option should be selected.

Before executing a recovery plan failover, it is highly recommendedto test the recovery plan first (preferably multiple times) using the“Test” feature offered by SRM. It should be noted that if the SRMservers are partitioned from each other over the network, testrecovery is not allowed. Information on configuring and running arecovery test was discussed in detail in Chapter 3, “Testing RecoveryPlans.”

This section will discuss disaster recovery failover in two parts:

◆ Recovery after failure of compute environment

◆ Recovery after failure of storage and compute environment

Recovery after failure of compute environmentSituations can arise where there is a failure that is limited to thecompute environment while storage and replication remains online.This section assumes that the VMware environment on the protectedsite is down and cannot be contacted over the network by therecovery site. This leads to a disconnected state for the SRM serverpairing and planned migrations will not be allowed. In this situationonly “Disaster Recovery” is offered as a valid option for failoverrecovery modes as seen in Figure 129. In this scenario, the storageenvironment remains online and replication has not been halted.

Recovery 257

258


Figure 129 Recovery options during a compute environment failure

Therefore, the RDF pair states are expected to be either:“Synchronized”, “SyncInProg”, “Consistent”, or “Suspended”1.

IMPORTANT

Readers may note that neither Planned Migration nor DisasterRecovery allows for the RDF pair state of “Split”. If the RDF pairstate is in a “Split” mode this indicates the user has split thereplicated devices themselves and that the link is still up and astorage failure has not occurred (or has been resolved). Thereforeusers must re-establish the replication manually to allow forrecovery.

1. Disaster recovery mode also tolerates RDF pair states of “Partitioned” and“TransmitIdle” but the use case for these two states is discussed in thenext section.



An example recovery plan executed during a compute environmentfailure of the protected site is shown in Figure 130.

Figure 130 Completed disaster recovery after failed compute environment

There are a few important things to note in Figure 130:

◆ The steps, “Pre-synchronize Storage” and “Synchronize Storage”fail. This is NOT due to a storage failure. For the SRDF SRA, thesesteps are non-operations as SRDF ensures that data issynchronized itself and has no need for this additional operation.These steps fail simply due to the fact that the protected sitecompute environment is down and attempts to contact theremote SRM server fail1.

1. These failed steps can be skipped by selecting the Forced Failover optionfor disaster recovery. This option was introduced in SRM 5.0.1.

Recovery 259

260


◆ “Shutdown VMs at Protected Site” and “Prepare Protected SiteVMs for Migration” fail. These are the steps where SRM attemptsto gracefully power-down and clean up the protected vCenterbefore failover. Since the site is down and this is not possible, theoperation fails.

◆ “Change Recovery Site Storage to Writeable” succeeds.

With the exception of these failures, the results of the recoveryprocess when the protected site compute environment is down is notdifferent than a normal planned migration.

In situations where the link is still in “SyncInProg” the SRA will waitfor the remote devices to fully synchronize and reach either“Consistent” or “Synchronized” (depending on whether or not theyare SRDF/A or SRDF/S respectively). As soon as the RDF pairssynchronize, the RDF failover operation will be run. Failover willbehave similarly when the RDF Pair state is “Suspended”. The SRDFSRA will detect this state and perform an “RDF Resume” to resumereplication. The adapter will intermittently query the state and waituntil the pairs synchronize. In both of these situations (depending onthe amount of data that needs to synchronize) this process could takea long time to complete and may exceed the default timeout forstorage operations of five minutes. Therefore, it is advisable to ensurethe storage is synchronized before a disaster recovery operation isattempted. The other option is to raise the storage operation timeoutsetting to a sufficiently high enough value to allow for a fullsynchronization.

When the RDF failover is issued, the RDF pair state changes from“Synchronized” (if SRDF/S) or “Consistent” (if SRDF/A) to “FailedOver”. A before and after example of this changed state for SRDF/Sdevices is shown in Figure 131 using Unisphere for VMAX.

The RDF Failover operation encapsulates the following steps that areautomatically executed in a disaster recovery when the storageremains online:

◆ Source (R1) devices are write disabled to protected site ESX(i)servers.

◆ SRDF links are suspended and change to a “Failed Over” status.

◆ The target (R2) devices are read/write enabled to recovery siteESX(i) servers.



Figure 131 RDF pair states as seen from the recovery site before and after failover

Once the R2 devices are write enabled, the devices can be mountedand the virtual machines can be registered and powered-on. Since therecovery site compute environment is down, the R1 volumes cannotbe unmounted and detached from the protection-side ESX hosts. Thisstep will be skipped.

The VMware kernel automatically renames VMFS volumes that havebeen resignatured by adding a “SNAP-XXXXXX” prefix to theoriginal name to denote that it is a copied file system. VMwarevCenter Site Recovery Manager provides an advanced setting(disabled by default), storageProvider.fixRecoveredDatastoreNames,that will cause this prefix to be automatically removed during therecovery plan.

Recovery after failure of storage and compute environmentIn a true disaster scenario, both the compute and storage mightbecome unavailable. This could be due to situations such as:

◆ Actual physical destruction of the infrastructure

◆ Loss of power

Recovery 261

262


◆ WAN failure between the two datacenters, partitioning bothnetwork and data replication traffic

In any of those cases, the recovery SRM server will not be able tocontact the protected site. This can be seen in Figure 132 which showsthe recovery site SRM server reporting that the protection site isdown.

Figure 132 Disconnected protected site SRM server

When SRDF replication halts, the RDF pairs enter either a“Partitioned” or “TransmitIdle” state. These RDF pair states are onlyallowed when the disaster recovery operation is selected. Since theentire site failed in this scenario, the array manager will not be able toupdate its information and the “Devices” screen will not show correctinformation for all device pairs. This is due to the fact that theprotected side SYMAPI server is down and the protected side arrayinformation cannot be retrieved. An example of this can be seen inFigure 133.

Figure 133 Failed device discovery due to down protected site SYMAPI server



The device group on the recovery site can be queried to verify the“Partitioned” or “TransmitIdle” state of the RDF pairs. Figure 134shows a device group that is in the “Partitioned” state. Note in thefigure that the remote Symmetrix and remote device information cannot be retrieved due to the not ready status of the RDF link. Also thelocal (R2) device is still write disabled.

Figure 134 Partitioned SRDF devices

To recover the virtual machines on the surviving site, the “Recovery”link should be selected and since the protected site is completelydown the “Disaster Recovery” mode is the only available option. TheSRA does not, in principle, act any differently if the “DisasterRecovery” mode is chosen as compared to “Planned Migration”. TheSRA will still attempt all operations as it would during a migrationand return success or failure. The difference is that SRM will not failthe recovery plan on reception of a failed operation by the SRA. In thecase of a protected site complete failure all steps involving theprotected site will accordingly fail. Figure 135 shows a recovery planthat has been executed when the protected site has experienced acomplete failure and the steps that will fail because of it.

Recovery 263

264


Figure 135 Recovery plan execution during a complete protected site failure

During the “Change Recovery Site Storage to Writeable” step theSRA will issue an RDF failover command to the SRDF devices. Thiswill cause the R2 devices to become read/write enabled to therecovery hosts. An important difference between an RDF failoverduring a complete disaster recovery and a planned migration is thatthe RDF pair state will not change to “Failed Over” after theoperation. In this scenario the RDF pair state will remain“Partitioned” until the remote storage is restored. Figure 136 shows ascreenshot of the RDF pairs from earlier in this section after the RDFfailover operation. Note the only change is that the local devices arenow read/write enabled.

Figure 136 RDF pair state after an RDF failover operation

The remainder of the recovery process is no different in result thanthe recovery processes discussed earlier in this chapter.



Reprotection with two-site SRDFAfter a recovery plan or planned migration has run, there are oftencases where the environment must continue to be protected againstfailure to ensure its resilience or to meet objectives for disasterrecovery. With Site Recovery Manager 5, reprotection is a newextension to recovery plans that enables the environment at therecovery site to establish replication and protection of theenvironment back to the original protected site.

After failover to the recovery site, selecting to reprotect theenvironment will establish synchronization and attempt to replicatedata between the protection groups now running at the recovery siteand the previously protected, primary site. This capability toreprotect an environment ensures that environments are protectedagainst failure even after a site recovery scenario. It also enablesautomated failback to a primary site following a migration orfailover.

It is important to note that an automatic reprotection by VMware SiteRecovery Manager may not always be possible depending on thecircumstances of the preceding recovery operation. Recovery plansexecuted in “Planned Migration” mode are the likeliest candidatesfor a subsequent successful automated reprotection by SRM.Exceptions to this are if certain failures or changes have occurredbetween the time the recovery plan was run and the reprotectionoperation was initiated. Those situations may cause the reprotectionto fail. Similarly, if a recovery plan was executed in “DisasterRecovery” mode any persisting failures may cause a partial orcomplete failure of a reprotection of a recovery plan.

Reprotect after Planned MigrationThe scenario that will most likely lead to a successful reprotection is areprotection after a planned migration. In the case of a plannedmigration there are no failures of either the storage or computeenvironment and therefore reversing recovery plans/protectionsgroups as well as swapping and establishing replication in thereverse direction is possible.

If failed-over virtual machines will eventually need to be returned tothe original site or if they require SRDF replication protection, it isrecommended to run a reprotect operation as soon as possible after amigration.

Reprotection with two-site SRDF 265

266


Reprotect is only available after a recovery operation has occurred,which is indicated by the recovery plan being in the “RecoveryComplete” state. A reprotect can be executed by selecting theappropriate recovery plan and selecting the “Reprotect” link asshown in Figure 137.

Figure 137 Executing a reprotect operation in VMware vCenter SRM

The reprotect operation does the following things:

◆ Reverses protection groups. The protection groups are deletedon the original protection SRM server and are recreated on theoriginal recovery SRM server. The inventory mappings areconfigured (assuming the user has pre-configured them in SRMon the recovery site) and the necessary “shadow” or“placeholder” VMs are created and registered on the newlydesignated recovery SRM server1.

1. For detailed information on the non-storage related portions of thereprotect operation refer to VMware documentation onwww.vmware.com



◆ Reverses recovery plan. The failed-over recovery plan is deletedon the original recovery SRM server and recreated with thenewly reversed protection group.

◆ Swaps personality of RDF devices. The SRDF SRA performs anRDF swap on the target RDF pairs which enables replication to beestablished back to the original site. R1 devices will become R2devices and R2 devices will become R1 devices.

◆ Re-establishes replication. After the RDF swap, the SRDF SRAincrementally re-establishes replication between the RDF pairs,but in the opposite direction from what it was before thefailover/migration.

The status of the devices after a migration, but before a reprotectoperation, can be seen in Figure 138. The top portion of the figureshows the RDF state in Unisphere for VMAX of the device groupcontaining the devices in the failed over protection group. Since theprotection group was just failed over using the planned migrationmode the RDF pair state is “Failed Over”. In the bottom half of thefigure, the EMC VSI Storage Viewer shows that the devices on therecovery site are still of type RDF2 indicating that an RDF swap hasnot yet occurred.

Figure 138 Device status before reprotect operation


268


Figure 139 shows the steps involved in a reprotect operation.

Figure 139 Reprotect operation steps

Step 1 of the reprotect operation (as shown in Figure 139) causes theSRA to issue an RDF personality swap and establish the replication inthe reverse direction.

Figure 140 shows the same information as Figure 137, except insteadit shows the device status after reprotection. The recovery site devicesare now of type RDF1 and the RDF pair state will be either“SyncInProg”, “Synchronized” or “Consistent”. It can also be notedin comparing the previously mentioned two figures that the devicegroups types have swapped as well (between RDF1 and RDF2).



Figure 140 Device status after reprotect operation

Reprotect after a temporary failureThe previous section describes the best possible scenario for a smoothreprotection because it follows a planned migration where no errorsare encountered. For recovery plans failed over in disaster recoverymode, this may not be the case.

Disaster recovery mode allows for failures ranging from the verysmall to a full site failure of the protection datacenter. If these failuresare temporary and recoverable a fully-successful reprotection may bepossible once those failures have been rectified. In this case, areprotection will behave similar to the scenario described in theprevious section. If a reprotection is run before the failures arecorrected or certain failures can not be fully recovered, an incompletereprotection operation will occur. This section describes this scenario.

For reprotect to be available, the following steps must first occur:

◆ A recovery must be executed with all steps finishing successfully.If there were any errors during the recovery, the user needs toresolve the issues that caused the errors and then rerun therecovery.


270


◆ The original site should be available and SRM servers at both sitesshould be in a connected state. An error message indicating suchcan be seen in Figure 141. If the original site cannot be restored(for example, if a physical catastrophe destroys the original site)automated reprotection cannot be run and manual recreation willbe required if and when the original protected site is rebuilt.

Figure 141 SRM servers must be connected to run a reprotect operation

If the protected site SRM server was disconnected during failover andis reconnected later, SRM will want to retry certain recoveryoperations before allowing reprotect. This typically occurs if therecovery plan was not able to connect to the protected side vCenterserver and power down the virtual machines due to networkconnectivity issues. If network connectivity is restored after therecovery plan was failed over, SRM will detect this situation andrequire the recovery plan to be re-run in order to power those VMsdown. Figure 142 shows the message SRM displays in the recoveryplan when this situation occurs.

Figure 142 Recovery plan message after SRM network partition

A reprotection operation will fail if it encounters any errors the firsttime it runs. If this is the case, the reprotect must be run a second timebut with the “Force” option selected.



Figure 143 Forcing a reprotect operation

Once the force option is selected, any errors will be acknowledgedand reported but ignored. This will allow the reprotect operation tocontinue even if the operation has experienced errors. It will attemptall of the typical steps and complete whichever ones are possible.Therefore, in certain situations, the SRDF replication may not beproperly reversed even though the recovery plan and protectiongroup(s) were. If the “Configure Storage to Reverse Direction” stepfails, manual user intervention with Unisphere for VMAX orSolutions Enabler CLI may be required to complete the process. Theuser should ensure that:

◆ An RDF swap has occurred by ensuring the replicatedtarget/source devices have changed personalities

◆ Replication has been re-established and is in an appropriate andsupported SRDF replication mode

Reprotect after a failover due to unrecoverable failureIn extreme circumstances, the storage and/or the computeenvironment may be rendered completely unrecoverable due to adisaster. In this scenario, reprotect will not be possible. Therefore theprocess of “re-protecting” the original recovery site is no differentthan the original setup of the protection groups and recovery plans


272


from scratch. Refer to Chapter 2, “Installation and Configuration,” forinstructions on that process.



FailbackAn automated failback workflow can be run to return the entireenvironment to the primary site from the secondary site. This willhappen after reprotection has ensured that data replication andsynchronization have been established to the original site.

Failback runs through the same workflow that was originally used tomigrate the environment to the recovery site. It guarantees that thecritical virtual machines encapsulated by the failed-over recoveryplan are returned to their original environment. The workflow willexecute only if reprotection has successfully completed.

Failback ensures the following:

◆ All virtual machines that were initially migrated to the recoverysite will be moved back to the primary site (assuming they stillexist).

◆ Environments that require that disaster recovery testing be donewith live environments with genuine migrations can be returnedto their initial site.

◆ Simplified recovery processes will enable a return to standardoperations after a failure.

Failback is no different in its execution or behavior than the originalfailover operation. Before failback can occur, valid protection groupsand recovery plans must be created or re-configured through areprotect operation or manual creation by the user.

Figure 144 Executing failback in VMware vCenter Site Recovery Manager 5

Since failback is in practice no different than failover, the process willnot be discussed any further in this chapter.

Failback 273

274


Recovery with non-VMware devicesIn most environments, not all applications have been virtualized forone reason or another. Consequently, there are environments thathave a mixture of virtual and physical servers, many of which havedependencies across their hosted applications regardless of the factthat they have different underlying server architectures.Consequently, during a disaster recovery operation, devices that arenot presented to the VMware environment may need to be failed overwith the VMware devices at the same time. Unfortunately, SRM doesnot have a built-in mechanism for including replicated devices thatare not presented to the VMware environments. SRM will excludethem in datastore groups if they do not either host VMFS volumes orare in use as Raw Device Mappings. Nevertheless, there is a way to“bypass” SRM to have the SRA control devices that are not presentedin the environment. In order for this to work there are a fewrequirements:


◆ The R1 non-VMware devices must manually be added to the R1device/composite group on the local SYMAPI server and the R2devices must be added to the R2 device/composite group on theremote SYMAPI server1.

◆ The non-VMware R1 and R2 devices must reside on the samerespective arrays as the R1 and R2 devices that are presented tothe VMware devices in the target protection group.

If these requirements are met, the SRDF SRA will failover both theVMware devices and the non-VMware devices during a recoveryoperation. Besides adding the devices to the device/composite groupno other special operations are required of the user besides thenormal recovery operations. If the recovery operation is going tomake use of the gold copy functionality discussed in Chapter 3,“Testing Recovery Plans,”the non-VMware devices and therespective TimeFinder target devices must also be added to the goldcopy device pairing files.




It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are RW enabled. Sincethey are not in the VMware environment, neither the SRA nor SRMhave methods of presenting them to a host. Therefore, the user mustperform whatever functions are necessary for the non-VMware hoststo be able to mount the R2 copies. If the virtual machines depend onthe non-VMware applications to be running before they can poweron, it would be advisable to add a pause or script into the recoveryplan before SRM powers on the virtual machines so steps can betaken by an administrator or script to prepare the non-VMwareapplications.

Recovery with non-VMware devices 275

276



6


◆ Introduction ...................................................................................... 278◆ Recovery ............................................................................................ 279◆ Reprotection...................................................................................... 298◆ Failback.............................................................................................. 304◆ Failover with non-VMware devices .............................................. 305

Recovery Operationswith Concurrent

SRDF/Star

Recovery Operations with Concurrent SRDF/Star 277

278

Recovery Operations with Concurrent SRDF/Star

IntroductionThe cornerstone of VMware vCenter Site Recovery Manager is theability to fail over a virtual infrastructure from one datacenter toanother. Understanding how this process works with the Symmetrixstorage array and Symmetrix Remote Data Facility (SRDF) is anextremely important part of datacenter migration or disasterrecovery.

This chapter will discuss the following topics as they pertain toConcurrent SRDF/Star configurations:

◆ Recovery with Concurrent SRDF/Star

• Planned Migration

• Disaster recovery

– Failure of the compute environment– Complete failure of protected site

◆ Reprotection

◆ Failback





With Site Recovery Manager 5, all recovery plans, whether they arefor migration or recovery, run as part of a planned workflow thatensures that systems are properly shut down and that data issynchronized with the recovery site prior to migration of theWorkloads. This ensures that systems are properly quiesced and thatall data changes have been completely replicated prior to starting thevirtual machines at the recovery site. If, however, an error isencountered during the recovery plan execution, planned migrationwill stop the workflow, providing an opportunity to fix the problemthat caused the error before attempting to continue.

As previously noted, all recovery plans in Site Recovery Manager 5now include an initial attempt to synchronize data between theprotection and recovery sites, even during a disaster recoveryscenario. During a disaster recovery event, an initial attempt will bemade to shut down the protection group’s virtual machines andestablish a final synchronization between sites. This is designed toensure that virtual machines are static and quiescent before runningthe recovery plan, to minimize data loss where possible during adisaster. If the protected site is no longer available, the recovery planwill continue to execute and will run to completion even if errors areencountered. This new attribute minimizes the possibility of data losswhile still enabling disaster recovery to continue, balancing therequirement for virtual machine consistency with the ability toachieve aggressive recovery point/time objectives.


Recovery 279

280



Concurrent SRDF/Star configurationDetailed step-by-step configuration of SRDF/Star is beyond thescope of this book but there are some important considerations tonote before configuring it for use with SRM.

The 5.0 version of the SRDF adapter only supported failover betweenthe original workload site and the synchronous site. The 5.1 release ofthe SRDF SRA introduces support for failover to the asynchronoussite.

The protected SRM server and vCenter server must be located at theappropriate SRDF/Star sites to support failover to the desired site. Inother words, the protected VMware compute environment mustalways be at the workload site and the recovery VMware computeenvironment must be at the SRDF/Star synchronous site in order tofailover to the synchronous site or at the asynchronous site in order tosupport failover to the asynchronous site. The location of thecompute environment in a SRDF/Star environment dictates wherethe SRDF SRA permits failover to for that given configuration. Once atopology has been configured, failover may only occur between the



original workload site and the configured target site, be it thesynchronous site or the asynchronous site. Whichever site does nothost one of the two SRM-protected compute environments is deemeda bunker site by the SRDF SRA and therefore cannot be managed bythe SRDF SRA.

IMPORTANT

The SRDF SRA advanced option FailoverToAsyncSite must be setto DISABLED to allow recovery to the Synchronous site orENABLED to allow recovery to the Asynchronous site. This optionmust be set identically on both the recovery and protected SRMservers.

Planned Migration—Recovery with Concurrent SRDF/StarThe recovery option “Planned Migration” assures a gracefulmigration of virtual machines from a local vCenter to a remotevCenter. Any errors that the recovery plan encounters willimmediately fail the operation and require the user to remediatethese errors and restart the migration process. Therefore, a PlannedMigration assumes the following things (among other details):

◆ The protected and recovery site VMware environment is up andrunning (including ESX hosts, vCenter, virtual machines, SRMserver etc.) without issues.



Before executing a recovery plan failover, it is highly recommendedto test the recovery plan first (preferably multiple times) using the“Test” feature offered by SRM. Information on configuring andrunning a recovery test is discussed in detail in Chapter 3, “TestingRecovery Plans.”

The first step is to ensure that SRDF/Star is in a proper state. In orderto run a successful recovery the SRDF/Star state must be “Protected”.If SRDF/Star is in an alternative state, it must either be changed to aproper state using Symmetrix management applications or a DisasterRecovery operation may need to be run to ignore an invalidSRDF/Star state.

Recovery 281

282


Generally, a good indicator of valid SRDF/Star status is shown in the“Devices” tab in a given array manager by selecting the arraymanager and view the “Devices” tab. If the “Direction” columnshows a blue directional arrow, it might be a valid candidate forPlanned Migration. This is slightly different than with two-site SRDFhowever, as three-site SRDF has additional limitations on what statethe devices can be in to allow failover which are not reflected withinthe “Devices” tab. Not only do the device pairs need to be replicating,but they need to be consistent. In SRDF/Star terminology, this meansthat both the Asynchronous site and the Synchronous site must be inthe “Protected” state and SRDF/Star must be in the overall“Protected” state. The SRA requires that SRDF/Star is “Protected” soif either site is not in the “Protected” state, SRDF/Star cannot be inthe proper state and Planned Migration will fail. The SRDF SRA logwill report errors such as those below, after a failed recovery plan dueto invalid SRDF/Star state:

[01/16 13:18:00 16212 0569SraStarGroup::IsValidStarState] STAR Group state isUnprotected.

[01/16 13:18:00 16212 0985SraStarGroup::ValidateInputDevices] [WARNING]: TheSTAR state is not valid. Exiting

[01/16 13:18:00 16212 3018PrepareFailoverCommand::RunOnGroup] [ERROR]: Inputdevice validation succeeded but one/many of theadapter's conditions is not met. Exiting withfailure

Example errors of a failed recovery plan due to an invalid SRDF/Starstate can be seen in Figure 146.



Figure 146 Failed planned migration due to invalid SRDF/Star state

If the “Direction” column shows a broken gray bar either manualintervention is required or the Disaster Recovery option should beselected. Planned migration will never be allowed in this state.

While the “Direction” column is, in general, a good indicator of RDFpair states it is inadequate to cover the many diverse possibilities ofSRDF/Star states. Therefore, it is advisable to use SymmetrixManagement Console1 (SMC) or Solutions Enabler to determine theexact status. A valid SRDF/Star group for Planned Migration isshown using Symmetrix Management Console in Figure 147. Notethat the 1st Target (synchronous site), the 2nd Target (theasynchronous site) and SRDF/Star state itself all report as“Protected”.

1. Version 1.5 of Unisphere for VMAX does not yet include SRDF/Starfunctionality. Therefore, SMC or SYMCLI must be utilized.

Recovery 283

284


Figure 147 Valid SRDF/Star state for Planned Migration in SMC

At this point, a planned migration can be initiated by selecting theappropriate recovery plan and the selecting the “Recovery” link ascan be seen in Figure 148.






As soon as the wizard completes the recovery operation willcommence. During the steps “Prepare Protected Site VMs forMigration” and “Change Recovery Site Storage To Writeable”, asshown in Figure 150, the SRDF SRA performs the necessary Staroperations on the devices in the protection group to failover.

Recovery 285

286


Re


The following two tables describe the steps for recovery withConcurrent SRDF/Star. The tables have seven columns:

1. SRDF SRA Step #: The chronological order of the operationsinitiated by the SRDF SRA during recovery.

2. Issuing SRDF SRA and SYMAPI: Due to certain Starrequirements certain operations must be issued from certainSYMAPI server (for example symstar halt must be issued bythe workload SYMAPI server). Therefore the first few operationsare issued by the protected site SRDF SRA and consequently theprotected site SYMAPI server. If an error occurs during a givenoperation refer to the specified SRDF SRA server or SYMAPIserver listed for that operation for the correct logs.

3. Step Detail: The description of the SRDF SRA operation.

4. Resulting SRDF/SRA State: The overall state of the SRDF/Starconfiguration after the listed SRDF SRA operation has completed.

5. Site A after step: The state of the original workload site after theoperation.



6. Site B after step: The state of the original synchronous target siteafter the operation.

7. Site C after step: The state of the original asynchronous target siteafter the operation.

For Concurrent SRDF/Star recovery to the Synchronous site, the SRAperforms the steps listed in Table 19 for recovery. For ConcurrentSRDF/Star recovery to the Asynchronous site, the SRA performs thesteps listed in Table 20 for recovery.

Table 19 Concurrent SRDF/Star steps for Synchronous Recovery

SRDFSRAStep #

IssuingSRDFSRA andSYMAPI Step detail

ResultingSRDF/StarState

Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite

Create protected site goldcopies

Protected Workload Protected Protected

2 Protectedsite

Halt Star Unprotected Workload Halted Halted

3 Recoverysite

Create recovery site goldcopies

Unprotected Workload Halted Halted

4 Recoverysite

Switch Workload site.The original Workloadsite is now the Sync siteand vice versa.

Unprotected Disconnected Disconnected Workload

5 Recoverysite

Connect Sync target site Unprotected Connected Disconnected Workload

6 Recoverysite

Connect Async target site Unprotected Connected Connected Workload

7 Recoverysite

Protect Sync target site Unprotected Protected Connected Workload

8 Recoverysite

Protect Async target site Unprotected Protected Protected Workload

9 Recoverysite

Enable Star Protected Protected Protected Workload

Recovery 287

288


Readers may note an important difference between theAsynchronous recovery and the Synchronous recovery--the end stateof Star is not fully protected in an Asynchronous recovery unlikeSynchronous recovery. This is due to the fact that when the workloadsite is located in the tertiary site, which is at an asynchronous distancefrom both other sites, it cannot replicate in a synchronous fashion tothe Synchronous target site as Star requires for a Protected state.Consequently the SRDF SRA can only “Connect” the Synchronoussite which leaves it in Adaptive Copy Disk mode.

Unlike two-site SRDF, the SRA will in fact reverse the replication ofdirection and enable consistency during the failover process. Fortwo-site SRDF solutions, this replication reversal is reserved for the“Reprotect” operation. Due to the requirements inherent inSRDF/Star these steps are included in the failover process.

Once the SRDF/Star switch workload process has completed, theSynchronous target site devices are now write enabled so the deviceswill be mounted and the virtual machines will be registered and

Table 20 Concurrent SRDF/Star steps for Asynchronous Recovery

SRDFSRAStep #



Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite

Created protected sitegold copies


2 Protectedsite


3 Recoverysite

Created recovery sitegold copies


4 Recoverysite

Switch Workload site.The original Workloadsite is now the Async siteand vice versa.


5 Recoverysite

Connect Async target site Unprotected Disconnected Connected Workload

6 Recoverysite

Connect Sync target site Unprotected Connected Connected Workload

7 Recoverysite

Protect Async target site Unprotected Connected Protected Workload



powered-on. The SRDF/Star switch workload process will swap theRDF personalities of the devices and change the site definitionswithin the SRDF/Star configuration. With Concurrent SRDF/Star, theR1 devices will become R2 devices. Similarly, the R2 devices (on theSynchronous site OR the Asynchronous site depending on theconfiguration) are now R1 devices.

In addition to the R1 devices being mounted on the recovery-sideESX hosts, the R2 volumes will be unmounted and detached from theprotection-side ESX hosts.

When the VMFS volumes on the (now) R1 devices are mounted theESX kernel must resignature the VMFS first because it is seen as acopy due to its invalid signature. The reason for the invalid signature,and therefore the subsequent resignaturing, is due to the fact that theR1 and R2 devices have different world wide names (WWNs) but anidentical VMFS volume. The VMFS volume was (most likely)originally created on the R1 device and the signature of the VMFS isbased, in part, on the WWN of the underlying device. Since theWWN changes between the R1 and the R2 and the signature is copiedover, the ESX kernel will identify a WWN/VMFS signature mismatchand resignature1.

The VMware kernel automatically renames VMFS volumes that havebeen resignatured by adding a “SNAP-XXXXXX” prefix to theoriginal name to denote that it is a copied file system. VMwarevCenter Site Recovery Manager provides an advanced setting(disabled by default), storageProvider.fixRecoveredDatastoreNames,that will cause this suffix to be automatically removed during therecovery plan. Check this option on the recovery site to enable thisautomatic prefix removal behavior.

Disaster RecoveryThe new recovery option “Disaster Recovery” should be selected forrecovery when there are issues with the infrastructure that willprevent a graceful recovery of virtual machines from a local vCenterto a remote vCenter. Unlike the “Planned Migration” option, mosterrors that the recovery plan encounters will be ignored by SRM. Theonly errors that will prevent a recovery in disaster recovery mode are


Recovery 289

290


failures in the recovery site infrastructure. Anything from minorerrors to complete failure of the protected site infrastructure will notprevent recovery.

If possible, the “Planned Migration” is preferable as it will morelikely allow for a clean subsequent reprotection and/or failback.Therefore, if errors are encountered an earnest attempt to remediatethem should be made. If these errors cannot be fixed (due toequipment failure or if time is of the essence and the virtualenvironment must be recovered as quickly as possible) the “DisasterRecovery” option should be selected.




Recovery after failure of compute environmentSituations can arise where there is a failure that is limited to thecompute environment while storage and replication remains online.This section assumes that the VMware environment on the protectedsite is down and cannot be contacted over the network by therecovery site. This leads to a disconnected state for the SRM serverpairing and planned migrations will not be allowed. In this situationonly “Disaster Recovery” is offered as a valid option for failoverrecovery modes.

In this scenario, the storage environment remains online andreplication has not been halted. Therefore, the SRDF/Star state isexpected to be “Protected”1. An example recovery plan executedduring a compute environment failure of the protected site is shownin Figure 152.

1. Disaster recovery mode also tolerates other SRDF/Star states, and thesestates are discussed in the ensuing section.





◆ The steps, “Pre-synchronize Storage” and “Synchronize Storage”fail. This is NOT due to a storage failure. For the SRDF SRA, thesesteps are non-operations as SRDF ensures synchronized dataitself and has no need for this outside additional operation. Thesesteps fail simply due to the fact that the protected site computeenvironment is down and attempts to contact the remote SRMserver fail.



Recovery 291

292


With the exception of these failures, the result of the recovery processwhen the protected site compute environment is down is no differentto a normal planned migration. When “Change Recovery Site Storageto Writeable” executes, the SRDF SRA follows the same workflow asshown in Table 19 on page 287 or Table 20 on page 288 (depending onwhether recovery is to the synchronous site or asynchronous siterespectively).

Recovery after failure of storage and compute environmentIn a true disaster scenario, both the protected site compute andstorage might become unavailable. This could be due to situationssuch as:


◆ Loss of power




When the link between the local and remote Symmetrix arraybecomes “Partitioned” (referred to as “PathFail” by SRDF/Star),SRDF/Star enters a “Tripped” state. Figure 153 on page 293 shows anexample Concurrent SRDF/Star environment that has been tripped



because the protected site has experienced a total failure. Since theWorkload site is down, the paths to the Synchronous site and theAsynchronous site are in a “PathFail” state.

Figure 153 Concurrent SRDF/Star in a “Tripped” state in SMC

While a variety of states are allowed for discovery of SRDF/Stardevices, only two SRDF/Star states are allowed for failover, be itDisaster Recovery mode or Planned Migration mode. The SRDF/Starstate must be either “Protected” (the only state supported by PlannedMigration) or “Tripped”. Disaster Recovery supports both states. Anyother state requires user intervention to configure it to a valid statebefore recovery.

Since the entire workload site failed in this scenario, the arraymanager will not be able to update its information and the “Devices”screen will not show correct information for all device pairs. This isdue to the fact that the protected side SYMAPI server is down and theprotected side array information cannot be retrieved. An example ofthis can be seen in Figure 154.

Recovery 293

294


Figure 154 Failed device discovery due to an off-line protected site SYMAPI server

To recover the virtual machines on the surviving site, the “Recovery”link should be selected. Since the protected site is completely downthe “Disaster Recovery” mode is the only available option. The SRAdoes not, in principle, act any differently if the “Disaster Recovery”mode is chosen as compared to “Planned Migration”. The SRA willstill attempt all operations as it would during a migration and returnsuccess or failure. The difference is that SRM will not fail the recoveryplan on reception of a failed operation by the SRA. In the case of acomplete failure of the protected site all steps involving the protectedsite will accordingly fail. Figure 155 shows a recovery plan that hasbeen executed when the protected site has experienced a completefailure and the steps that will fail because of it.




These failed steps can be skipped by selecting the Forced Failoveroption for disaster recovery. This option was introduced in SRM 5.0.1.

In a complete failure of the protected site, both the Asynchronous siteand the Synchronous site enter the “PathFail” state. Figure 156schematically depicts a Concurrent SRDF/Star environment when ithas experienced a complete protected site failure.

Figure 156 Concurrent SRDF/Star with workload site failure

Table 21 on page 296 shows the process steps taken by the SRDF SRAwhen recovering a Concurrent SRDF/Star setup to the Synchronoussite with the Workload site down.

Recovery 295

296


Table 21 Concurrent SRDF/Star recovery to Sync site workflow

Step # Step detail

SRDF/StarState afterstep Site A after step Site B after step Site C after step

1 State before failover Tripped Workload site PathFail PathFailCleanReqa

2 Check if MSC cleanup isneeded and perform a symstarcleanup on affected site.Required cleanup is denotedby the path being in the“Pathfail CleanReq” state

Tripped Workload site PathFail PathFail

3 Create and/or activate goldcopy


4 Switch Workload site (site A)to site B with “keep data” set tosite B

Unprotected Disconnected Workload site Disconnected

5 Attempt to connect site A (willfail)


6 Connect site C Unprotected Disconnected Workload site Connected

7 Attempt to protect site A (willfail)

Unprotected Disconnected Workload site Connected

8 Protect site C Unprotected Disconnected Workload site Protected

a. State may be “Pathfail” or “Pathfail CleanReq” depending on the state of MSC. The SRA will perform a cleanup if the latter isthe case.



All of these steps occur during the step entitled “Change RecoverySite Storage to Writeable”. The remainder of the recovery process isno different in result than the recovery processes discussed earlier inthis chapter.

Note: If the protected site failure is not permanent, users will have tomanually connect and protect the original Workload site and then enableSRDF/Star using SYMCLI or Symmetrix Management Console. The SRDFSRA will not re-attempt these failed operations once the recovery plan hasbeen completed.

Table 22 Concurrent SRDF/Star recovery to Async site workflow

Step # Step detailSRDF/Star Stateafter step Site A after step Site B after step Site C after step

1 Protected site (storage andcompute) goes down

Tripped Workload site PathFail PathFailCleanReqa



3 Create and activate recoveryside gold copies


4 Switch Workload site to site Cfrom site A with “keep data”set to site C

Unprotected Disconnected Disconnected Workload site

5 Attempt to connect originalworkload target site (site A)which will fail

Unprotected Disconnected Disconnected Workload site

6 Connect site B (adaptive copy) Unprotected Disconnected Connected Workload site

7 Attempt to protect site A targetsite (will fail)

Unprotected Disconnected Connected Workload site

8 Protect site B (set RDF modeto Asynchronous)

Unprotected Disconnected Protected Workload site

a. State may be “Pathfail” or “Pathfail CleanReq” depending on the state of MSC. The SRA will perform a cleanup if the latter is the case.

Recovery 297

298


ReprotectionAfter a recovery plan or planned migration has run, there are oftencases where the environment must continue to be protected againstfailure, to ensure its resilience or to meet objectives for disasterrecovery. With Site Recovery Manager 5, reprotection is a newextension to recovery plans that enables the environment at therecovery site to establish replication and protection of theenvironment back to the original protected site.




Reprotect is only available after a recovery operation has occurred,which is indicated by the recovery plan being in the “RecoveryComplete” state. A reprotect can be executed by selecting theappropriate recovery plan and selecting the “Reprotect” link as



shown in Figure 157.


The reprotect operation does the following things for ConcurrentSRDF/Star environments:

◆ Reverses protection groups. The protection groups are deletedon the original protection SRM server and are recreated on theoriginal recovery SRM server. The inventory mappings areconfigured (assuming the user has pre-configured them in SRMon the recovery site) and the necessary “shadow” or“placeholder” VMs are created and registered on the newlydesignated recovery SRM server.


Reprotection 299

300




In the case of Concurrent SRDF/Star environments, the RDF swapand the re-establish are not executed in the reprotect operation bydefault. Due to the nature and requirements of SRDF/Star, thesesteps must be included in the failover operation. Therefore, the SRAdetects Concurrent SRDF/Star configurations and these reprotectsteps become non-operation events for the SRA. If for some reasonthe Concurrent SRDF/Star environment is not in a fully-protectedmode, the SRDF SRA will connect, protect and enable Star as needed.


Disaster recovery mode allows for failures ranging from the verysmall to a full site failure of the protection datacenter. If these failuresare temporary and recoverable a fully-successful reprotection may bepossible once those failures have been rectified. In this case, areprotection will behave similar to the scenario described in theprevious section. If a reprotection is run before the failures arecorrected or certain failures can not be fully recovered, an incompletereprotection operation will occur. This section describes this scenario.




◆ A recovery must be executed with all steps finishing successfully.If there were any errors during the recovery, the user needs toresolve the issues that caused the errors and then rerun therecovery.



If the protected site SRM server was disconnected during failover andis reconnected later, SRM will want to retry certain failoveroperations before allowing reprotect. This typically occurs if therecovery plan was not able to connect to the protected side vCenterserver and power down the virtual machines due to networkconnectivity issues. If network connectivity is restored after therecovery plan was failed over, SRM will detect this situation andrequire the recovery plan to be re-run in order to power those VMsdown. Figure 160 shows the message SRM displays in the recoveryplan when this situation occurs.


Reprotection 301

302


A reprotection operation will fail if it encounters any errors the firsttime it runs. If this is the case, the reprotect must be run a second timebut with the “Force” option selected1.


Once the force option is selected, any errors will be acknowledgedand reported but ignored. This will allow the reprotect operation tocontinue even if the operation has experienced errors. It will attemptall of the typical steps and complete whichever ones are possible.Therefore, in certain situations, the SRDF replication may not beproperly reversed even though the recovery plan and protectiongroup(s) were. If the “Configure Storage to Reverse Direction” stepfails, manual user intervention may be required to complete theprocess. The SRDF SRA will attempt to connect, protect and enableStar during the reprotection operation.

If any of these steps fail during reprotection, a device discoveryoperation can be executed when paths are back online and one orboth of the sites are in the “disconnected” state the SRDF SRA willconnect, protect and enable discovered those Star environments.

1. Readers may recognize the “Force” option in reprotect. This is because, inessence, it is very similar behavior to the “Force” option in the TestFailover cleanup operation that was previously discussed in Chapter 3,“Testing Recovery Plans.”



Reprotect after a failover due to unrecoverable failureIn extreme circumstances, the storage and/or the computeenvironment may be rendered completely unrecoverable due to adisaster. In this scenario, reprotect will not be possible. Therefore theprocess of “reprotecting” the original recovery site is no differentthan the original setup of the protection groups and recovery plansfrom scratch. Refer to Chapter 2, “Installation and Configuration,” forinstructions on that process.

Reprotection 303

304










Since failback is in practice no different than failover, the process willnot be discussed any further in this chapter.



Failover with non-VMware devicesIn most environments, not all applications have been virtualized forone reason or another. Consequently, there are environments thathave a mixture of virtual and physical servers, and many of whichhave dependencies across their hosted applications regardless of thefact that they have different underlying server architectures.Consequently, during a disaster recovery operation, devices that arenot presented to the VMware environment may need to be failed overwith the VMware devices atomically. Unfortunately, SRM does nothave a built in mechanism for including replicated devices that arenot presented to the VMware environments. SRM will exclude themin datastore groups if they do not host VMFS volumes or are in use asRaw Device Mappings. Nevertheless, there is a way to bypass SRM tohave the SRA control devices that are not presented in theenvironment. SRDF/Star setup requires creation of consistencygroups and all devices in those groups will be controlled by anySRDF/Star operations. SRDF/Star does not allow operations on asubset of devices in the SRDF/Star group, so whatever devices arepresent in the SRDF/Star group, those devices will be failed over bythe SRA during recovery.

If the recovery operation is going to make use of the gold copyfunctionality discussed in Chapter 4, “Gold Copy Protection DuringFailover,” the non-VMware devices and the respective TimeFindertarget devices must also be added to the gold copy device pairingfiles.

It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are RW enabled. Sincethey are not in the VMware environment, neither SRM or the SRAhave any method of presenting them to a host. Therefore, the usermust perform whatever functions are necessary for the non-VMwarehosts to be able to mount the R2 copies the SRA read/write enabled.If the virtual machines depend on the non-VMware applications to berunning before they can power on, it would be advisable to add apause/script into the recovery plan before SRM powers on the virtualmachines so steps can be taken by the administrator or the script toprepare the non-VMware applications.

Failover with non-VMware devices 305

306



7


◆ Introduction ...................................................................................... 308◆ Recovery ............................................................................................ 309◆ Reprotection...................................................................................... 330◆ Failback.............................................................................................. 336◆ Failover with non-VMware devices .............................................. 337

Recovery Operationswith Cascaded

SRDF/Star

Recovery Operations with Cascaded SRDF/Star 307

308

Recovery Operations with Cascaded SRDF/Star


This chapter will discuss the following topics as they pertain toCascaded SRDF/Star configurations:

◆ Recovery with Cascaded SRDF/Star




◆ Reprotection

◆ Failback








Recovery 309

310



Cascaded SRDF/Star configurationDetailed step-by-step configuration of SRDF/Star is beyond thescope of this book but there are some important considerations tonote before configuring for SRM.

The 5.0 version of the SRDF adapter only supported failover betweenthe original workload site and the synchronous site. The 5.1 release ofthe SRDF SRA introduces support for failover to the asynchronoussite (also sometimes referred to as the tertiary site).

The protected SRM server and vCenter server must be located at theappropriate SRDF/Star sites to support failover to the desired site. Inother words, the protected VMware compute environment mustalways be at the workload site and the recovery VMware computeenvironment must be at the SRDF/Star synchronous site in order tofailover to the synchronous site or at the asynchronous site in order tosupport failover to the asynchronous site. The location of thecompute environment in accordance with the SRDF/Starenvironment dictates where the SRDF SRA permits failover to forthat given configuration. Once a topology has been configured,



failover may only occur between the original workload site and theconfigured target site, be it the synchronous site or the asynchronoussite. Whichever site does not host one of the two SRM-protectedcompute environments is deemed a bunker site by the SRDF SRA andtherefore cannot be managed by the SRDF SRA.

IMPORTANT

The SRDF SRA advanced option FailoverToAsyncSite must be setto DISABLED to allow recovery to the Synchronous site orENABLED to allow recovery to the Asynchronous site. This optionmust be set identically on both the recovery and protected SRMservers.

Planned Migration—Recovery with Cascaded SRDF/StarThe recovery option “Planned Migration” assures a gracefulmigration of virtual machines from a local vCenter to a remotevCenter. Any errors that the recovery plan encounters willimmediately fail the operation and require the user to remediatethese errors and restart the migration process. Therefore, a PlannedMigration assumes the following things (among other details):




Before executing a recovery plan failover, it is highly recommendedto test the recovery plan first (preferably multiple times) using the“Test” feature offered by SRM. Information on configuring andrunning a recovery test is discussed in detail in Chapter 3, “TestingRecovery Plans.”

The first step is to ensure that SRDF/Star is in a proper state. In orderto run a successful recovery the SRDF/Star state must be “Protected”.If SRDF/Star is in an alternative state, it must either be changed to aproper state using Symmetrix management applications or a DisasterRecovery operation may need to be run to ignore an invalidSRDF/Star state.

Recovery 311

312


Generally, a good indicator of valid SRDF/Star status is shown in the“Devices” tab in a given array manager by selecting the arraymanager and view the “Devices” tab. If the “Direction” columnshows a blue directional arrow, it might be a valid candidate forPlanned Migration. This is slightly different than with two-site SRDFhowever, as three-site SRDF has additional limitations on what statethe devices can be in to allow failover which are not reflected withinthe “Devices” tab. Not only do the device pairs need to be replicating,but they need to be consistent. In SRDF/Star terminology, this meansthat both the Asynchronous site and the Synchronous site must be inthe “Protected” state and SRDF/Star must be in the overall“Protected” state. The SRA requires that SRDF/Star is “Protected” soif either site is not in the “Protected” state, SRDF/Star cannot be inthe proper state and Planned Migration will fail. The SRDF SRA logwill report errors, such as the errors below, after a failed recoveryplan due to invalid SRDF/Star state:

[01/16 13:18:00 16212 0569SraStarGroup::IsValidStarState] STAR Group state isUnprotected.

[01/16 13:18:00 16212 0985SraStarGroup::ValidateInputDevices] [WARNING]: TheSTAR state is not valid. Exiting

[01/16 13:18:00 16212 3018PrepareFailoverCommand::RunOnGroup] [ERROR]: Inputdevice validation succeeded but one/many of theadapter's conditions is not met. Exiting withfailure

Example errors of a failed recovery plan due to invalid SRDF/Starstate can be seen in Figure 164.



Figure 164 Failed planned migration due to invalid SRDF/Star state

If the “Direction” column shows a broken gray bar either manualintervention might be required or the Disaster Recovery option mustbe selected. Planned migration will never be allowed in this state.

While the “Direction” column is, in general, a good indicator of RDFpair states it is inadequate to cover the many diverse possibilities ofSRDF/Star states. Therefore, it is advisable to use SymmetrixManagement Console1 (SMC) or Solutions Enabler to determine theexact status. A valid SRDF/Star group for Planned Migration isshown using Symmetrix Management Console in Figure 165. Notethat the 1st Target (synchronous site), the 2nd Target (theasynchronous site) and SRDF/Star state itself all report as“Protected”.

1. Version 1.5 of Unisphere for VMAX does not yet include SRDF/Starfunctionality. Therefore, SMC or SYMCLI must be utilized.

Recovery 313

314


Figure 165 Valid SRDF/Star state for Planned Migration in SMC

At this point, a planned migration can be initiated by selecting theappropriate recovery plan and the selecting the “Recovery” link ascan be seen in Figure 166.






As soon as the wizard completes the recovery operation willcommence. During the steps “Prepare Protected Site VMs forMigration” and “Change Recovery Site Storage To Writeable”, asshown in Figure 168, the SRDF SRA performs the necessary Staroperations on the devices in the protection group to failover.

Recovery 315

316


Re


The following two tables describe the steps for recovery withCascaded SRDF/Star. The tables have seven columns:


2. Issuing SRDF SRA and SYMAPI: Due to certain Starrequirements certain operations must be issued from certainSYMAPI server (for example symstar halt must be issued bythe workload SYMAPI server). Therefore the first few operationsare issued by the protected site SRDF SRA and consequently theprotected site SYMAPI server. If an error occurs during a givenoperation refer to the specified SRDF SRA server or SYMAPIserver listed for that operation for the correct logs.







For Cascaded SRDF/Star recovery to the Synchronous site, the SRAperforms the steps listed in Table 23 for recovery. For CascadedSRDF/Star recovery to the Asynchronous site, the SRA performs thesteps listed in Table 24 for recovery.

Table 23 Cascaded SRDF/Star steps for Synchronous Recovery

SRDFSRAStep #



Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite



2 Protectedsite


3 Recoverysite



4 Recoverysite

Switch Workload site.The original Workloadsite is now the Sync siteand vice versa.

Unprotected Disconnected Workload Disconnected

5 Recoverysite

Connect Sync target site Unprotected Connected Workload Disconnected

6 Recoverysite

Connect Async target site Unprotected Connected Workload Connected

7 Recoverysite

Protect Sync target site Unprotected Protected Workload Connected

8 Recoverysite

Protect Async target site Unprotected Protected Workload Protected

9 Recoverysite

Enable Star Protected Protected Workload Protected

Recovery 317

318


Readers may note an important difference between theAsynchronous recovery and the Synchronous recovery--the end stateof Star is not fully protected in an Asynchronous recovery unlikeSynchronous recovery. This is due to the fact that it is not allowed tohave a Cascaded environment that replicates Asynchronously on thefirst hop and Synchronously on the second hop. Consequently theSRDF SRA can only Connect the “Synchronous” site (the originalworkload site) which leaves it in Adaptive Copy Disk mode.

Table 24 Cascaded SRDF/Star steps for Asynchronous Recovery

SRDFSRAStep #



Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite

Created protected sitegold copies


2 Protectedsite


3 Recoverysite

Created recovery sitegold copies


4 Recoverysite

Switch Workload site.The original Workloadsite is now the Async siteand vice versa.


5 Recoverysite

Connect Async target site Unprotected Disconnected Connected Workload

6 Recoverysite

Connect Sync target site(adaptive copy)

Unprotected Connected Connected Workload

7 Recoverysite

Protect Async target site Unprotected Connected Protected Workload



IMPORTANT

In Cascaded SRDF/Star environments that have been recovered tothe Asynchronous target site, test failover back to the originalworkload site, with or without TimeFinder, ("without" referring toenabling the advanced settingTestFailoverWithoutLocalSnapshots) is not supported. Only fullrecovery back to the original workload site is supported with theSRDF SRA.

Unlike two-site SRDF, the SRA will in fact reverse the replication ofdirection and enable consistency during the failover process. Fortwo-site SRDF solutions, this replication reversal is reserved for the“Reprotect” operation. Due to the requirements inherent inSRDF/Star these steps are included in the failover process.

Once the SRDF/Star switch workload process has completed, theSynchronous target site devices are now write enabled so the deviceswill be mounted and the virtual machines will be registered andpowered-on. The SRDF/Star switch workload process will swap theRDF personalities of the devices and the change the site definitionswithin the SRDF/Star configuration. With Cascaded SRDF/Starrecovery to the Synchronous site the R21 devices become R1 devicesand the R1 devices become R21 devices while the R2 devices are notchanged. For Asynchronous recovery, the R21 devices remain staticand the R1 and R2 devices personalities are swapped.

In addition to the R1 devices being mounted on the recovery-sideESX hosts, the R2 or R21 volumes will be unmounted and detachedfrom the protection-side ESX hosts.


Recovery 319

320


The VMware kernel automatically renames VMFS volumes that havebeen resignatured by adding a “SNAP-XXXXXX” prefix to theoriginal name to denote that it is a copied file system. VMwarevCenter Site Recovery Manager provides an advanced setting(disabled by default), storageProvider.fixRecoveredDatastoreNames,that will cause this suffix to be automatically removed during therecovery plan.

Disaster RecoveryThe new recovery option “Disaster Recovery” should be selected forrecovery when there are issues with the infrastructure that willprevent a graceful recovery of virtual machines from a local vCenterto a remote vCenter. Unlike the “Planned Migration” option, mosterrors that the recovery plan encounters will be ignored by SRM. Theonly errors that will prevent a recovery in disaster recovery mode arefailures in the recovery site infrastructure. Anything from minorerrors to a complete failure of the protected site infrastructure will notprevent recovery.





Recovery after failure of compute environmentSituations can arise where there is a failure that is limited to thecompute environment while storage and replication remains online.This section assumes that the VMware environment on the protectedsite is down and cannot be contacted over the network by therecovery site. This leads to a disconnected state for the SRM serverpairing and planned migrations will not be allowed. In this situation




only “Disaster Recovery” is offered as a valid option for failoverrecovery modes as seen in Figure 169.


In this scenario, the storage environment remains online andreplication has not been halted. Therefore, the SRDF/Star state isexpected to be “Protected”1.

1. Disaster recovery mode also tolerates other SRDF/Star states, but thesestates are discussed in the ensuing section.

Recovery 321

322










With the exception of these failures, the result of the recovery processwhen the protected site compute environment is down is no differentto a normal planned migration. When “Change Recovery Site Storageto Writeable” executes, the SRDF SRA follows the same workflow asshown in Table 23 on page 317 or Table 24 on page 318 (depending onwhether recovery is to the synchronous site or asynchronous siterespectively).



◆ Loss of power




When the link between the local and remote Symmetrix arraybecomes “Partitioned” (referred to as “PathFail” by SRDF/Star),SRDF/Star enters a “Tripped” state. Figure 172 on page 324 shows anexample Cascaded SRDF/Star environment that has been trippedbecause the protected site has experienced a total failure. Since theWorkload site is down, the paths to the Synchronous site and theAsynchronous site are in a “PathFail” state.

Recovery 323

324


Figure 172 Cascaded SRDF/Star in a “Tripped” state in SMC

Note: While a variety of states are allowed for discovery of SRDF/Stardevices, only two SRDF/Star states are allowed for failover, be it DisasterRecovery mode or Planned Migration mode. The SRDF/Star state must beeither “Protected” (the only state supported by Planned Migration) or“Tripped”. Disaster Recovery supports both states. Any other state requiresuser intervention to configure it to a valid state before recovery.

In a complete failure of the protected site, the Asynchronous siteremains protected while the Synchronous site enters the “PathFail”state. schematically depicts a Cascaded SRDF/Star environmentbefore recovery when it has experienced a complete protected sitefailure.



Cascaded SRDF/Star with workload site failure

Since the entire workload site failed in this scenario, the arraymanager will not be able to update its information and the “Devices”screen will not show correct information for all device pairs. This isdue to the fact that the protected side SYMAPI server is down and theprotected side array information cannot be retrieved. An example ofthis can be seen in Figure 173.

Figure 173 Failed device discovery due to off-line protected site SYMAPI server

Recovery 325

326


To recover the virtual machines on the surviving site, the “Recovery”link should be selected. Since the protected site is completely downthe “Disaster Recovery” mode is the only available option. The SRAdoes not, in principle, act any differently if the “Disaster Recovery”mode is chosen as compared to “Planned Migration”. The SRA willstill attempt all operations as it would during a migration and returnsuccess or failure. The difference is that SRM will not fail the recoveryplan on reception of a failed operation by the SRA. In the case of aprotected site complete failure all steps involving the protected sitewill accordingly fail. Figure 174 shows a recovery plan that has beenexecuted when the protected site has experienced a complete failureand the steps that will fail because of it.



Disaster Recovery to the Synchronous site with CascadedSRDF/StarThe SRDF SRA recovers Cascaded SRDF/Star environments in adifferent way than it recovers Concurrent SRDF/Star environmentswhen prompted by a protected site disaster. After attempting to andfailing to switch the SRDF/Star Workload site to the Sync target, theSRA will then perform a reconfiguration of the Cascaded setup toturn it into a Concurrent environment. The reason for this is that ifthe Cascaded environment is just switched, so that the previous Syncsite is the workload site, it will attempt to connect the new Sync site.



Since the newly designated Sync site is the failed previouslydesignated Workload site this operation would fail. Accordingly,since Cascaded requires the Synchronous site to be connected beforethe Async site can be connected (when the workload site is at theprimary or secondary site), the Async site cannot be connected.Therefore no remote protection will be possible until the Sync site isback up. This is an unacceptable situation for most customers whoare using three-site replication. The solution to this problem is toreconfigure to Concurrent SRDF/Star. Concurrent SRDF/Star doesnot have this dependency, and either site can be connected andprotected regardless to the state of the other site. Therefore in asituation where the protected site fails, the SRDF SRA willreconfigure the Cascaded SRDF/Star environment into a ConcurrentSRDF/Star one.

Figure 175 depicts the final state of a Cascaded SRDF/Starenvironment which has been reconfigured into ConcurrentSRDF/Star after disaster recovery to the secondary (sync) site.

Figure 175 Cascaded SRDF/Star reconfigured into Concurrent SRDF/Star

Recovery 327

328


Table 25 shows the process steps taken by the SRDF SRA whenrecovering a Cascaded SRDF/Star setup with the workload sitedown.

Table 25 Cascaded SRDF/Star recovery to Sync site workflow

Step # Step detail

SRDF/StarState afterstep Site A after step Site B after step Site C after step

1 State before failover Tripped Workload site PathFail Protected


Tripped Workload site PathFail Protected

3 Create and/or activate goldcopy


4 Perform a disconnect trip onsite C.


5 Switch Workload site to site Bwith “keep data” set to site B




7 Attempt to connect site C (willfail)


8 Perform a SRDF/Starreconfiguration from Cascadedto Concurrent




10 Connect site C Unprotected Disconnected Workload site Connected

11 Protect site C Unprotected Disconnected Workload site Protected

12 Attempt to enable SRDF/Star(will fail)

Unprotected Disconnected Workload site Protected



Disaster Recovery to the Asynchronous site with CascadedSRDF/StarThe disaster recovery workflow for recovery to the asynchronous sitein Cascaded SRDF/Star is a somewhat more straight forward processas compared to recovery to the synchronous target site. Areconfiguration is not required and the SRDF topology can remain ina Cascaded configuration (albeit still not completely protected).

When the workload site completely fails (storage and compute)recovery can be initiated to the tertiary/asynchronous target site ofthe Cascaded SRDF/Star configuration (site C). Site B and site Ccontinue to function properly and the links between them remain up.Table 26 shows the workflow to recover to the asynchronous site.

Table 26 Cascaded SRDF/Star recovery to Async site workflow

Step # Step detailSRDF/Star Stateafter step Site A after step Site B after step Site C after step

1 State before failover Tripped Workload site PathFail Protected

2 Check if MSC cleanup is neededand perform a symstar cleanupon affected site. Requiredcleanup is denoted by the pathbeing in the “Pathfail CleanReq”state


3 Create and/or activate gold copy Tripped Workload site PathFail Protected

4 Perform a disconnect trip on siteC.


5 Switch Workload site to site Cwith “keep data” set to site B

Unprotected Disconnected Connected Workload site

6 Attempt to connect site A (will fail) Unprotected Disconnected Connected Workload site

7 Protect site B Unprotected Disconnected Protected Workload site

Recovery 329

330


ReprotectionAfter a recovery plan or planned migration has run, there are oftencases where the environment must continue to be protected againstfailure, to ensure its resilience or to meet objectives for disasterrecovery. With Site Recovery Manager 5, reprotection is a newextension to recovery plans that enables the environment at therecovery site to establish replication and protection of theenvironment back to the original protected site.




Reprotect is only available after a recovery operation has occurred,which is indicated by the recovery plan being in the “RecoveryComplete” state. A reprotect can be executed by selecting theappropriate recovery plan and selecting the “Reprotect” link as



shown in Figure 176.


The reprotect operation does the following things for CascadedSRDF/Star environments:

◆ Reverses protection groups. The protection groups are deletedon the original protection SRM server and are recreated on theoriginal recovery SRM server in the reverse direction. Theinventory mappings are configured (assuming the user haspre-configured them in SRM on the recovery site) and thenecessary “shadow” or “placeholder” VMs are created andregistered on the newly designated recovery SRM server1.

1. For detailed information on the non-storage related portions of thereprotect operation refer to VMware documentation onwww.vmware.com

Reprotection 331

332





In the case of Cascaded SRDF/Star environments, the RDF swap andthe re-establish are not executed in the reprotect operation. Due to thenature and requirements of SRDF/Star, these steps must be includedin the failover operation. Therefore, the SRA detects CascadedSRDF/Star configurations and these reprotect steps becomenon-operation events for the SRA. If the Cascaded SRDF/Starenvironment has one or more legs in a disconnected state, the SRDFSRA will attempt to connect, protect and enable Star.


Disaster recovery mode allows for failures ranging from the verysmall to a full site failure of the protection datacenter. If these failuresare temporary and recoverable a fully-successful reprotection may bepossible once those failures have been rectified. In this case, areprotection will behave similar to the scenario described in the



previous section. If a reprotection is run before the failures arecorrected or certain failures can not be fully recovered, an incompletereprotection operation will occur. This section describes this scenario.


◆ A recovery must be executed with all steps finishing successfully.If there were any errors during the recovery, the user needs toresolve the issues that caused the errors and then rerun therecovery again.



If the protected site SRM server was disconnected during failover andis reconnected later, SRM will want to retry certain failoveroperations before allowing reprotect. This typically occurs if therecovery plan was not able to connect to the protected side vCenterserver and power down the virtual machines due to networkconnectivity issues. If network connectivity is restored after therecovery plan was failed over, SRM will detect this situation andrequire the recovery plan to be re-run in order to power those VMsdown. Figure 179 on page 333 shows the message SRM displays inthe recovery plan when this situation occurs.


Reprotection 333

334


A reprotection operation will fail if it encounters any errors the firsttime it runs. If this is the case, the reprotect must be run a second timebut with the “Force” option selected.


Once the force option is selected, any errors will be acknowledgedand reported but ignored. This will allow the reprotect operation tocontinue even if the operation has experienced errors. It will attemptall of the typical steps and complete whichever ones are possible.Therefore, in certain situations, the SRDF replication may not beproperly reversed even though the recovery plan and protectiongroup(s) were. If the “Configure Storage to Reverse Direction” stepfails, manual user intervention may be required to complete theprocess. The SRDF SRA will attempt to connect, protect and enableStar during the reprotection operation.

If any of these steps fail during reprotection, a device discoveryoperation can be executed when paths are back online (and the sitesare in the “disconnected” state only) and the SRDF SRA will connect,protect and enable discovered Star environments as needed.



Reprotect after a failover due to unrecoverable failureIn extreme circumstances, the storage and/or the computeenvironment may be rendered completely unrecoverable due to adisaster. In this scenario, reprotect will not be possible. Therefore theprocess of “re-protecting” the original recovery site is no differentthan the original setup of the protection groups and recovery plansfrom scratch. Refer to Chapter 2, “Installation and Configuration,” forinstructions on that process.

Reprotection 335

336










Since failback is in practice no different than failover, the process willnot be discussed any further in this section.



Failover with non-VMware devicesIn most environments, not all applications have been virtualized forone reason or another. Consequently, there are environments thathave a mixture of virtual and physical servers, and many of whichhave dependencies across their hosted applications regardless of thefact that they have different underlying server architectures.Consequently, during a disaster recovery operation, devices that arenot presented to the VMware environment may need to be failed overwith the VMware devices atomically. Unfortunately, SRM does nothave a built in mechanism for including replicated devices that arenot presented to the VMware environments. SRM will exclude themin datastore groups if they do not host VMFS volumes or are in use asRaw Device Mappings. Nevertheless, there is a way to bypass SRM tohave the SRA control devices that are not presented in theenvironment. SRDF/Star setup requires creation of consistencygroups and all devices in those groups will be controlled by anySRDF/Star operations. SRDF/Star does not allow operations on asubset of devices in the SRDF/Star group, so whatever devices arepresent in the SRDF/Star group, those devices will be failed over bythe SRA during recovery.

If the recovery operation is going to make use of the gold copyfunctionality discussed in Chapter 4, “Gold Copy Protection DuringFailover,” the non-VMware devices and the respective TimeFindertarget devices must also be added to the gold copy device pairingfiles.

It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are read/write enabled.Since they are not in the VMware environment, neither SRM or theSRA have any method of presenting them to a host. Therefore, theuser must perform whatever functions are necessary for thenon-VMware hosts to be able to mount the R2 copies the SRAread/write enabled. If the virtual machines depend on thenon-VMware applications to be running before they can power on, itwould be advisable to add a pause/script into the recovery planbefore SRM powers on the virtual machines so steps can be taken bythe administrator or the script to prepare the non-VMwareapplications.


338



8


◆ Introduction ...................................................................................... 340◆ Recovery ............................................................................................ 341◆ Reprotection and Failback .............................................................. 353◆ Failover with non-VMware devices .............................................. 354

Recovery Operationswith Concurrent SRDF

Recovery Operations with Concurrent SRDF 339

340

Recovery Operations with Concurrent SRDF


This chapter will discuss the following topics as they pertain toConcurrent SRDF (non-Star) configurations:

◆ Recovery with Concurrent SRDF




◆ Reprotection and failback








Recovery 341

342



Planned Migration—Recovery with Concurrent SRDFThe recovery option “Planned Migration” assures a gracefulmigration of virtual machines from a local vCenter to a remotevCenter. Any errors that the recovery plan encounters willimmediately fail the operation and require the user to re-mediatethese errors and restart the migration process. Therefore, a PlannedMigration assumes the following things (among other details):








The first step is to ensure that RDF Pair is in a proper state. In order torun a successful recovery the RDF Pair state must be either,“Synchronized”, “SyncInProg”, “Consistent”, or “Suspended”. If theRDF pair is in an alternative state, it must either be changed to aproper state using Symmetrix management applications or a DisasterRecovery operation may need to be run to allow the SRA to ignoreinvalid RDF pair states.

Generally, a good indicator of valid SRDF status is shown in the“Devices” tab in a given array manager by selecting the arraymanager and view the “Devices” tab. If the “Direction” columnshows a blue directional arrow, it might be a valid candidate forPlanned Migration. If the “Direction” column shows a broken graybar either manual intervention is needed or the Disaster Recoveryoption might be required. Planned migration will never be allowed inthis state.

While the “Direction” column is, in general, a good indicator of RDFpair states it is inadequate to cover the many diverse possibilities ofRDF pair states. Therefore, it is advisable to use Unisphere forVMAX1 or Solutions Enabler to determine the exact status.

A planned migration can be initiated by selecting the appropriaterecovery plan and the selecting the “Recovery” link as can be seen inFigure 183.


Once the Recovery link has been selected, a short confirmationwizard appears asking to confirm the initiation of the recoveryoperation and in which mode the recovery plan should be run. This

1. If the group is a composite group, Symmetrix Management Console mustbe used to query RDF status as composite groups are not yet supportedfor use with Unisphere 1.5

Recovery 343

344


screen is shown in Figure 184.


As soon as the wizard completes the recovery operation willcommence. During the steps “Prepare Protected Site VMs forMigration” and “Change Recovery Site Storage To Writeable”, asshown in Figure 185, the SRDF SRA performs the necessary RDFoperations on the devices in the protection group to failover.



Re


The following two tables describe the steps for recovery withConcurrent SRDF. The tables have seven columns:


2. Issuing SRDF SRA and SYMAPI: Due to particular SRDFrequirements certain operations must be issued from a certainSYMAPI server. Therefore the first few operations are issued bythe protected site SRDF SRA and consequently the protected siteSYMAPI server. If an error occurs during a given operation referto the specified SRDF SRA server or SYMAPI server listed for thatoperation for the correct logs.





Recovery 345

346


For Concurrent SRDF recovery to the asynchronous site, the SRAperforms the steps listed in Table 27 for recovery.



Table 27 Concurrent SRDF Failover steps for Asynchronous Recovery

SRDFSRAStep #


Originalworkload sitestate

Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite


Read/writeenabled

Workload Synchronizedand writedisabled

Consistent andwrite disabled

2 Protectedsite

Suspend replication Read/writeenabled

Workload Suspended andwrite disabled

Suspended andwrite disabled

3 Protectedsite

Write disable R1 devices Write disabled Workload andwrite disabled



4 Recoverysite


Write disabled Workload andwrite disabled



5 Recoverysite

Perform RDF Failover torecover on Asynchronoussite

Write disabled Suspended andwrite disabled


Workload andFailed Over





Disaster RecoveryThe new recovery option “Disaster Recovery” should be selected forrecovery when there are issues with the infrastructure that willprevent a graceful recovery of virtual machines from a local vCenterto a remote vCenter. Unlike the “Planned Migration” option, mosterrors that the recovery plan encounters will be ignored by SRM. Theonly errors that will prevent a recovery in disaster recovery mode arefailures in the recovery site infrastructure. Anything between minorerrors to a complete failure of the protected site infrastructure will notprevent recovery.





Recovery after failure of compute environmentSituations can arise where there is a failure that is limited to thecompute environment while storage and replication remains online.This section assumes that the VMware environment on the protectedsite is down and cannot be contacted over the network by therecovery site. This leads to a disconnected state for the SRM server

Recovery 347

348


pairing and planned migrations will not be allowed. In this situationonly “Disaster Recovery” is offered as a valid option for failoverrecovery modes as seen in Figure 186.


In this scenario, the storage environment remains online andreplication has not been halted. Therefore, the SRDF site state isexpected to be “Consistent”1.

1. Disaster recovery mode also tolerates other SRDF states, but these statesare discussed in the ensuing section.









Recovery 349

350


With the exception of these failures, the result of the recovery processwhen the protected site compute environment is down is no differentto a normal planned migration. When “Change Recovery Site Storageto Writeable” executes, the SRDF SRA follows the same workflow asshown in Table 27 on page 346 and that is described in theparagraphs following that table.



◆ Loss of power




When the link between the local and remote Symmetrix array goesdown the replicated devices both links will enter a state of“Partitioned”.

Since the entire workload site failed in this scenario, the arraymanager will not be able to update its information and the “Devices”screen will not show correct information for all device pairs. This isdue to the fact that the protected side SYMAPI server is down and theprotected side array information cannot be retrieved. An example of



this can be seen in Figure 189.



Recovery 351

352


.



The following list shows the workflow of the SRDF SRA whenrecovering to the asynchronous leg of Concurrent SRDF when the R1site has experienced a complete failure:

1. R1 site storage and compute fails and RDF links to Synchronoussite and Asynchronous site become partitioned.

2. SRDF SRA checks for MSC Cleanup flag on the consistency groupon the recovery SYMAPI server. If this flag is set the SRDF SRAperforms a MSC Cleanup on the group.

3. If configured, the SRDF SRA creates and activates the recoverysite gold copies.

4. The SRDF SRA performs an RDF Failover on the Asynchronousdevices, setting those devices to be read/write enabled and ableto be mounted at the recovery site. Since the R1 site is still downthe RDF Pair state remains as Partitioned and not Failed Over.



Reprotection and FailbackReprotection and failback with SRM is not supported with non-StarConcurrent SRDF configurations. Due to the lack of flexibility andinsight into the three sites in non-Star configurations the process toreprotect and failback is too complex for the SRDF SRA to handlewithout user intervention (this is why EMC recommendsSRDF/Star). If a reprotection is executed against a Cascaded SRDFprotection group the following error will be logged in the SRDF SRAlog:

[ERROR]: ReverseReplication is not supported in thecurrent SRDF configuration.

Therefore, if reprotection and/or failback is desired, users mustmanually re-initiate replication and re-create protection groups. Theuser must:

◆ Delete the protection groups.

◆ Delete the recovery plan or remove the Concurrent SRDFprotection groups from it before executing a reprotection.

◆ Re-establish replication and create a configuration that issupported by the SRDF SRA (if possible) so that it can failover thedevices back to the original site.

◆ Create new protection groups and recovery plans for thereconfigured Concurrent SRDF devices.

Reprotection and Failback 353

354


Failover with non-VMware devicesIn most environments, not all applications have been virtualized forone reason or another. Consequently, there are environments thathave a mixture of virtual and physical servers, many of which havedependencies across their hosted applications regardless of the factthat they have different underlying server architectures.Consequently, during a disaster recovery operation, devices that arenot presented to the VMware environment may need to be failed overwith the VMware devices at the same time. Unfortunately, SRM doesnot have a built in mechanism for including replicated devices thatare not presented to the VMware environments. SRM will excludethem in datastore groups if they do not either host VMFS volumes orare in use as Raw Device Mappings. Nevertheless, there is a way to“bypass” SRM to have the SRA control devices that are not presentedin the environment. In order for this to work there are a fewrequirements:








It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are read/write enabled.Since they are not in the VMware environment, neither the SRA norSRM have methods of presenting them to a host. Therefore, the usermust perform whatever functions are necessary for the non-VMwarehosts to be able to mount the R2 copies the SRA read/write enabled.If the virtual machines depend on the non-VMware applications to berunning before they can power on, it would be advisable to add apause or script into the recovery plan before SRM powers on thevirtual machines so steps can be taken by an administrator or scriptto prepare the non-VMware applications.


356



9


◆ Introduction ...................................................................................... 358◆ Recovery ............................................................................................ 359◆ Reprotection and Failback .............................................................. 371◆ Failover with non-VMware devices .............................................. 372

Recovery Operationswith Cascaded SRDF

Recovery Operations with Cascaded SRDF 357

358

Recovery Operations with Cascaded SRDF


This chapter will discuss the following topics as they pertain toCascaded SRDF (non-Star) configurations:

◆ Recovery with Cascaded SRDF




◆ Reprotection and Failback








Recovery 359

360



Planned Migration—Recovery with Cascaded SRDFThe recovery option “Planned Migration” assures a gracefulmigration of virtual machines from a local vCenter to a remotevCenter. Any errors that the recovery plan encounters willimmediately fail the operation and require the user to remediatethese errors and restart the migration process. Therefore, a PlannedMigration assumes the following things (among other details):








The first step is to ensure that RDF Pair is in a proper state. In order torun a successful recovery the RDF Pair state must be either,“Synchronized”, “SyncInProg”, “Consistent”, or “Suspended”. If theRDF pair is in an alternative state, it must either be changed to aproper state using Symmetrix management applications or a DisasterRecovery operation may need to be run to allow the SRA to ignoreinvalid RDF pair states.

Generally, a good indicator of valid SRDF status is shown in the“Devices” tab in a given array manager by selecting the arraymanager and view the “Devices” tab. If the “Direction” columnshows a blue directional arrow, it might be a valid candidate forPlanned Migration. If the “Direction” column shows a broken graybar either manual intervention is needed or the Disaster Recoveryoption might be required. Planned migration will never be allowed inthis state.

While the “Direction” column is, in general, a good indicator of RDFpair states it is inadequate to cover the many diverse possibilities ofRDF pair states. Therefore, it is advisable to use Unisphere forVMAX1 or Solutions Enabler to determine the exact status.

A planned migration can be initiated by selecting the appropriaterecovery plan and the selecting the “Recovery” link as can be seen inFigure 192.


Once the Recovery link has been selected, a short confirmationwizard appears asking to confirm the initiation of the recoveryoperation and in which mode the recovery plan should be run. This

1. If the group is a composite group Symmetrix Management Console mustbe used to query RDF status as this group type is not yet supported foruse with Unisphere 1.5

Recovery 361

362


screen is shown in Figure 193.


As soon as the wizard completes the recovery operation willcommence. During the steps “Prepare Protected Site VMs forMigration” and “Change Recovery Site Storage To Writeable”, asshown in Figure 194, the SRDF SRA performs the necessary RDFoperations on the devices in the protection group to failover.



Re


The following two tables describe the steps for recovery withCascaded SRDF. The tables have seven columns:


2. Issuing SRDF SRA and SYMAPI: Due to particular SRDFrequirements certain operations must be issued from a certainSYMAPI server. Therefore the first few operations are issued bythe protected site SRDF SRA and consequently the protected siteSYMAPI server. If an error occurs during a given operation referto the specified SRDF SRA server or SYMAPI server listed for thatoperation for the correct logs.





Recovery 363

364


For Cascaded SRDF recovery to the asynchronous site, the SRAperforms the steps listed in Table 28 for recovery.



Table 28 Cascaded SRDF Failover steps for Asynchronous Recovery

SRDFSRAStep #


Site A afterstep

Site B afterstep

Site C afterstep

1 Protectedsite


Workload Synchronizedand writedisabled


2 Protectedsite

Suspend replication onhop one



3 Protectedsite

Suspend replication onhop two



4 Protectedsite

Write disable R1 devices Workload andWritedisabled



5 Recoverysite


Workload andWritedisabled



6 Recoverysite

Perform RDF Failover torecover on Asynchronoussite

Suspendedand Writedisabled


Workload andread/writeenabled




Disaster RecoveryThe new recovery option “Disaster Recovery” should be selected forrecovery when there are issues with the infrastructure that willprevent a graceful recovery of virtual machines from a local vCenterto a remote vCenter. Unlike the “Planned Migration” option, mosterrors that the recovery plan encounters will be ignored by SRM. Theonly errors that will prevent a recovery in disaster recovery mode arefailures in the recovery site infrastructure. Anything between minorerrors to a complete failure of the protected site infrastructure will notprevent recovery.





Recovery after failure of compute environmentSituations can arise where there is a failure that is limited to thecompute environment while storage and replication remains online.This section assumes that the VMware environment on the protectedsite is down and cannot be contacted over the network by therecovery site. This leads to a disconnected state for the SRM server


Recovery 365

366


pairing and planned migrations will not be allowed. In this situationonly “Disaster Recovery” is offered as a valid option for failoverrecovery modes as seen in Figure 195.


In this scenario, the storage environment remains online andreplication has not been halted. Therefore, the SRDF site state isexpected to be “Consistent”1.

1. Disaster recovery mode also tolerates other SRDF states, but these statesare discussed in the ensuing section.









Recovery 367

368


With the exception of these failures, the result of the recovery processwhen the protected site compute environment is down is no differentto a normal planned migration. When “Change Recovery Site Storageto Writeable” executes, the SRDF SRA follows the same workflow asshown in Table 28 on page 364 and that is described in theparagraphs following that table.



◆ Loss of power




When the link between the local and remote Symmetrix array goesdown the replicated devices both links will enter a state of“Partitioned”.

Since the entire workload site failed in this scenario, the arraymanager will not be able to update its information and the “Devices”screen will not show correct information for all device pairs. This isdue to the fact that the protected side SYMAPI server is down and theprotected side array information cannot be retrieved. An example of



this can be seen in Figure 198.



Recovery 369

370


.



The following list shows the workflow of the SRDF SRA whenrecovering to the asynchronous site of Cascaded SRDF when the R1site has experienced a complete failure:

1. R1 site storage and compute fails and RDF links to Synchronoussite becomes partitioned and the RDF link between theSynchronous (secondary/bunker site) and the Asynchronous(tertiary/recovery site) remains consistent.

2. SRDF SRA checks for MSC Cleanup flag on the consistency groupon the recovery SYMAPI server. If this flag is set the SRDF SRAperforms a MSC Cleanup on the group.

3. If configured, the SRDF SRA creates and activates the recoverysite gold copies.

4. The SRDF SRA performs an RDF Failover on the Asynchronousdevices, setting those devices to be read/write enabled so thatthey can be mounted at the recovery site. The status of the RDFlink between the Synchronous site and the Asynchronous site isnow Failed Over.



Reprotection and FailbackReprotection and failback with SRM is not supported with non-StarCascaded SRDF configurations. Due to the lack of flexibility andinsight into the three sites in non-Star configurations the process toreprotect and failback is too complex for the SRDF SRA to handlewithout user intervention (for this reason SRDF/Star is highlyrecommended). If a reprotection is executed against a CascadedSRDF protection group the following error will be logged in the SRDFSRA log:

[ERROR]: ReverseReplication is not supported in thecurrent SRDF configuration.

Therefore, if reprotection and/or failback is desired, users mustmanually re-initiate replication and re-create protection groups. Theuser must:

◆ Delete the protection groups.

◆ Delete the recovery plan or remove the Cascaded SRDFprotection groups from it before executing a reprotection.

◆ Re-establish replication and create a configuration that issupported by the SRDF SRA (if possible) so that it can failover thedevices back to the original site.

◆ Create new protection groups and recovery plans for thereconfigured Cascaded SRDF devices.

Reprotection and Failback 371

372


Failover with non-VMware devicesIn most environments, not all applications have been virtualized forone reason or another. Consequently, there are environments thathave a mixture of virtual and physical servers, many of which havedependencies across their hosted applications regardless of the factthat they have different underlying server architectures.Consequently, during a disaster recovery operation, devices that arenot presented to the VMware environment may need to be failed overwith the VMware devices at the same time. Unfortunately, SRM doesnot have a built in mechanism for including replicated devices thatare not presented to the VMware environments. SRM will excludethem in datastore groups if they do not either host VMFS volumes orare in use as Raw Device Mappings. Nevertheless, there is a way to“bypass” SRM to have the SRA control devices that are not presentedin the environment. In order for this to work there are a fewrequirements:








It is important to note that the non-VMware devices will not becontrolled in anyway by the SRA once they are RW enabled. Sincethey are not in the VMware environment, neither the SRA nor SRMhave methods of presenting them to a host. Therefore, the user mustperform whatever functions are necessary for the non-VMware hoststo be able to mount the R2 copies the SRA read/write enabled. If thevirtual machines depend on the non-VMware applications to berunning before they can power on, it would be advisable to add apause or script into the recovery plan before SRM powers on thevirtual machines so steps can be taken by an administrator or scriptto prepare the non-VMware applications.


374



10


◆ Introduction ...................................................................................... 376◆ Symmetrix Access Controls ............................................................ 377◆ Symmetrix Authorizations.............................................................. 389

Symmetrix Security withEMC SRDF Adapter

Symmetrix Security with EMC SRDF Adapter 375

376

Symmetrix Security with EMC SRDF Adapter

IntroductionEMC SRDF Adapter utilizes the installation of Solutions Enablersoftware on the VMware vCenter Site Recovery Manager server toperform management operations on an EMC Symmetrix storagearray. This is frequently viewed as a potential security issue bystorage administrators since unauthorized users could intentionallyor unintentionally perform malicious activities. The security concerncan be addressed by the use of Symmetrix Access Control Listsand/or Symmetrix Authorizations to manage access and control ofthe Symmetrix storage arrays visible to the VMware vCenter SiteRecovery Manager server.

This chapter will discuss the minimum required rights necessary toallow the user to perform all of the operations required to fully useVMware vCenter Site Recovery Manager. The first section will dealwith Symmetrix Access Control Lists (host-based security) and thesecond section will discuss Symmetrix Authorizations (user-basedsecurity). Each section is further divided into two sections: oneconcerning the EMC SRDF SRA itself and the second concerning theEMC VSI Symmetrix SRA Utilities.

It is important to note that the Symmetrix Access Control Lists andAuthorizations are complementary pieces of security functionalitythat do not rely on or interact with one another. Therefore they can beimplemented together or individually.

Detailed discussions (initial implementation, best practices etc.) ofthese security offerings are beyond the scope of this book. Readersshould consult the document EMC Solutions Enabler Symmetrix ArrayManagement CLI available at support.emc.com for furtherinformation.

Note: This discussion presumes all Symmetrix arrays involved in SRDFreplication will have security enabled. Therefore, security requirements forall arrays will be discussed. If, however, security is disabled on any arrays inthe SRDF environment, they can be excluded from these requirements.



Symmetrix Access ControlsMany product applications, such as EMC Ionix™ ControlCenter®,Symmetrix Management Console, TimeFinder, SRDF, Optimizer,Resource View, Database Tuner, and various ISV products, can issuemanagement commands to any device in a Symmetrix array. Bydefault, anyone with access to Symmetrix-based managementsoftware can execute any function on any Symmetrix device.

Shared systems such as these may be vulnerable to one host,accidentally or intentionally, tampering with another’s devices. Toprevent this, the symacl command can be used by an administratorof the Symmetrix storage site to set up and restrict host access todefined sets of devices, or access pools, across the various Symmetrixarrays.

This SYMCLI component supports Symmetrix Access Controlrequirements. The Access Control command allows you to set up andmaintain an access controlled environment over the Symmetrixaccess pools. The command, symacl, sets up access controlmechanisms and changes access control entries in the access controldatabase1.

A Symmetrix-based access control database contains all themechanisms or information to govern access to Symmetrix accesspools.

Information about the following access control mechanisms comprisethe access control database:

◆ Access Control groups. Unique access IDs and names areassigned (together) to hosts and then sorted into access controlgroups according to similar needs (determined by anAdministrator). Access groups are allowed to act on access poolsbased on permissions (access types) granted by theAdministrator.

◆ Access pools. Permissions (or access types), such as BCV, SRDF,ADMIN, are assigned to allow a host to perform certain SolutionsEnabler functionality on a specified set of devices. These sets ofdevices are referred to as access pools.

1. For more information about the syntax of the symacl command, refer tothe EMC Solutions Enabler Symmetrix CLI Command Reference.

Symmetrix Access Controls 377

378


◆ Access Control Entries (ACEs). Once the group and access poolmechanisms are established, the access control entries (ACEs) arecreated, which grant permissions to these pools. The ACEs of thevarious access groups (along with groups and pools) aremanaged and stored in the access control database.

◆ Access Control Lists (ACLs). A group of ACEs that are associatedwith the same group.

SYMACL Unique IdentifiersThe first step in the process of configuring SYMACLs is to create theaccess control groups by gathering the SYMACL unique identifiers.

If the SYMAPI servers are configured to be local (they have directaccess to Gatekeepers) the unique identifier of the Solutions Enablerinstance installed on the SRM server should be retrieved. Thisidentifier can be obtained by opening up the Windows commandprompt on the local server and executing the following command:

symacl -unique

Figure 200 shows a printout of this command where the uniqueidentifier of the local install of Solutions Enabler is returned.

Figure 200 Retrieving the SYMACL unique identifier with Solutions Enabler

Symmetrix access control identifies individual management hostswith an access ID. There are two different approaches to generatingaccess IDs:

◆ Hardware-based access ID: By default, a host's access ID isderived from hardware characteristics of that host. On x86 (32-bitIntel), x86_64 (64-bit Intel/AMD), and IA 64 platforms, a networkinterface MAC address is used. On other platforms, differentcharacteristics of the host, such as a processor identifier, are used.



◆ Alternate access ID: Optionally, a host's access ID can begenerated at random or from a provided passphrase and storedto a secure location on the local disk. This functionality issupported for all platforms but is strongly recommended onplatforms where the access ID is derived from a networkinterface MAC address.

Alternate access IDsAlternate access IDs are available for all platforms. Alternate accessIDs do not utilize the host’s hardware identifiers (such as MACaddress) to generate an encrypted access ID. When enabled, SolutionsEnabler can:

◆ Randomly generate an alternate access ID

◆ Generate an alternate access ID based on a passphrase or file

To use alternate access IDs users must enable theSYMAPI_ALTERNATE_ACCESS_ID option in the following file:

<SYMAPI_HOME>/config/options

To activate an alternate access ID randomly:

symacl -unique -passphrase Passphrase

To activate an alternate access ID using a passphrase:

symacl -unique -passphrase Passphrase

To activate an alternate access ID using a passphrase stored in a fileon the local disk:

symacl -unique -passphrase -file pathname

Access control in client/server modeAnyone with access to Symmetrix management software can executefunctions on any Symmetrix device. Many applications can issuemanagement commands to any device in a Symmetrix deployment.

Such shared systems may be vulnerable to a host (accidentally orintentionally) tampering with another’s devices. To prevent this, thesymacl command can be used by to set up and restrict host access todefined sets of devices across the Symmetrix arrays.

By default, client/server mode operations are executed on the serverhost using the access ID of the server. Access control checks areperformed against the rules established for the server host, regardlessof which client host initiated the operations.


380


Users can use the access ID of the client host instead of the serverhost. When this is enabled, access control rules must be establishedfor, and checked against, the client hosts from which the operationsare issued.

To use the access ID of the client host, users must make changes in theoptions file on the client and the server host, as explained in thefollowing sections.

Server host options file settingsOn the server host, the following option controls the source of theaccess ID used for the client/server sessions:

SYMAPI_USE_ACCESS_ID = CLIENT | SERVER | ANY

The behavior of this option is as follows:

◆ When set to CLIENT, an access ID supplied by the client host isused. If the client did not provide an access ID, operations fail.This can occur if the client is running a version of SolutionsEnabler lower than 7.4 or if this functionality was not configuredon the client (recommended setting).

◆ When set to SERVER (default), the server always uses its ownaccess ID and ignores an access ID, if any, provided by the clients.

◆ When set to ANY, the server uses an access ID provided by aclient. If one is not provided, the server uses its own access ID.

Client host options file settingsThe use of the alternate access ID, described earlier, must be enabledto use this functionality:

SYMAPI_ALTERNATE_ACCESS_ID = ENABLE

Additionally, users must set the following option to control whetherthe client can send its own access ID to the server for use there:

SYMAPI_CLIENT_SIDE_ACCESS_ID = ENABLE

The behavior of this option is as follows:

◆ When set to ENABLE, the client sends its access ID to the serverin client/server mode (recommended).

◆ When set to DISABLE (default), the client does not send its accessID to the server in client/server mode.



IMPORTANT

After enabling the above two options, you must then run thesymacl -unique command on the client side to generate theaccess ID and store it in the lockbox on the client side.

Enabling Symmetrix ACLsSYMACLs can be created and configured through the use ofSymmetrix Management Console or Solutions Enabler command lineutilities. SMC is the preferred method for configuration as it offers asimple, wizard-based interface for setting up ACLs. Figure 201 onpage 382 shows how to access SYMACL configuration from withinSMC.

IMPORTANT

Configuring SYMACLs requires the PIN created when theSYMACL database was initialized. This PIN, while customerprovided, can only be set by EMC personnel when the database iscreated. For security reasons, this PIN cannot be changed or viewedwith customer management tools such as Symmetrix ManagementConsole or Solutions Enabler CLI.


382


Figure 201 Configuring SYMACLs with Symmetrix Management Console

EMC SRDF SRAThe following paragraphs present the access controls that were put inplace in the test environment used in this TechBook. This examplecan be used as a guide when implementing Symmetrix AccessControls in VMware vCenter Site Recovery Manager configurationsutilizing EMC SRDF Adapter.

EMC SRDF Adapter acts as a proxy for VMware vCenter SiteRecovery Manager and performs a number of storage array-relatedoperations. These include:

◆ Discovery of the storage arrays attached to the VMware ESXhosts.

◆ Discovery of replicated devices and the state of the replication onthe storage arrays connected to the VMware ESX hosts.

◆ Creation of copies of replicated devices at the recovery site fortesting and other activities.

◆ Failing over from the protection site to the recovery site in case ofa planned or unplanned event.



The Symmetrix Access Controls configuration has to be applied suchthat all of the operations listed above here can be performed on theappropriate set of devices and storage arrays.

This example takes advantage of the alternate access ID method withclient/server mode enabled. This means that the alternate access IDof the Solutions Enabler instance local to the SRM server is the IDused to create the ACLs.

The relevant example environment information is shown below:

◆ Protected Site:

• Solutions Enabler client unique identifier:

2C590AA6-544084C0-9C3F7478

• Symmetrix array serial number 194900281

◆ Recovery Site:

• Solutions Enabler client unique identifier:

2C5C06A6-54488CC0-9C3F747B

• Symmetrix array serial number 192603603

The following process exemplifies how to configure ACLs for theSRDF SRA:

1. Create the Access Groups. Before access rules can be createdaccess groups have to be configured. Access group must becreated on all Symmetrix arrays that are configured to useSYMACLs. Symmetrix access groups contain user-defined servernames and associated unique identifiers of the Solutions Enablerclients. Users can either create two separate groups, one for eachSolutions Enabler instance or put both client IDs in the samegroup. Putting both clients in the same group is simpler as onegroup is easier to manage then two, but the granularity of controlis diminished as both Solutions Enabler clients are forced to havethe same access permissions. The decision to use a shared accessgroup or separate ones depends entirely on the configuration ofindividual environments. Figure 202 shows the process to createan access group with Symmetrix Management Console. Thefigure shows an access group called “SRMGRP” being created onSymmetrix VMAX array 194900281 that contains both the localand remote Solutions Enabler client IDs.


384


Figure 202 Creating SYMACL access group

Figure 202 also shows the access group definitions on theprotection site Symmetrix VMAX array (0281). It can be seen fromthe figure that the group definitions include the unique IDs forthe Solutions Enabler clients on the protection and recovery sitesfor EMC SRDF Adapter. This process should be repeated for allarrays using SYMACLs until every array involved in a SRMprotection group has a group that includes the identifier for boththe remote and local Solutions Enabler clients.

2. Create device pools. The second step of the process is to create adevice pool. For each array, a pool must be created that includesall of the devices that will be used by VMware SRM and the SRDFSRA. This includes SRDF replication devices (R1, R2, R21 etc...)and TimeFinder replication devices (BCVs, VDEVs or Clone



targets). Examining masking records or utilizing VSI StorageViewer can help assist with finding the appropriate devices toinclude in the device pool. It is important to note that gold copydevices may not be presented to the VMware environment soother methods might be required to identify devices targeted foruse in that capacity. Figure 203 shows the creation of a devicepool called “SRMPOOL” that contains R1s, R2s, Clone targets andBCV devices.

Figure 203 Display of devices in the device pool utilized by the SYMAPI server

3. Create the access control entries. Associate the access groupswith access pools and assigning that association privileges. BothSYMAPI servers needs the following privileges on each array:

a. “BASE” access for all devices on the Symmetrix. “BASE”access to all devices is required for the SRDF SRA to be able todiscover the array.


386


b. “BASECTRL” and “RDF” access for only the SRM-relateddevices is required for basic functionality of the SRDF SRA.“BASECTRL” access allows for the SRA to manipulatedevice/composite groups and “RDF” access allows for theRDF control operations required for failover.

c. “BCV” (Optional) If test failover/gold copy functionality withTimeFinder/Clone or VP Snap technologies is desired, “BCV”access for only the SRM-related devices is necessary.

d. “SNAP” (Optional) If test failover/gold copy functionalitywith TimeFinder/Snap technology is desired, “SNAP” accessfor only the SRM-related devices is necessary.

Since “BASE” control is required for all devices and the otherpermissions only require access to the SRM-related subset ofdevices, two different access control entries are required. Bydefault, there is a device pool (named ALL_DEVS) that ispre-created which automatically contains all of the devices on theSymmetrix array. This is the device pool that should be associatedwith “BASE” control for the Solutions Enabler clients. Figure 204shows the ACEs for the previously described exampleenvironment.

Figure 204 Access control entries necessary for the SRDF SRA



It can be noted in Figure 204 that there are two separate ACLs,one with “BASE” access for all devices and a second with “RDF”,“BASECTRL”, “BCV” and “SNAP” permissions for the devices inthe pool which was previously created called “SRMPOOL”.These two ACLs should be created on each Symmetrix array toallow for the proper access for both Solutions Enabler clients.

Three site SRDF (Star and non-Star) and SYMACLsFor environments implementing three site SRDF the requirements forthe protected and recovery site Symmetrix array ACLs are nodifferent than two-site ACL requirements. However, in addition, thelocal and remote Solutions Enabler clients must also be grantedaccess to the tertiary array at the bunker site (for SRDF/Star thiscould be the Asynchronous or Synchronous site, for non-Star SRDFthis will always be the Synchronous site). Even though the SRA willnot recover the workload on the bunker array, the Solutions Enablerclients need “RDF” and “BASECTRL” access to the devices on thetertiary array. Similar to the other arrays, the local and remoteSolutions Enabler clients also require a second ACE on the tertiaryarray that allows them “BASE” access to device pool “ALL_DEVS”.Since the SRA also cannot execute a test failover on the bunker site orcreate gold copies, “SNAP” or “BCV” access is not needed by theSRDF SRA on the bunker array.

A quick evaluation of the ACEs discussed in the precedingparagraphs clearly shows that the Solutions Enabler clients used bythe EMC SRDF Adapter have no authority over and above of which isrequired for support of the four operations listed in the introductionto this section. Therefore, Symmetrix Access Controls can be usedeffectively to address security concerns that are raised by the use ofSolutions Enabler with VMware vCenter Site Recovery Manager andEMC SRDF Adapter.

EMC VSI Symmetrix SRA UtilitiesCertain features of the EMC VSI Symmetrix SRA Utilities issueSYMAPI calls and therefore are affected by SYMACLs and require aminimum level of permissions to function properly. The SRA Utilitiesinteracts with SYMAPI in three different cases:

◆ Test failover pairings configuration

◆ Gold copy pairings configuration

◆ Consistency group creation


388


Gold copy and consistency group creation functionality have noadditional requirements as compared to the SRDF SRA.

The SRA Utilities is running on the same server as the SRDF SRA (theSRM server itself) and is therefore sharing the same Solutions Enablerclient and its respective unique ID. Consequently, the process toconfigure SYMACLs as described in the previous section for the SRAwill also allow the SRA Utilities to function for gold copy andconsistency group creation.

However, the SRA Utilities test failover configuration does requireextra permissions in order to fully function. The SRDF SRA requires“BASE” access for all devices and “BASECTRL”, “RDF”, “SNAP”and “BCV” access for only the SRM-related devices. The SRA Utilitiestest failover configuration additionally requires “VLOGIX” accessbecause the SRA Utilities (unlike the SRDF SRA) examines maskingrecords when searching for valid replica candidates for test failover.This is a unique requirement to the test failover configuration feature.Figure 205 shows the ACEs required for the Solutions Enabler clientin use by the SRDF SRA and the SRA Utilities. In contrast toFigure 204 on page 386, one can note that the only addition is the“VLOGIX”1 ACE.

Figure 205 ACEs required for the SRA Utilities and the SRDF SRA

1. VLOGIX can only be associated with ALL_DEVS. VLOGIX access cannotbe granted to a subset of devices.



Symmetrix AuthorizationsSymmetrix Authorizations provide an additional tool to restrict themanagement operations allowed for individual users or groups in aSymmetrix array. Symmetrix Authorizations and the previouslydiscussed Symmetrix Access Control Lists are independent utilitiesbut can be used together for maximum security.

By using the SYMCLI symauth command or SymmetrixManagement Console, a user or group is mapped to a specific rolethat defines the operations the user or group can perform on an entireSymmetrix array. Authorization is configured independently for eachSymmetrix array.

A role is a predefined set of permissions, or access types, thatdetermines what operations a user can perform. Unlike host-basedaccess control, a user is assigned a particular role for the entireSymmetrix array. Roles are predefined in Solutions Enabler andcannot be modified. For each Symmetrix array, a given user or groupcan only be assigned a single role.

These are the roles defined in Symmetrix Authorizations:

◆ None. Has no rights.

◆ Monitor. Performs read-only operations on a Symmetrix arrayexcluding the ability to read the audit log or Access Controldefinitions or masking records.

◆ StorageAdmin. Performs all management operations on aSymmetrix array. A StorageAdmin can modify the GNS groupdefinitions and monitor all operations (even if only grantedrights to one component).

◆ SecurityAdmin. Performs security operations (symaudit, symacl,symauth, symaccess) on a Symmetrix array in addition to allmonitor operations. Users or groups assigned the SecurityAdminor Admin roles can create or delete component-specificauthorization rules.

◆ Admin. Performs all operations on a Symmetrix array withoutrestriction.

Symmetrix Authorizations 389

390


◆ Auditor. Grants the ability to view, but not modify, securitysettings for a Symmetrix array (including reading the audit log,symacl list, and symauth) in addition to all monitor operations.This is the minimum role required to view the Symmetrix auditlog.

The following authentication types are supported:

◆ D: Indicates a user or group authenticated by a Windows domain.The qualifier specifies the domain or realm name. For example:

• “D:sales\putman” User putman logged in through theWindows domain “sales”.

• “D:jupiter\Sales” Group “Sales” logged in through theWindows domain on host “jupiter”.

◆ H: Indicates a user or group authenticated (by logging in) to somehost. On Windows, this corresponds to logging into “localhost”.The qualifier specifies the hostname. For example:

• “H:jupiter\mason” User “mason” logged in on host “jupiter”.

• “H:jupiter\Sales” Group “Sales” logged in on host “jupiter”.

User authorizations can be added using Solutions Enabler CLI orSymmetrix Management Console. Figure 206 on page 391 shows theprocess to add an authorization using Symmetrix ManagementConsole and Figure 207 on page 391 shows how to add anauthorization with Solutions Enabler. Both figures show the sameexample, but through different methods of adding an authorizationfor the domain user “hostec” in a domain named “EBC” and grantingthe “Monitor” role.

Note that the Solutions Enabler “symauth” command requires thatthe user information and authorization level be entered into auser-created input file and passed into the command via the “-file”option.



Figure 206 Adding Symmetrix authorizations with Symmetrix ManagementConsole

Figure 207 Adding Symmetrix authorizations with Solutions Enabler CLI

If the SYMAUTH database has missing or incorrect credentials, thetarget SYMAPI server will report authentication errors in its SYMAPIlog for each array for which it has local or remote access to and that


392


has Symmetrix Authorizations enabled. These errors will indicate theuser name, group name, syntax and the authorization level requiredfor the SYMAPI request. The error will look similar to the one below:

01/31/2012 10:15:50.781 2736 1368 EMC:VICLIENTPLUGIN

check_user_perms User Authorization Failure [Enforce

Mode] for User D:EBC\hostec, Group D:EBC\Domain Users,

SID 000192603603 -- 'BASE' rights not present (minimum

necessary role: Monitor)

EMC SRDF SRAThe SRDF Storage Replication Adapter supports the user ofSymmetrix Authorizations. For the SRA to properly function, certainauthorization roles are required to be assigned to the user accountrunning the VMware vCenter Site Recovery Manager service. It isimportant to note that the authorization roles should not be assignedto the user account logged into the vSphere Client managing SRM.The account that actually issues the SYMAPI calls to the Symmetrix isthe account which the calls the SRA. The SRA is called by the SRMservice and therefore inherits the user account from the SRM servicerunning on the respective SRM server.

The account that is used by the SRM service can be verified byexamining the SRM service details in the Windows Server “Services”console. The following example shows how to perform thisverification in Windows Server 2008 R2. While there are some minordifferences between Windows Server editions, the basic processremains the same.

1. Figure 208 on page 393 shows the “Run” command windowwhich can be accessed from the Start menu. Enter “services.msc”and select OK. This will bring up the “Services” console.



Figure 208 Launching the Windows “Services” console

2. Search the “Services” console list for the service entitled,“VMware vCenter Site Recovery Manager Server”. Right-clickthis service and choose “Properties”. This is shown in Figure 209.

Figure 209 Locating the SRM service

3. In the subsequent window that appears, select the “Log On” tabto identify the user account running the SRM service. By defaultthe service is configured to execute via the local system account. Ifthis is the case, the radio button for the “Local System account” isselected. Otherwise, the radio button for “This account” isselected and an alternative local or Windows domain account canbe configured. These two options can be seen in Figure 210. In theexample in Figure 210, a special domain account has been created


394


called “srmuser2” from the “EBC” domain.

Figure 210 Identifying user account running SRM service

This process must be followed for each SRM server to correctlyidentify the user accounts that will require Symmetrix authorizationsroles assigned to them. Once the accounts have been identified, eachaccount needs the “Storage Admin” role assigned to it on everySymmetrix array that has Symmetrix Authorizations enabled andthat has replicated devices in use by VMware SRM.

For example, here is a typical environment:

◆ Protected site

• SRM Server name: SRM-SERVER1

• R1 Symmetrix array serial numbers: 194900281, 194900785

• SRM service account: EBC\srmuser1

◆ Recovery site


• R2 Symmetrix array serial numbers: 194903603, 194900474

• SRM service account: EBC\srmuser2

In order for both SRDF SRA’s to function properly the following roleson the following Symmetrix arrays are necessary:

On Symmetrix 194900281:

“Storage Admin” role to user D:EBC/srmuser1













If the local system accounts are running the SRM services the useraccounts entered into the symauth database would instead look likeD:SRM-SERVER1/SYSTEM and D:SRM-SERVER2/SYSTEM for SRMserver one and two respectively.

In the case of three site SRDF, both user accounts will need to beauthorized as “Storage Admin” on the bunker target site array aswell.

EMC VSI Symmetrix SRA UtilitiesThe EMC Symmetrix SRA Utilities for VSI interacts with SYMAPIand is affected by Symmetrix Authorizations in three different cases:

◆ Test failover pairings configuration

◆ Gold copy pairings configuration

◆ Consistency group creation

The requirements for all three are slightly different and will bediscussed separately.

Required authorizations for test failover configurationTest failover uses the user account in which the user logged into thevSphere Client. This is also the account that is used to interact withthe SYMAPI server. Therefore, in order for test failover configurationto properly function in an environment where SymmetrixAuthorizations is enabled, an admin must provide that user accountwith the proper authorization role.


396


In the example in Figure 211, a user has logged into the vSphereClient with domain credentials of EBC.EMC.LOCAL\hostec. Sincethis is a domain account, the credential should be entered into theSYMAUTH database as D:EBC\hostec.

Figure 211 User account for VSI Symmetrix Authorization rules

Test failover configuration requires a role with higher authorizationthen “monitor” even though it is not performing configurationchanges. This requirement is due to the fact that for test failoverconfiguration the SRA Utilities needs to examine masking records toensure that the replica candidates are masked to the cluster.Consequently, since the non-admin roles do not include the ability toexamine masking records in the way that VSI requires, the “StorageAdmin” role is required for any arrays that contain R2 devices thatwill need to be paired with replica devices for test failover.

The “monitor” role will still need to be assigned to the user accountfor all other arrays with Symmetrix Authorizations enabled that arevisible to the target SYMAPI server. If the “monitor” role is notassigned to the user for these other arrays the SRA Utilities will failthe operation and no pairings can be configured (even if these arraysare not involved in any valid SRDF pairs for test failover).


◆ Recovery site SRM Server name: SRM-SERVER2

◆ R2 Symmetrix array serial number: 194903603



◆ User logged into vSphere client: EBC\hostec (domain account).The SYMAPI server also has access to remote Symmetrix arrays194900785 and 194900474; these arrays are not in use by SRM butdo have Symmetrix authorizations enabled.

In order for a user to be able to log on to the recovery site and createtest failover pairings for R2 devices on array 194903603, the followingauthorizations must be granted:

◆ On Symmetrix 194903603:

“Storage Admin” role to user D:EBC\hostec


“Monitor” role to user D:EBC\hostec



Required authorizations for gold copy configurationThe gold copy pairings configuration feature has very similarrequirements to the test failover pairing configurations feature. Goldcopy configuration also uses the user account currently logged intothe vSphere Client to interact with the SYMAPI server. Therefore, inorder for gold copy configuration to properly function in anenvironment where Symmetrix Authorizations is enabled, the usermust provide that user account with the proper authorization role.

Since the gold copy configuration operation does not actually createTimeFinder sessions but instead only searches for proper devices, theuser does not need storage admin permissions. Furthermore, unliketest failover configuration, the basic authorization role of “monitor”will suffice. This difference between gold copy and test failover is dueto the fact that, as mentioned previously, test failover configurationrequires that the SRA Utilities examines the masking records toensure the replica candidates are masked to the cluster. Gold copyreplicas do not have this requirement and as a result the SRA Utilitiesgold copy feature does not need the elevated permissions offered inthe “Security Admin” role.

The “monitor” role will need to be assigned to the user account for allother arrays with Symmetrix Authorizations enabled that are visibleto the target SYMAPI server. If the “monitor” role is not assigned tothe user for these other arrays the SRA Utilities will fail the operationand no pairings can be configured (even if these arrays are notinvolved in any valid SRDF pairs for gold copy configuration).


398



◆ Recovery site SRM Server name: SRM-SERVER2

◆ R2 Symmetrix array serial number: 194903603

◆ User logged into vSphere client: EBC\hostec (domain account)

◆ The SYMAPI server also has access to remote Symmetrix arrays194900785 and 194900474; these arrays are not in use by SRM butdo have Symmetrix Authorizations enabled.

In order for a user to be able to log on to the recovery site and creategold copy pairings for R2 devices on array 194903603, the followingauthorizations must be granted:







Required authorizations for consistency group creationThe authorizations required to allow the SRA Utilities to createconsistency groups are different than that of gold copy and testfailover configuration for two reasons:

1. The user account issuing SYMAPI calls for consistency groupcreation is not the user account that was used to login to thevSphere Client.

2. The user account issuing these calls require higher permissionsthan the user account required for gold copy pairing.

Difference number one is due to fact that the interaction between VSIand the SYMAPI server for consistency group creation is not direct.The Windows Management Instrumentation provider (WMI) is theservice actually issuing the SYMAPI calls. VSI leverages WMI forconsistency group creation and therefore the user account runningthe WMI service is the actual user seen by the Symmetrix issuing theSYMAPI calls. By default, this user account is the local systemaccount on the SRM server. For a SRM server named“SRM-SERVER1” running the WMI provider with the system accountthe SYMAUTH authorization would look like“H:SRM-SERVER1\system”. It should be noted that in rare



circumstances the service could be running with a non-default useraccount. To verify the user account running the WMI service, checkthe “Log On” settings for the WMI service. Figure 212 shows anexample of the WMI service configured to run with the local systemaccount.

Figure 212 Verifying the WMI service user account

Difference number two stated that the consistency group creationfeature requires elevated permissions. This is because this operationperforms more than just querying the Symmetrix forinformation—device/composite groups are created and, in manysituations, deleted as well. The minimum role required to createdevice/composite groups is the “Storage Admin” role.

Additionally, since the consistency group creation operation createsan RDF1 group on the local SYMAPI server and an RDF2 group onthe remote SYMAPI server, more than one account authorization isneeded. Therefore, in addition to granting the “Storage Admin” roleto the local WMI account on the R1 Symmetrix array, it must also begranted the “Storage Admin” role on the R2 Symmetrix array.Additionally, the “Monitor” role will need to be assigned to theaccount running the WMI provider on the remote SRM server for theremote array.


400



◆ Protected site


• R1 Symmetrix array serial number: 194900281

• WMI service account: wmiservice1 (non-domain)

◆ Recovery site


• R2 Symmetrix array serial number: 194903603

• WMI service account: wmiservice2 (non-domain)

In order for a user to be able to log on to the protection site and createconsistency groups the following authorizations must be granted:


“Storage Admin” role to user H:SRM-SERVER1\wmiservice1




“Monitor” role to user H:SRM-SERVER2\wmiservice2

In cases when the consistency group creation operation has to deletean existing composite group (this is not the case for a device group) inorder to create a new group, elevated authorizations are required forthe remote WMI provider user account to be able to query thecomposite group for any existing locks. The “Storage Admin” rolewill need to be assigned (or reassigned as the case may be) to theaccount running the WMI provider on the remote SRM server for theremote array. Using the example environment just described the thirdauthorization listed above would be need to be changed to:



For bi-directional support, assign “Storage Admin” permissions toboth user accounts on both Symmetrix arrays.


A

This appendix presents this topic:

◆ Testing recovery plans using EMC TimeFinder/Mirror ............ 402

Testing Recovery Planswith TimeFinder/Mirror

Testing Recovery Plans with TimeFinder/Mirror 401

402

Testing Recovery Plans with TimeFinder/Mirror

Testing recovery plans using EMC TimeFinder/MirrorVersion 5.0.x and earlier of the EMC SRDF Adapter for VMwarevCenter Site Recovery Manager support the use of EMCTimeFinder/Mirror1 technology to test recovery plans. Starting withversion 5.1 of the SRDF SRA, TimeFinder/Mirror has been dropped.In replacement of Mirror, more flexible and efficient methods such asSnap, Clone or VP Snap should be used.

For users who remain on earlier versions of the SRDF SRA and areusing TimeFinder/Mirror the following appendix describes therequirements and usage of Mirror with Test Failover.

The following steps should be adhered to when testing recoveryplans with TimeFinder/Mirror technology:

1. Ensure the target of the cloning operation is presented to theappropriate VMware ESX hosts at the recovery site. ForTimeFinder/Mirror operations, it is important to check that thecorrect amount of properly sized devices with the BCV identifierare present on the recovery cluster. This can be checked using theTest Failover configuration feature of VSI Symmetrix SRAUtilities as shown in Figure 213 on page 403. VSI discovers all ofthe replicated devices and all of the possible candidates that canbe used for TimeFinder/Mirror replica devices. VSI then makessuggestions for device pairing based on TimeFinder/Mirror rulesfor valid target devices (assuming there are appropriatecandidates available). VSI can discover multiple valid candidatedevices for a given R2 device and will list them in the ReplicaDevice column. This cell is a drop down list that displays allpossibilities and from which one can be selected as seen inFigure 213. VSI will display what it decides to be the mostappropriate device in the top of the column. Priority is given todevice pairings that are already configured for that R2 in the

1. For Symmetrix configurations running Enginuity release level 5874 andSolutions Enabler v7.0 and later, the TimeFinder/Mirror functionsdescribed herein will be performed through TimeFinder/Clone softwareusing a process called Clone Emulation. Clone Emulation mode makes theuse of RAID-protected BCVs transparent to the TimeFinder/Mirror user.

For backward compatibility, TimeFinder/Clone Emulation modetransparently performs TimeFinder/Mirror commands and executesscripts written for Solutions Enabler up through version 6.5.2 running onSymmetrix arrays using Enginuity release levels 5773 and earlier.



options file. Second in priority are pairs that have existingTimeFinder replication sessions. Therefore, if no valid pairs arealready configured in the options file, VSI will next opt to list anycurrently active TimeFinder target devices and their replica state(Split, Synchronized etc.). Users may choose any of the deviceslisted in the dropdown regardless of the priority assigned to it byVSI (these are merely suggestions).

Figure 213 Determining devices pairings with VSI for Test failover TimeFinder/Mirroroperations

2. VSI will include BCV pairings that have a replica state of“Synchronized”, “Synchronizing”, “Split” etc. VSI requires apre-existing BCV relationship, so unassociated BCV devices willbe filtered out and will not be included as valid replica devices.Therefore the initial full establish between the R2 devices and theBCVs must be performed prior to using VSI to configure theoptions file. Although the initial synchronization is a manualoperation1, EMC SRDF Adapter automatically manages the stateof the device pairs for future tests. The following describes thebehavior and requirements of the SRA for test failover andTimeFinder/Mirror:

1. Refer to TimeFinder documentation on support.emc.com for instructionson how to fully establish BCV pairs using SYMCLI or SMC.

Testing recovery plans using EMC TimeFinder/Mirror 403

404


• If some or all of the device pairs are in a “Split” state, theadapter performs an incremental establish on such devicepairs and brings every device pair that was in "Split" state to"Synchronized" state.

• If none of the devices have BCV devices associated at the startof a test failover, the adapter will fail.

• If any of the device pairs are in 'SyncInProg' or 'SplitInProg',the adapter waits until all of the device pairs have enteredtheir final state of “Synchronized.”

3. Once a user has selected the appropriate pairs in VSI, click “SaveOptions File” shown in Figure 213 on page 403. This will save thedevice pairs to the EmcSrdfSraTestFailoverConfig.xml file. Theadapter uses the pairings defined in the options file to determinethe devices to use when a recovery plan is tested. The options filealso reflects the TimeFinder technology to be used when arecovery plans is tested.

4. Figure 214 shows the options file containing the pairings requiredto test a recovery plan. The figure shows the organization of theoptions file and the tag, ArrayID, which enables the adapter tosupport the feature that allows definition of a recovery plan thatinclude multiple storage arrays. Thus, with the SRDF Adapterwhen a recovery plan involves multiple Symmetrix arrays at the



target, a CopyInfo stanza with appropriate TimeFindertechnology, Symmetrix serial number, and device pair definitionsshould be created for each Symmetrix array.

Figure 214 Test failover options XML file for TimeFinder/Mirror

5. After the BCVs used for the test has been fully established, clickthe Test link after selecting a recovery plan to be tested. The EMCSRDF Adapter for VMware vCenter Site Recovery Manager usesthe information provided in the options file and performs theappropriate TimeFinder/Mirror operations.

The testing of the recovery plan is shown in Figure 215. It can be seenfrom the figure that EMC SRDF Adapter has successfully created alocal copy of the datastores associated with the protection groupusing EMC TimeFinder/Mirror technology. VMware vCenter SiteRecovery Manager has also successfully recovered the replica of the


406


datastore on the VMware ESX server(s) at the recovery site andpowered on the replica virtual machine(s) protected by the recoveryplan.

Figure 215 Running a test of a recovery plan using EMC TimeFinder/Mirrortechnology

The state of the BCV pairs when the test environment is running canbe verified by utilizing the Solutions Enabler command, symmir orSymmetrix Management Console. An example screenshot ofSymmetrix Management Console is shown in Figure 216 on page 407.



Figure 216 Determining state of BCV devices during testing of recovery plans

It is important to note that the EMC SRDF Adapter splits theTimeFinder/Mirror pairs only if the state of the SRDF relationship forthe devices involved in the recovery plan is either “Synchronized” or“Consistent.” Without the additional check, it is impossible for theadapter to determine if the source devices for the TimeFinder/Mirrorpairs contain valid data for testing recovery plans. However, there aresituations (for example, enterprise consistency configurations thatcontain data not within a VMware environment) in which users maynevertheless desire to test the recovery plan even though the SRDFlink is in either a “Suspended” or “Split” state. To accommodate this,EMC SRDF Adapter provides the option, TestFailoverForce. Whenthis option is changed from the default value of “No” to “Yes”, theEMC SRDF Adapter ignores the state of the SRDF links and splits theTimeFinder/Mirror pairs. It is the responsibility of the user to ensurethat the source devices in the TimeFinder/Mirror pairs contain aconsistent and valid copy of the data. This setting, like otheradvanced settings can be enabled/disabled using the EMC VSISymmetrix SRA Utilities.

The termination of the recovery plan test is accomplished by clickingthe Cleanup link as discussed previously in this chapter. At thispoint, the EMC SRDF Adapter automatically re-establishes the BCV


408


with the source device. However, as mentioned earlier, the option,TerminateCopySessions, can be used to cancel the relationshipbetween the BCVs and the associated R2 devices so the BCVs can bereused with other recovery plans. If the adapter is set to terminate thecopy session, it will also remove the target devices from thedevice/composite group. If it is not set to terminate the sessions, thetarget devices will remain in the group.

Figure 217 Re-establishing BCV device during Cleanup operation after TestFailover

TimeFinder/Mirror and IgnoreActivatedSnapshots advanced optionAs pervasive as server virtualization has become, certainenvironments have a mixture of virtual and physical servers whichhave dependencies across their hosted applications regardless ofunderlying server architecture. Consequently, during a disasterrecovery test, enterprise consistent copies of application data acrossphysical servers and virtual machines may be required in order forthe test to be a true success.



The default behavior of the SRA does not allow for this to happen. Ifthe customers have consistently split the R2 and BCV devices of thephysical and virtual environment themselves the SRA willincrementally re-establish the VMware-related devices and then splitthem again during test failover. Therefore, the VMware BCV devices,while still internally consistent, will no longer be consistent with theBCV devices external to VMware.

Therefore, version 5.01 of the SRDF SRA introduces new behavior toaccept the already split state of the VMware BCV devices. In order toaddress this need, a new global option "IgnoreActivatedSnapshots"has been added to the config file "EmcSrdfSraGlobalOptions.xml".The default value of this option will be "No". If set to "Yes", theadapter will be tuned to satisfy the this scenario. TheIgnoreActivatedSnapshots option allows the BCV devices to becontrolled externally to SRM. This behavior enables the user toconsistently split the VMware-related BCVs along with any externalBCVs at the appropriate time before executing a test failoveroperation within SRM. The SRA will ignore the already split devicesand report success.

IMPORTANT

This option is only supported with TimeFinder/Mirror with SRDFSRA 5.0.1 and earlier. The SRDF SRA 5.1 supports all methods ofTimefinder with this option with the exception ofTimeFinder/Mirror as support for it has been dropped entirely inthe SRDF SRA 5.1.

The SRDF SRA requires a correctly configured"EmcSrdfSraTestFailoverConfig.xml" file in order to perform the testoperation. This configuration file contains information such as"ArrayId", "Copy Type" and "Device Pairs". The device pair sourcecorresponds to each of the "R2" devices contained in the consistencygroup that is being test failed over. The device pair target representsthe BCV on which the user wants to take the snapshots of the R2devices.

The information in the above options file is critical for SRA in order toperform the test operation. SRA returns results to SRM with thesesnapshot device names along with their respective WWNs. Thisenables SRM to mount the datastores and boot VMs from the targetdevices reported in the SRA response.


410


In order to have the SRDF SRA work in this specific scenario, one offollowing two requirements must be met:

◆ After the consistent split of the BCV devices, a separateconsistency group should be created for VMware related RDFdevices. The XML configuration file"EmcSrdfSraTestFailoverConfig.xml" should be appropriatelyconfigured to contain all the device pairs of (R2, BCV) for the R2devices contained in the consistency group. In this approach,EMC VSI can be used to populate the XML config file.

◆ If a separate consistency group cannot be created for VMwarerelated RDF devices, the XML configuration file"EmcSrdfSraTestFailoverConfig.xml" should be appropriatelyconfigured to contain all the device pairs (R2, BCV) for the R2devices contained in the consistency group. This means that theR2 devices presented to the VMware environment and the R2devices presented to servers external to the VMwareenvironment (and their respective BCV pairs) must be added tothe XML file. Since VSI does not have the capability tomanipulate and configure the XML options file with devices notpresented to the VMware environment, the user must manuallyenter the device pairs to match the device group. The SRM testfailover request contains the name of the consistency group onwhich a specified operation needs to be performed and as aresult, the SRA performs the requested operation on the entireconsistency group as a unit. Since the SRA has no knowledgeabout whether or not the devices in the group are presented tothe VMware environment, the SRA will fail the operation as itwill look for device pairs for all of the R2s in the group.

In the first scenario, the user is responsible to perform an"incremental establish" on the BCV links prior to performing the"Cleanup" operation from within the VMware SRM. If they do not,the SRA will only perform an establish on the VMware devices andnot the external ones since they are not in the device group. If the userre-establishes the BCVs before the “Cleanup” operation (and theIgnoreActivatedSnapshots is still enabled) the SRA will ignore thealready established devices and report success. In the second scenariothe SRA will establish the VMware and non-VMware devices duringthe Cleanup operation so user intervention is not required in thiscircumstance.
