Open Replicator Module

Copyright © 2011 EMC Corporation. All rights reserved.

This module focuses on designing an Open Replicator solution. An overview of Open Replicator, its restrictions and limitations are discussed. Designing an Open Replicator solution based on requirements and data analysis is also covered.

Module 5: Open Replicator Design 1


This lesson covers an overview of Open Replicator concepts and terminology. Configuration considerations, restrictions and use of Open Replicator with Virtually Provisioned devices are also presented. Consideration and requirements for Federated Live Migration are also discussed.



Symmetrix Open Replicator utility with Enginuity 5671 and above provides a method of copying DMX/VMAX volumes to and from various types of arrays (Symmetrix, CLARiiON and other qualified 3rd party arrays) within a SAN (Storage Area Network) infrastructure.

For example, Symmetrix Open Replicator provides a tool that can be used to migrate data from older Symmetrix arrays, CLARiiON, and certain third-party storage arrays. Alternatively, the Open Replicator utility can also be used to migrate data from a Symmetrix DMX/VMAX array to other types of storage arrays within the SAN infrastructure. Copying data from a Symmetrix DMX/VMAX to devices on remote storage arrays allows for data to be copied fully or incrementally.

3 Module 5: Open Replicator Design


There are two Open Replicator Licenses – Open Replicator/DM (which includes all features except “Hot Pull” and “Restore”) and Open Replicator/LM (for “Hot Pull” and “Restore”).

Starting August 2009 Open Replicator/LM is available to new and existing customers of DMX/VMAX at no charge as part of the Symmetrix Migrator software package.

The no-charge Symmetrix Migrator package includes Open Migrator, Open Replicator/LM and SRDF/DM. Customers have been purchasing and using these products to migrate host and array data for years, and now they can do so at no charge. Use this packaging and pricing change as an opportunity to discuss your customer’s current and future migration requirements; including what migration activities they would prefer to perform on their own versus those requiring the added expertise of an EMC delivered Migration service.



In Open Replicator, the DMX/VMAX array performing the replication operations is called the Control Array. Data can be moved to or from devices on the controlling array to or from a remote array. The remote array can be a Symmetrix, a CLARiiON or any of the qualified 3rd party arrays.

The devices in the Control array are called Control devices. For every control device, there is a counterpart (a remote device) on the remote array.

The names Control or Remote do not indicate the direction of data flow, they only indicate the array which is performing the replication operation.

The data movement could be from the control array to the remote array or vice-versa. The direction of data movement is determined by the Open Replicator replication operation. We will discuss the Open Replicator replication operations in the next slide.



Please note that the Control Arrays are shown on the right hand in the graphics on the slide.

Open Replicator Operations: The replication operations are with respect to the Control Array.

Push: A Push operation indicates that data is being Pushed from the Control Array to the Remote Array. Thus in a push the Control Device would be the Source while the Remote device would be the Target. The application that needs to be replicated would be running on devices on the Control Array. A push operation can be Incremental, i.e. only changed tracks since the last activate are copied over to the remote device.

Pull: A Pull operation indicates that data is being Pulled to the Control Array from the Remote Array. In a pull the Remote Device would the Source and the Control Device would be the Target. The application that needs to be replicated would be running on the Remote Array.



The Push/Pull operations can be either Hot or Cold. Hot or Cold is used to refer to the state of the Control device during the push/pull operation. The remote devices should not be accessed by a host during either push or pull operations. In Cold operations, the control device is made host inaccessible (Not Ready). In a hot operation, the control device is allowed to be online for host operations.

The reason for not allowing host operations on the remote devices is due to the fact the Control Array has no control over the remote array and cannot keep track of any changes on the remote device. Data integrity cannot be guaranteed if changes are made to the remote device during push/pull operations.

Hot operations, where the changes can be made to the control device during push/pull operations, is possible because the Control Array can keep track of all changes and thus ensure data integrity.

Cold operations guarantee consistency because both the control and remote devices are offline to any host operations.

For pull operations from devices with SCSI reservations, if the remote devices have a cluster running against them or the devices are AIX LVM devices, you must shut down the cluster, AIX host or other software that is creating the SCSI reservations before creating the Open Replicator Session.



The Push/Pull operations can be either Hot or Cold. Hot or Cold is used to refer to the state of the Control device during the push/pull operation. The remote devices should not be accessed by a host during either push or pull operations. In Cold operations, the control device is made host inaccessible (Not Ready). In a hot operation, the control device is allowed to be online for host operations.

The reason for not allowing host operations on the remote devices is due to the fact the Control Array has no control over the remote array and cannot keep track of any changes on the remote device. Data integrity cannot be guaranteed if changes are made to the remote device during push/pull operations.

Hot operations, where the changes can be made to the control device during push/pull operations, is possible because the Control Array can keep track of all changes and thus ensure data integrity.

Cold operations guarantee consistency because both the control and remote devices are offline to any host operations.

For pull operations from devices with SCSI reservations, if the remote devices have a cluster running against them or the devices are AIX LVM devices, you must shut down the cluster, AIX host or other software that is creating the SCSI reservations before creating the Open Replicator Session.



If left unchecked, Open Replicator will consume the full bandwidth of the SAN. However, this can be a problem if the SAN is also being shared by hosts performing I/O to the local volumes on the controlling DMX/VMAX.

Controlling the data transfer rate with either the pace or ceiling settings allows users to limit the SAN bandwidth used by Open Replicator.



The Protection Bitmap is counted as one of the 16 SDDF Sessions.

Incremental push operations use an additional SDDF session. Open Replicator is limited to a maximum of 15 incremental sessions per device.

SDDF sessions are used to monitor changes in BCVs, Clones, Snaps, Change Tracker, Open Replicator and SRDF/Star.



As soon as a pull or push operation is created, a protection bitmap is created by the control array to keep track of the replication process. All tracks on the control devices are marked as protected. When the replication operation is created all the bits are set to “one” indicating that all the entire contents of the source volume needs to copied over to the target device. As the replication process copies data, the bits are flipped over to “zero” indicating that a particular track has been copied over. At the end of the replication process, all the bits will be zero. The protection bitmap takes up one of the 16 SDDF sessions that a device is allowed.



Open Replicator can only keep track of changes made to the control devices, thus incremental operations are allowed only for push operations.

The changes are tracked by using SDDF bitmaps. All the bits in the SDDF bitmap are set to zero when the push operation is initiated. As changes are made to the control devices, the bits are flipped from zero to one indicating that changes have occurred. When re-synch push has to be done, then the SDDF bitmap gets copied to protection bitmap and the SDDF bits are zeroed out, and only those tracks that changed would have to be sent over to the remote device.

A very important point to note here is that if changes are made to the remote device, Open Replicator would be unaware of these changes and thus data integrity cannot be ensured if an incremental push is performed.



For incremental push operations only, data can be restored back to the control device by pulling back only the changed tracks from the remote device. The session must have been created using the -differential option and must be in the copied state. Hot or cold differential push sessions can be restored.

The restore functionality requires a Solutions Enabler product license for Open Replicator/LM (Online Pull).

For example, if you copied all data from the control device to the remote device(s) and then made changes to the control device, you could then recover the original data from the remote device by using the symrcopy restore command. When the command is issued, the session is recreated in restore mode and automatically activated. At the start of the restore operation, all control devices will be set to Not Ready status. If running a hot session, control devices will be returned to Ready status at the end of the operation (as the data begins copying). If running a cold session, the control devices will remain in Not Ready status.

If a restore is being performed to a hot push session, new writes to the control are only allowed after the appropriate track has been restored.



The SAN for the remote storage array must have connectivity to the control Symmetrix SAN. Open Replicator requires that at least one port on the remote array that allows access to the remote device have access to the control device through at least one port for a cold copy and all ports for a hot copy on the control array.

In Open Replicator the Control array front end ports act as HBAs from the perspective of the remote array. Front end ports on the control and remote arrays have to be zoned to each other. In addition LUN masking operations have to be performed on the remote array – the control array ports must be allowed access to the remote devices via the front end ports on the remote array.



During push/pull activities, if a host attached to the remote array writes to the devices during an Open Replicator session, data corruption to devices may occur. It is highly recommended that the remote device be un-mounted by hosts attached to the remote array and made unavailable for host writes during the copy process.



Restrictions of Open Replicator are listed. Open Replicator within the same array is not allowed. Only FBA devices can participate in Open Replication copy sessions. From 5874 and 5773.150 onwards AS400 devices are supported as long as the remote device is a Symmetrix device.

Maximum Open Replicator active sessions for Enginuity 5772 or earlier is 512. The default is 512 active sessions. Maximum active sessions for Enginuity 5773 or later is 1024. The default is 1024 active sessions.



For hot push sessions, there is a delay in completing a write to a protected track until it has been moved to the remote device. This penalty, paid for the first write to a protected track only, is referred to as the COFW (Copy on First Write). Subsequent writes to the same track on the control device occur without any delays.

Use of the precopy option (introduced in 5772) with the create or recreate commands will initiate data copy immediately in the background before the session is activated. Precopy can be used to minimize the COFW penalty. The idea is to use precopy when the session is created and then activate the session at a later time. Thus, the number of protected tracks would be less and hence the probability of writing to a protected track after activation would be reduced.

Starting with 5874 (VMAX) TimeFinder/Snap Virtual Devices (VDEV) can be used as the Control device for Cold Push operations. Only one Open Replicator session is allowed per VDEV control device.



For hot pull sessions, there is a delay when completing any I/O (read or write) to a protected track until it has been copied over from the remote device. This Copy on First Access penalty is paid only on a first read or write to a protected track. Subsequent I/Os to the same track on the control device occur without any delays.

Without donor update, there is a possibility of data loss if the hot pull session fails during the migration process. If donor update is not used, data is not written to the originating array and any updates to the data made during the active session will be lost.

The use of Donor Update will add an additional response time elongation to all writes. The penalty for writes to protected tracks is the worst because one has to factor in the COFW penalty, in addition to the synchronous response time elongation.



Thin devices can be used as control devices for all Open Replicator copy operations. If a push operation is done using a thin device as the source, zeroes will be sent for any regions of the thin device that have not been allocated, or that have been allocated but have not been written to.

Open Replicator can also be used to copy data from a regular device to a thin device. If a pull or push operation is initiated from a regular device that targets a thin device, then a portion of the target thin device, equal in size to the reported size of the source volume, will become allocated. Space on the fully allocated thin device can be reclaimed with virtual provisioning space reclamation (requires 5874 Q4 2009 SR).



Point-in-Time Remote Data Distribution and Migration:

• (Distribute test and development copies) Single BCV can be distributed to up to 16 SAN attached systems, and get incremental updates from hot device.

• (Migrate an application to tier-2 storage) Improve utilization of tier-1 investments. If the application becomes more critical, it is bi-directional to migrate back to tier-1.

• (Centralize backup and restore) Eliminate the need for backup resources for each campus environment (remote Backup to Disk).

Point-in-Time Remote Vaulting:

• Create a BCV from the standard; production never stops on the standard

• Push initial copy to remote location at 12PM

• Update changes and Push incremental updates offsite to vault at 6AM

• Utilize tier 2 and 3 storage within the SAN

• Improve recovery time objectives vs. tape

Cold Push from a Snap/BCV/Clone copy can avoid primary application writes performance penalty:

• This is true because there is no COFW penalty

• However, since the same FA supporting other devices will be performing the push - there will still probably be some performance penalty to any application accessing devices through the same FA.



Full Volume Recovery from Vault:

• Declare disaster, determine location of new primary site

• Pull 6:00 copy from tier-2 storage back to DMX

• Restart application

Hot Recovery from Remote Copy:

• Create empty volume for recovery

• Start remote copy, data transfer begins

• Start applications

Data not yet transferred is retrieved on first access

High-Speed Online Data Migration:

• Symmetrix DMX/VMAX installed between hosts and original storage

• Create empty volumes for migration

• Start remote copy, data migration occurs while application remains online

Data not yet transferred is retrieved on first access

• When all data is migrated, unplug and remove original storage



Federated Live Migration ties together the array-based migration of the data, provided by EMC Open Replicator for Symmetrix, with the host-level application redirection, provided by EMC PowerPath. It does this by using a set of coordinated commands through EMC Symmetrix Management Console or SYMCLI to initiate the migration session and coordinate the host application redirection from one central point, making the migration truly non-disruptive. Additionally, Federated Live Migration supports a number of pre-qualified stacks of arrays, PowerPath, and host operating systems that help eliminate time-consuming remediation processes. Federated Live Migration is flexible. It is capable of supporting combinations of migrating thick-to-thick, thick-to-thin, and thin-to-thin as well as consolidating multiple systems to one Symmetrix.

The first version of Federated Live Migration supports migrating from Enginuity 5671, 5773, 5875 with PowerPath 4.5 or higher to Symmetrix VMAX with Enginuity 5875. Consult EMC Symmetrix Procedure Generator, EMC Federated Live Migration Simple Support Matrix (located on http://powerlink.emc.com) for supported multi-pathing software, operating systems, file systems and logical volume managers.



FLM has unique requirements that must be met before a migration is attempted. It is sensitive to errors in preparation and procedure. Procedures vary from Operating System to Operating System so you should follow the process as documented for your particular environment.



Federated Live Migration (FLM) operates by having the new VMAX device assume the identity and geometry of the donor Symmetrix DMX device and then performing an Open Replicator hot pull operation as the current data movement method between new and donor arrays. The new VMAX device must be equal or larger than the donor DMX device for the migration operation to be allowed.

The donor storage must be a Symmetrix and the donor device cannot be involved in any type of local or remote replication. This restriction is necessary to ensure data integrity on the new device as an Open Replicator pull session will not be able to copy consistent data if new data is written to the donor DMX device while the session is running.

There is some management overhead involved in identity and geometry spoofing. That is why EMC recommends that native identities be restored to the VMAX devices at the user’s convenience soon after completion of data migration.

Restoring native device identity is a disruptive process. After completion of the migration and termination of the FLM session, the host application must be taken off-line or the application host must be shutdown. Then the native identity and geometry can be restored. When the application is brought back on-line or the application host is rebooted, the VMAX device will now be accessed by it’s original identity.



As the migration (and underlying Open Replicator technology) is SAN based, a Fibre Channel switch is required. Direct connections are not supported. As with other Open Replicator SAN design, the control array (the new VMAX) should be given access to the donor DMX devices. This will involve the appropriate VCM (symmask) entries in the donor DMX. The migration create operation will fail if the new VMAX device is already a part of any Auto Provisioning Masking View. After successful creation of the session, the new VMAX device should be made visible to the application host.



These two host access modes have been introduced in Enginuity 5875. Note that even after successful migration and termination of the FLM session, the donor DMX device will remain in host_passive mode. After restoring the native device identity to the new VMAX device, the donor DMX device can be set to host_active mode. This can be done using SYMCLI at Solutions Enabler 7.3 and above. Prior version would require contacting EMC support to set the donor DMX to host_active. When planning to re-use the donor DMX device, the new VMAX device and the donor DMX device either must not be on the same SAN, or the native identity of the new VMAX device must be restored.



The FLM terms source and target are a departure from the Open Replicator terms Control and Remote. Since FLM is intended for use by customers who may not be familiar with Open Replicator, the term source is used to identify the DMX donor devices which will be the remote devices during FLM migration. The term target is used to identify the new VMAX control devices.

FLM introduces a new distinction called host access mode. This is layered on top of the traditional device states such as Ready or Not Ready. Active mode is the normal device access mode with the device fully visible to the host. Passive mode is what FLM uses to make an otherwise ready device not visible to the host. Supported host multipath I/O drivers such as PowerPath recognize the active and passive modes.

Symmetrix arrays present a unique device identity for each host visible Symmetrix Logical Volume. The identity is made up of WWN, front end Director Path(s) and the device geometry. During the course of the migration FLM changes the native identities on the target to resemble the source device identities. The spoofed identity can be recognized by the fact that the director ports are offset by 2. Thus, in this example the VMAX device is reported on port 7E:2 instead of 7E:0.



Some of the salient considerations for FLM migration are listed on this page.



For redundancy and performance, multiple FAs are required for FLM. If the donor DMX devices are accessed as a cluster resources, they might have SCSI reservations on them. The cluster must be disabled to perform FLM for these devices. Also if the donor DMX device is a part of an AIX Volume Group, the VG must be varied off or the host shut down to perform the migration. In general any application/software that creates SCSI reservations on the donor DMX devices must be shut down for FLM.



This lesson covers the methodology for designing an Open Replicator solution. An example of sizing an Open Replicator solution is also presented.



With all forms of replication, risk mitigation revolves around planning. With SAN and array based solutions, there are numerous components involved. To effectively mitigate the risks, all parts of the solution need to be assessed and potential bottlenecks identified and resolved.

Open Replicator can generate a great deal of activity that may impact host activities on the control or remote storage arrays if it is started without any restraints. It is important to review the control frame and remote arrays current performance to determine available resources for Open Replicator. The Control DMX/VMAX and each remote array’s front-end utilization must be reviewed. During the modeling process, SymmMerge or Performance view can be used to identify potential issues.

SAN based replication can add a large amount of traffic. If unchecked, the traffic can saturate the backplane on older switches. If the session runs over Inter Switch Links, the traffic may swamp the link. During the discovery process, the native switch tools can be used to evaluate current traffic load to prevent Open Replicator from interfering with other SAN activities.

Open Replicator only functions over Fibre Channel. To replicate data across distance, a Fibre Channel distance extension solution must be in place. EMC supports multiple vendors and each has benefits and limitations. Please consult the latest version of the EMC support Matrix to insure the customers solution is supported. A network assessment is recommended in order to validate the current or proposed network infrastructure, bandwidth capabilities, packet loss and roundtrip latencies. The resulting assessment report is required by the Solutions Validation Center and can be used during the modeling and design process.



An overview of the Open Replicator modeling process is shown.



Currently there is no single tool for modeling Open Replicator. However, a sizing tool located in the GS Tools eRoom can provide an estimated push or pull transfer time. The tool can be downloaded from the GS Tools eRoom:

EMC TS Applied Technologies → Applied Technology Practice Areas → GS Tools → Applied Technology Practice Tools → Data Migration → Open Replicator Sizing.

Currently the only approach available is to use SymmMerge or WLA Performance View for the Symmetrix DMX/VMAX and Symmetrix 8000 series and Navisphere Analyzer for CLARiiON. For supported third party arrays, their native tools must be used to gather the required information. Also, if the customer has EMC ControlCenter installed, Performance Manager can be used for any of the supported components.



To accurately model the solution, data collection is required for each component identified in the data path.



It is important to identify the specific devices and front end ports that will be participating in the Open Replicator sessions.



The data input for the sizing spread sheet is determined from the data that was collected (e.g. FA Utilization can be determined from the BTP data).

The slowest component in the data path will determine the copy time. A pair of OC-12 links is a large data pipe in networking terms. But when compared to Fibre Channel, they are about the same size as a single link. Most Open systems servers have multiple fibre channel links for throughput and redundancy.



Before you start, make sure you have collected the appropriate data for the control and target arrays. Validate the data path and determine if Interswitch links will be crossed. If links will be crossed, verify the available bandwidth.



Review the utilization of processors 5A, 6B on the Remote and 4A and 4B on the Control. STP Navigator shows that the worst case for the remote directors is that they are at least 75% free. The control directors are at least 95% free. Use this data in the OR Spreadsheet.

Note: In this example, the FA utilization has been determined using STP Navigator. The preferred method is to use SymmMerge to determine utilization data.



This is a simple tool to help you determine the amount of time a transfer will take and the gating factors. The highlighted fields are data entry fields supplied by the user.



By looking at the time chart, you are able to determine the gating factor. In this case, the Target (Remote) frame is the gating factor.



For each of the FA processors on the control frame, reduce the available percentage until the Control DMX time is equal to, or a bit less than, the transfer time. Save the information for inclusion into the technical specification.

In this example, we could set a ceiling value of 50% on the Control DMX FA ports and still achieve a transfer time of 1:40:07.



Walk through the model and ensure that any and all potential risks are identified. Create a list of potential resolutions for all risks that have been identified. Generate a technical specification; it will need to be referred to and updated throughout the design process.



This module covered designing an Open Replicator solution. An overview of Open Replicator, its restrictions and limitations were discussed. Designing an Open Replicator solution based on requirements and data analysis was also provided.


Copyright © 2011 EMC Corporation. All rights reserved. Module 5: Open Replicator Design 44



Documents

Open Replicator Module