BCO2155-vCloud DR for Oxford University Computing Services - Real World Example_Final_US.pdf

7/27/2019 BCO2155-vCloud DR for Oxford University Computing Services - Real World Example_Final_US.pdf

1/44

vCloud DR for Oxford

University ComputingServices - Real WorldExample

Gary Blake, VMware, Inc.

Aidan Dalgleish, VMware, Inc.

Adrian Parks, Oxford UniversityComputing Services

INF-BCO2155

#vmworldinf


2/44

2

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibi lity and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.


3/44

3

Introductions

Aidan Dalgleish

Consulting Architect, VMware Global Center of Excellence

VCDX#010

Twitter: @AidersD

Gary Blake

Senior Consultant, VMware NEMEA PSO VCAP-DCA4 / VCAP-DCD4

Twitter: @GaryJBlake

Adrian Parks

Senior Systems Administrator, Oxford University Computing Services (OUCS)

VCP5

Blog: http://blogs.oucs.ox.ac.uk/oxcloud
http://blogs.oucs.ox.ac.uk/oxcloudhttp://blogs.oucs.ox.ac.uk/oxcloud


4/44

4

Agenda

Terminology

vCloud DR

Challenges

Solution Overview

OUCS Overview

OUCS vCloud DR

Challenges

Solution Overview

Tips and Tricks Key Takeaways

Questions


5/44

5

TerminologyLets clear up a few things


6/44

6

Terminology

Provider Recovery

The vCloud management components

In principle individual VMs but may also include vShield Edge appliances

Consumer Recovery

Workloads deployed by the consumer(s)

In principle vApp(s) but may also include vShield Edge appliances

DR of the Cloud (referred to as vCloud DR from herein)

Recovery of a vCloud solution from a protected site to a recovery site

Managed by the provider

DR to the Cloud

Recovery of a workload from a consumer site to a provider recovery site

vSphere to vSphere, SRM to SRM

Workloads not hosted by vCloud Director (vCD)

Out of Scope forthis discussion


7/44

7

vCloud DR ChallengesI hope you are sitting comfortably


8/44

8

vCloud DR Challenges

vCloud Director limitations

Correlates vSphere resources using Managed Object References (MoRef)

No resilience to external change of MoRef

Site Recovery Manager support

No integration between vCloud Director and Site Recovery Manager (SRM)

Maps resource, network and storage between vCenter Server(s) (MoRef change)

Resignature of VMFS Datastore(s) (MoRef and UUID Change) Necessitates re-registering of VMs (MoRef change)


9/44

9

vCloud DR Solution OverviewDespite all the challenges, we have a solution!


10/44

10

vCloud DR Solution Overview

Management Cluster

Protected using SRM and storage replication (requires support by SRM)

Resource Cluster(s)

Native storage replication used to protect VMFS Datastores (no SRM)

Better suited to implementations with stretched layer 2 networking

Supported by GSS and CPD

Products are being used as intended

Validated by QE!

Internally agreed as the 2012 solution


11/44

11


Limitations and Considerations

Granularity currently restricted to that of a Provider Virtual Datacenter

Could use Resource Pool based Provider vDCs (not tested deviation from vCAT)

Potential future capabilities with later releases of vCloud Director

Becomes complex in the absence of stretched layer 2 networking

Address changes in virtual machines

Impact on vShield Edge firewall rules


12/44

12


Management Cluster Failover Procedure

Execute the SRM Recovery Plan to recover management components

Specific virtual machine recovery order

SRM Priority Groups

SRM Virtual Machine Dependencies

Defined recovery order:

Databases Server(s)

vCenter Server(s)

vShield Manager

vCloud Director Cell(s)


13/44

13


Overall Resource Cluster Failover Process

Present the replicated volumes to the recovery hosts

Force Mount VMFS datastores on recovery hosts

Remove the recovery hosts from Maintenance Mode

Power On recovered workloads (VMware HA)

If enabled, VMware HA will initiate this automatically

FailoverManagement

Cluster SRM

PresentReplicatedVolumesManual/Automated

Mount VMFSVolumes

Manual/Automated

Disable VMwareHA

Manual/Automated

vApp Power On

Manual/Automated

Enable VMwareHA

Manual/Automated

Exit MaintenanceMode

Automated

VM Power On

VMware HA


14/44

14


Overall Resource Cluster Failover Process

Present the replicated volumes to the recovery hosts

Force Mount VMFS datastores on recovery hosts


Power On recovered workloads (vSphere & vCloud API)

Disable VMware HA Default Restart Priority

Register VMs on recovery hosts Set UUID Action Option Value

Initiate Power On requests to vCloud Director API

Restore VMware HA Default Restart Priority

FailoverManagement

Cluster SRM

PresentReplicatedVolumesManual/Automated

Mount VMFSVolumes

Manual/Automated

Disable VMwareHA

Manual/Automated

vApp Power On

Manual/Automated

Enable VMwareHA

Manual/Automated


Automated

VM Power On

VMware HA

Map vApp(s)Map Organization

vDC(s)Map Provider

vDC(s)Power On vShield

Edge(s)

Register VirtualMachines & Set

UUID Action


15/44

15

OUCS OverviewJust in case you havent heard of Oxford University


16/44

16

OUCS Overview

English-speaking worlds oldest

university

Founded in the 11th century

38 independent colleges, 21,000

students

Over 300 independent units

Devolved IT infrastructure

No central mandate


17/44

17

OUCS vCloud DR ChallengesWhat were the specific challenges for OUCS ...


18/44

18

OUCS vCloud DR Challenges

Primary Goal

Protect Shared Datacenter Virtual Infrastructure Environment from site failure

Secondary Goals

Implement additional application availability checking for Provider workloads

Offer an Active/Active failover design for two Resource Clusters

Automate the vCloud Director Resource Cluster(s) recovery

Allow prioritization of Organization Virtual Datacenter(s) during recovery

Honor any boot priorities set within vApp(s) during recovery

Workload Categories

Virtual Datacenter Service

Hosted Virtual Machine Service DaaS (Database as a Service)


19/44

19

OUCS vCloud DR Solution OverviewHow was the vCloud DR solution applied in the real world...


20/44

21

OUCS vCloud DR Solution Overview

Management Cluster Failover Procedure

Execute the SRM Recovery Plan to recover management components

Controlled recovery order:

Priority 1 VMs

DNS Server

vCenter / Update Manager Database Server

vCloud Director Database Server

Priority 2 VMs

vCenter Server

vShield Manager Server

vCloud Director Cell

Priority 3 VMs

Chargeback Server

Recovery Plan Prompts (Pause the recovery plan)


21/44

22


Additional application availability checking

PowerShell Advanced Function Test-WindowsServiceStatus

Microsoft SQL Server

DNS Server

vCenter Server

PowerShell Advanced Function Test-LinuxServiceStatus

vCloud Director Service


22/44

25

Resource Cluster (Datacenter) vCloud DR Configuration

USDC

Provider vDC

Provided by stretched cluster but second OUCShosts in maintenance mode

Storage VMFS synchronous replication to OUCS

OUCS

Provider vDC

No services running Single host in maintenance mode

Storage No replication to USDC

Resource Cluster (Live) Resource Cluster (Maintenance Mode)Stretched

A B A B

Asynchronous Replication with Active Replay

Provider Virtual Datacenter (Protected)

vApp #2

WRK#1WRK#1WRK#1WRK#1WRK#1WRK#1VM#1 VM#n

vApp #1


Org vDC #1 (Protected)

MAN

Org vDC #1 (Protected)


23/44

26

Resource Cluster (Campus) vCloud DR Configuration

USDC

Provider vDC

No services running Single host in maintenance mode

Storage No replication to OUCS

OUCS

Provider vDC

Provided by stretched cluster but second USDChosts in maintenance mode

Storage VMFS synchronous replication to OUCS

Resource Cluster (Maintenance Mode) Resource Cluster (Live)Stretched

A B A B

Asynchronous Replication with Active Replay

Provider Virtual Datacenter (Protected)

MAN

vApp #2


vApp #1


Org vDC #1 (Protected) Org vDC #1 (Protected)


24/44

27


Resource Cluster(s) Failover Procedure

Break replication and present the replicated volumes to the recovery hosts

Advanced Function Invoke-CompellentFailover (Compellent Cmdlets)

Force Mount the VMFS datastores on the replicated volumes

Advanced Function Start-HBARescan (vSphere Cmdlets)

Advanced Function Set-VmfsForceMount (vSphere Cmdlets)


Advanced Function Set-VMwareHADisabled (vSphere Cmdlets)

Advanced Function Set-VMHostsConnected (vSphere Cmdlets)

Power On recovered workloads (vSphere and vCloud API)

Advanced Function Register-VmxFilePaths (vSphere Cmdlets)

Advanced Function Set-VMOptionValues (vSphere Cmdets)

Advanced Function Set-vShieldEdgesPowerState (vSphere Cmdlets)

Advanced Function Start-OrgvDC (vSphere and vCloud Director Cmdlets)

ManagementCluster Failover

PresentReplicatedVolumes

Mount VMFSVolumes

Disable VMwareHA


VM Power On


25/44

28


Advanced Function Invoke-CompellentFailover

Description

Connect to Compellent array(s) and failover replicated volumes

Process

Validate SRM Recovery Mode (mandatory) and parameters (optional)

Attempt connections to default and destination arrays (Recovery vs Migration)

Identify replicated volumes (using search string)

Default array tasks (if present)

Disable replication jobs and remove source volumes mappings

Destination array tasks

Map replicated volumes to recovery ESXi Server hosts



Mount VMFSVolumes

Disable VMwareHA


VM Power On


26/44

29


Advanced Function Start-HBARescan

Description

Force an HBA rescan on all ESXi Server hosts within a defined cluster (Provider vDC)

Process


Initiate an HBA rescan on the identified ESXi Server hosts



Mount VMFSVolumes

Disable VMwareHA


VM Power On


27/44

30


Advanced Function Set-VmfsForceMount

Description

Identify un-mounted VMFS Datastores (UUID Conflict) and force mount

Process


Individually connect to each ESXi Server host and perform the tasks:

Identify unresolved VMFS Datastores

Resolve the UUID Conflict and Force Mount



Mount VMFSVolumes

Disable VMwareHA


VM Power On


28/44

31


Advanced Function Set-VMwareHADisabled

Description

Set the Default Restart Priority for a defined cluster to Disabled

Process


Update virtual machine Default Restart Priority for the defined cluster to Disabled



Mount VMFSVolumes

Disable VMwareHA


VM Power On


29/44

32


Advanced Function Set-VMHostsConnected

Description

Remove recovery ESXi Server hosts from Maintenance Mode in a defined cluster

Process


Determine desired state for ESXi Server hosts (Maintenance Mode or Connected)

Determine current state of ESXi Server hosts

Adjust connection states of ESXi Server hosts accordingly



Mount VMFSVolumes

Disable VMwareHA


VM Power On


30/44

33


Advanced Function Register-VmxFilePaths

Description

Register Inactive virtual machines on ESXi Server hosts within a defined cluster

Process


Identify inactive virtual machines and record a subset of their properties

Identify ESXi Server hosts that are in a Connected state

Establish a random distribution plan for the virtual machines

Individually connect to each ESXi Server host and perform the tasks:

Register a subset of virtual machines on the ESXi Server host



Mount VMFSVolumes

Disable VMwareHA


VM Power On

OUCS Cl d DR S l i O i


31/44

34


Advanced Function Set-VMOptionValues

Description

Set option value on virtual machine(s) within a cluster

Process


Identify virtual machines within a defined cluster

Set option value key/value pair (uuid.action = keep)



Mount VMFSVolumes

Disable VMwareHA


VM Power On

OUCS Cl d DR S l ti O i


32/44

35


Advanced Function Set-vShieldEdgesPowerState

Description

Change the power state of vShield Edge devices used for Organization Networks

Process


Identify vShield Edge appliances for Organization networks (exclude vApp networks)

Update the power state of the vShield Edge appliance as required



Mount VMFSVolumes

Disable VMwareHA


VM Power On



33/44

36


Advanced Function Start-OrgvDC

Description

Power on the consumer workloads running within a defined cluster

Process


Identify the defined cluster and derive the associated Provider Virtual Datacenter

Identify Organization Virtual Datacenters within the Provider Virtual Datacenter

Identify vApps within the Organization Virtual Datacenter

Read Restart Priorities on the Organization Virtual Datacenters (Metadata)

High, Medium, Low, Undefined, or Disabled

Read Restart Priorities on the vApps (Metadata)

High, Medium, Low, Undefined, or Disabled

Commence Power On process accounting for any defined Restart Priorities



Mount VMFSVolumes

Disable VMwareHA


VM Power OnVM Power On



34/44

38


Limitations

No support for SRM in Test Mode

Would require extensive networking changes in Resource Cluster(s) Not tested/evaluated during the original vCloud DR solution development

In the event of a Failover (Recovery Mode) Compellent failover is manual

Limitations of Compellent PowerShell Cmdlets and/or Storage Center

Replication jobs owned by default array

In event of failover (Recovery Mode) default array is inaccessible

Risk of replication resuming if failed site comes back online

Currently cannot identify previous running state of vApp(s)

Power On decision based purely on defined Restart Priority


35/44

39

Tips and TricksWhat did we learn ?

Ti d T i k


36/44

40

Tips and Tricks

PowerShell Automation

Incorporate extensive initial validation of parameters

Include extensive logging throughout

Confirm every task with a corresponding action (Continue/Stop/Warn)

Define all exit points with a defined exit code of 0/1 (Success/Fail)

Ensure any scripts can be run multiple times over

Upon failure SRM re-runs the entire Recovery Plan

Design for Planned Migration, the rest will come

Recovery is effectively Planned Migration with appropriate error handling

Failback is effectively the same as Planned Migration but reversed

Tips and Tricks


37/44

41

Tips and Tricks

PowerShell Configuration

PowerShell Profile

Required to define loading module files (.psm1) Add additional Cmdlets

Module Configuration

Located in C:\Windows\SysWOW64\WindowsPowerShell\v1.0\Modules

Create folder to store .psm1 file the same name as the module file

(example: vCloudDR.psm1 stored in ..\Modules\vCloudDR folder)

Tips and Tricks


38/44

42

Tips and Tricks

SRM Configuration

SRM 5.x still a 32-bit Application so 32-bit PowerShell must be used

C:\Windows\SySWoW64\WindowsPowerShell\v1.0\Powershell.exe

SRM Command Syntax

Use full paths to all executables

Use of quotes (Single/Double)

SRM Interpretation of error codes must be 0/1 (Success/Fail)

Authentication

Test scripts logged into the SRM server using the Service Account

SRM Variables

VMware_VC_Host = Host name of the vCenter host at the recovery site

VMware_RecoveryMode = Recovery mode (Test or Recovery)

Testing, testing and more testing!

Planned Migration & Failback, the Recovery and Failback


39/44

43

Key TakeawaysLets start wrapping up

Key Takeaways


40/44

44

Key Takeaways

vCloud DR is challenging

No SRM Integration (no tolerance to MoRef change)

vCloud DR is achievable

Documented and validated case study available Weve developed it!

This solution is based on the case study Now weve built it!

Recovery, Planned Migration and Failback are achievable Weve tested it!

Not restricted to a manual process (INF-VSP2164) Weve automated it!

Despite missing vCloud Director integration SRM can be used

Recovery Plan Steps can be used to drive automation (other options exist)

PowerCLI is a good fit and offers significant capability (other options exist)

You retrofit vCloud DR to your existing vCloud Understanding of the available case study information

SRM, vCloud Director and Automation skills


41/44

45

Questions?

References


42/44

46

References

vCloud Director Infrastructure Resiliency Case Study

http://www.vmware.com/resources/techresources/10254

vCloud Director Infrastructure Resiliency Automation Case Study

Coming Soon

Running Commands with SRM 5.x

http://vmland.blogspot.co.uk/2012/07/running-commands-with-srm-5x.html

Working with vCloud Metadata in PowerCLI

http://blogs.vmware.com/vipowershell/2012/03/working-with-vcloud-metadata-in-powercli.html

vCloud DR PowerShell Scripting

www.virtu-al.net/

Tweet: #wvmworldinf and #BCO2155
http://www.vmware.com/resources/techresources/10254http://vmland.blogspot.co.uk/2012/07/running-commands-with-srm-5x.htmlhttp://blogs.vmware.com/vipowershell/2012/03/working-with-vcloud-metadata-in-powercli.htmlhttp://blogs.vmware.com/vipowershell/2012/03/working-with-vcloud-metadata-in-powercli.htmlhttp://www.virtu-al.net/http://www.virtu-al.net/http://blogs.vmware.com/vipowershell/2012/03/working-with-vcloud-metadata-in-powercli.htmlhttp://blogs.vmware.com/vipowershell/2012/03/working-with-vcloud-metadata-in-powercli.htmlhttp://vmland.blogspot.co.uk/2012/07/running-commands-with-srm-5x.htmlhttp://www.vmware.com/resources/techresources/10254


43/44

FILL OUT

A SURVEY

EVERY COMPLETE SURVEY

IS ENTERED INTO

DRAWING FOR A

$25 VMWARE COMPANY

STORE GIFT CERTIFICATE


44/44

vCloud DR for Oxford

University ComputingServices - Real WorldExample

Gary Blake, VMware, Inc.

Aidan Dalgleish, VMware, Inc.

Adrian Parks, Oxford UniversityComputing Services

INF-BCO2155

Documents

BCO2155-vCloud DR for Oxford University Computing Services - Real World Example_Final_US.pdf