26
SA L07 Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature known as Virtual Business Service (VBS). VBS ensures high availability and manageability across multiple tiers, hardware architectures, virtualization technologies and operating systems. Gain hands-on experience on how to use VCS and ApplicationHA together with VBS to ensure 24x7 availability for a complete Multi- tier business service from the top tier to the bottom tier. Attendees will learn how VBS integrates with different components in a datacenter, such as virtualization technologies. Attendees should have a working knowledge of VCS and preferably VERITAS Operations Manager (VOM). They should be able to install and configure both products. At the end of this lab, you should be able to Explain the relationship between VBS, VCS, ApplicationHA and VOM Install the VBS infrastructure from VOM Configure Virtual Business Services Understand VBS dependency types and fault management model Understand and configure Recovery Plans Environment The current lab simulates two datacenters, one in California (CA) and one on New York (NY). The labs are based on 8 Virtual Machines Database Tier: 3 nodes running Solaris and VCS (Global Cluster configuration). Oracle database Application Tier: 2 nodes running Linux and VCS (single node configurations) Websphere application Web Tier: 2 VMWare Virtual Machines running Windows and ApplicationHA. IIS web server

SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

SA L07 – Virtual Business Services Lab

Hands-On Lab

Description VERITAS Cluster Server (VCS) includes a feature known as Virtual Business Service (VBS). VBS ensures high availability and manageability across multiple tiers, hardware architectures, virtualization technologies and operating systems. Gain hands-on experience on how to use VCS and ApplicationHA together with VBS to ensure 24x7 availability for a complete Multi-tier business service – from the top tier to the bottom tier. Attendees will learn how VBS integrates with different components in a datacenter, such as virtualization technologies. Attendees should have a working knowledge of VCS and preferably VERITAS Operations Manager (VOM). They should be able to install and configure both products.

At the end of this lab,

you should be able to Explain the relationship between VBS, VCS, ApplicationHA

and VOM

Install the VBS infrastructure from VOM

Configure Virtual Business Services

Understand VBS dependency types and fault management model

Understand and configure Recovery Plans

Environment The current lab simulates two datacenters, one in California (CA) and one on New York (NY). The labs are based on 8 Virtual Machines Database Tier: 3 nodes running Solaris and VCS (Global Cluster

configuration). Oracle database

Application Tier: 2 nodes running Linux and VCS (single node

configurations) Websphere application

Web Tier: 2 VMWare Virtual Machines running Windows and

ApplicationHA. IIS web server

Page 2: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Feature Descriptions

This section contains a short introduction of the features that we will configure in this lab

session.

Virtual Business Services

Virtual Business Services (VBS) is an add-on to Symantec’s leading clustering software VERITAS Cluster Server (VCS). VBS combines the proven High Availability technology from VCS, as well as ApplicationHA and the power of the VERITAS Operations Manager (VOM) management interface.

VBS allows you to manage multiple Service Groups (applications), grouping them into a single entity. The service groups can be located in different clusters, with a different operating system and hardware architecture. Key features of VBS:

One-click start/stop operations of applications in different tiers

Fault propagation between tiers, providing availability on the service level

One-stop shop for HA management, regardless of operating system, hardware platform or virtualization technology

Start/Stop of VMWare Virtual Machines as a part of VBS operations

One click disaster recovery by using the Recovery Plan.

VBS architectural features:

No central brain orchestrates HA operations. Each tier is managing its own High Availability. The VOM server is required for initial configuration. Visualization and Management is also provided by VOM. However, VBS operations can be carried out independently of the VOM server by using the VBS CLI

VBS is supported from VCS 5.1/ApplicationHA 5.1 SP2 and onwards. All later releases of VCS/ApplicationHA supports VBS

Support all hardware architectures and operating systems that VCS and ApplicationHA supports

Supports operations using the VOM Central Management Server or VBS CLI.

Page 3: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Start/stop of multiple tiers

Coordination to start/stop a complete multi-tier application has been a pain point since the

introduction of multi-tiered concepts. Usually, these procedures are carried out manually,

which involves interaction from several different teams within an organization. These

procedures are time consuming and there is a lot of room for human errors.

VBS introduces the ability to start/stop a complete multi-tier application with a single operation.

Fault Management

Keeping components in each tier highly available is still the foundation for service level

availability. This is maintained by VCS and/or ApplicationHA.

However, if a failure occurs in one tier, this fault may need to be propagated to another tier.

For example, an application in a middle tier may need to be restarted following a failure on a

lower tier. VBS introduces fault propagation between different clusters running on different

platforms.

By propagating failures, VCS/ApplicationHA, together with VBS can maintain the availability for

the complete multi-tier business service.

Start/Stop of Virtual Machines

VBS includes the capability to start and stop individual VMware Virtual Machines. In a disaster

recovery scenario, VBS can spin up “cold” VMs running on a DR site.

Note: Virtual Machine start/stop is not covered in this lab.

Disaster Recovery

VBS includes extensive disaster recovery support. To carry out a successful disaster recovery event in a real-world disaster situation, automation is key. VBS together with the Recovery Plan provides this functionality. Bringing up a datacenter on the disaster recovery site can be done by executing a single command.

Page 4: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Lab Exercises

Deploying VBS infrastructure

In this exercise, you will deploy required packages to support the VBS functionality on some of

the cluster nodes and on the VOM Central Management Server.

Installing the infrastructure packages is done in two steps.

1. Install the VOM Virtual Business Services Availability Add-On. This add-on is

installed onto the VOM Central Management Server only.

2. Deploy the VRTSvbs package. Note – this is required for cluster nodes prior to VCS 6.0

and on ApplicationHA nodes. In this lab, you will deploy this package to the

ApplicationHA nodes. All other clusters are running VCS 6.0.1.

NOTE: The VOM Virtual Business Services Availability Add-on and the VRTSvbs

packages have already been uploaded to the VOM Central Management Server to save time.

Eliminating this step will give you more time to complete labs. In a real-world environment, the

VBS Availability Add-On and the VRTSvbs package are required to be uploaded to the VOM

Central Management Server.

Installing the VOM VBS Availability Add-on onto the VOM Central Management Server

1. Click on the “vomserver” virtual machine.

2. Login to the server by using the credentials on the presentation screen or on the last page of this document (root/symc4now)

3. Click on the “VOM Central Management Server Login Page” short cut, located on the desktop. The short cut will take you directly to the VOM splash screen. Login credentials are available on the presentation screen or on the last page of this document (root/symc4now)

4. In the Veritas Operations Manager console, select Settings > Domain Management > Deployment

5. In the Deployment Management view, select the Virtual Business Services Availability add-on and click Actions, Install

6. In the Install Solution panel, you will get prompted to install the VBS Add-On on the “vomserver” Central Management Server. Click Install

Page 5: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

7. Click on Goto Solutions Summary. NOTE: VOM will be restarted in the background. This may take some time. Refresh the session and login to the VOM Central Management Server again

8. Select Settings > Domain Management > Deployment and verify that the Virtual Business Service Availability Add-on is installed and enabled successfully.

Installing the VRTSvbs package

1. In the VOM UI, select Settings > Domain Management > Deployment

2. In the Deployment Management view, select VRTSvbs and click Actions, Install

NOTE: Select the Windows version of the VRTSvbs package.

3. In the Install Solution panel, select all hosts that will participate in the VBS configuration. Select the following nodes:

ca-webwin1

ny-webwin2

Click Install

NOTE: The VRTSvbs package is NOT required on the VOM Central Management

Server (vomserver)

4. Click on Goto Solutions Summary. In the Deployment Summary view, verify that the VRTSvbs package installs correctly on all nodes. This may take some time

Virtual Business Services Setup and Management

Initial configuration of a VBS is performed from the VOM UI. After a VBS has been created, it

can be managed and displayed using VOM or CLI. In this lab, we will configure three different

Virtual Business Services, one for each dependency type. The dependency types are

explained in a later lab.

Page 6: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

This picture displays how the lab environment will look like when the VBS configuration is

completed. Initially, you will configure the California site. When reaching to the Disaster

Recovery labs, you will configure the New York site.

The table below describes the VBS configuration and dependencies.

VBS Name

Lowest Tier / Child Service

Group

Dependency Type

Depending Tier 1 / Parent Service

Group

Dependency Type

Depending Tier 2 / Parent Service

Group

HR_VBS ca_oracle1 Soft ca_websphere1 N/A N/A

Sales_VBS ca_oracle2 Firm ca_websphere2 N/A N/A

Finance_prod_VBS

global_oracle3 Restart ca_websphere3

(child SG for CA_IIS_SG)

Soft CA_IIS_SG

Page 7: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

NOTE: Please disregard the DR VBS (Finance_DR_VBS) for the moment. We will configure that VBS in a later lab exercise. The following picture shows an example of a parent/child relationship

Virtual Business Services creation

You will now create the three VBS:s specified in the table above. Repeat the following steps

for each VBS.

1. In the VOM GUI, click Manage > Business Entities > Virtual Business Services

2. Click Actions, Create Virtual Business Service

3. Type in a Name from the table above and click Next

4. In the Select Base Object Types panel, select Service Groups and click Next

5. In the Select Base Service Groups panel, select Service Groups according to the table above and click Next.

NOTE for Finance_prod_VBS: Two service groups with the same name (global_oracle3) is present, one for each site. Make sure to select the global_oracle3 service group from the ca-dbsolclus1 cluster)

6. In the Virtual Business Service Summary panel, click Finish

7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies

8. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table above. Select Parent and Child, and click Link.

Page 8: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

NOTE: For the Finance_prod_VBS configuration, you will configure two service group dependencies

Make sure that the Configure Fault Management checkbox is selected. This checkbox

should be set to true for new VBS deployments.

Click Next

9. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish

10. In the Results panel, click OK

11. Repeat the above steps for each VBS (HR_VBS, Sales_VBS and Finance_prod_VBS)

Start/Stop and visualization of Virtual Business Services

The objective in this lab is to visualize the state of a VBS and to test start/stop functionality.

VBS can be managed using VOM or CLI.

Start/stop using VOM:

1. Click Manage > Business Entities > Virtual Business Services

2. Right-click on a VBS, select VBS Availability, Start VBS or Stop VBS

3. Click OK to confirm that you want to start up the VBS

4. The Service Group Order panel is displayed. Examine and validate start/stop of the VBS. Note that the progress of the operation is displayed on the right hand side of the VOM GUI.

Start/stop using CLI:

1. Open a terminal session as the “root” user to any node participating in the VBS (ca-dbsol1, ca-dbsol2, ca-applinux1. Note that those commands cannot be executed from the VOM Central Management Server)

Page 9: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Double-click on the “Terminal” icon on the desktop of the “vomserver” virtual machine. Connect to the remote node by using SSH:

# ssh ca-dbsol1

If prompted for a password, use “symc4now”

NOTE: In remaining labs, it’s assumed that the participant know how to use SSH to connect to the cluster nodes. From now on, the document will not specify how to connect to the cluster nodes.

2. Use the following commands to start/stop and visualize a VBS from CLI:

# vbssvc –display Displays the VBS configuration

# vbssvc –start/stop <vbs> Start/stop a VBS

# vbssvc –state <vbs> <vbs> entry is optional. If not specified, all VBS

configured in the cluster are displayed

# vbssvc –grpstate <vbs> displays the availability state for each group per

VBS

# vbsgrp –state <vbs> <vbs> entry is optional. If not specified, groups

from all VBS configured on the cluster will be

displayed

Understanding VBS Fault Management

High Availability High Availability decisions, such as failing over a service group from one node to another, are handled by VCS/ApplicationHA on the local cluster level. In addition, VBS provides the ability to propagate an event from one tier to another. Fault propagation is not dependent on the VOM Central Management Server. Even if the VOM Central Management Server is unavailable, an event will still be propagated. Note that VBS will propagate events upon failures only. A service group have to fail to trigger fault propagation. If a service group is switched over manually, a fault will not be propagated to another cluster/tier. In our examples, we have one 3-tier VBS (Finance_prod_VBS) and two 2-tier VBS (HR_VBS and Sales_VBS)

Page 10: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

The following table describes the VBS dependency types:

Dependency

Type Explanation Use case

Soft When a fault occurs, the event is not propagated to another tier

Used for start/stop orchestration only.

Firm When the child faults, the parent is taken OFFLINE. When the child recovers, the parent is brought

ONLINE

Some applications may cause issues such as crashes, unnecessary memory allocation etc if for example a database is unavailable. Using

the Firm dependency avoids this situation.

Restart When the child faults, the parent

ignores the event.

When the child recovers, the parent is taken OFFLINE and then brought

ONLINE.

Some applications require a restart following a failure of for example a database. Application may need to restart to reconnect to the

database. The Restart dependency resolves this situation and will provide another layer of High Availability.

Understanding the “soft” service group dependency

1. Open a Terminal Session to “ca-applinux1” and “ca-dbsol1” using SSH. Issue the following command in both sessions:

# tail –f /var/VRTSvcs/log/engine_A.log

2. Start the HR_VBS Virtual Business Service using VOM. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated

3. Open a terminal session to “ca-dbsol1”

4. Issue the following # hastatus command to locate the ca_oracle1 service group.

# hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A ca-dbsol1 RUNNING 0

A ca-dbsol2 RUNNING 0

-- GROUP STATE

Page 11: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

-- Group System Probed AutoDisabled State

B ClusterService ca-dbsol1 Y N ONLINE

B ClusterService ca-dbsol2 Y N OFFLINE

B ca_oracle1 ca-dbsol1 Y N ONLINE Online

B ca_oracle1 ca-dbsol2 Y N OFFLINE

B ca_oracle2 ca-dbsol1 Y N ONLINE

B ca_oracle2 ca-dbsol2 Y N OFFLINE

B global_oracle3 ca-dbsol1 Y N ONLINE

B global_oracle3 ca-dbsol2 Y N OFFLINE

-- WAN HEARTBEAT STATE

-- Heartbeat To State

M Icmp ny-dbsolclus2 ALIVE

-- REMOTE CLUSTER STATE

-- Cluster State

N ny-dbsolclus2 RUNNING

-- REMOTE SYSTEM STATE

-- cluster:system State Frozen

O ny-dbsolclus2:ny-dbsol3 RUNNING 0

-- REMOTE GROUP STATE

-- Group cluster:system Probed AutoDisabled State

P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE

5. If the ca_oracle1 service group is online on ca-dbsol1, continue with the next step. If the ca_oracle1 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step

6. Simulate a failure of the ca_oracle1 service group by removing the following lock file: # rm /testfiles/oracle1

Examine the output in the terminal sessions started in step 1, as well as the

visualization in the VOM UI. The ca_oracle1 service group should be failed over from

one system to another, and the fault should not be propagated to another tier

Page 12: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

7. Clear the resource failure of the ca_oracle1 service group. This command can be

executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from

VOM.

# hagrp –clear ca_oracle1

Understanding the “Firm” service group dependency

To display the Firm dependency type, a double failure is required. We will fail the ca_oracle2

service group twice (once on node ca-dbsol1 and once on node ca-dbsol2)

1. Open a Terminal Session to “ca-applinux1” and “ca-dbsol1” using SSH. Issue the following command in both sessions: # tail –f /var/VRTSvcs/log/engine_A.log

2. Start the Sales_VBS Virtual Business Service using VOM. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated

3. Open a terminal session to “ca-dbsol1”

4. Issue the following # hastatus command to locate the ca_oracle2 service group # hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A ca-dbsol1 RUNNING 0

A ca-dbsol2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B ClusterService ca-dbsol1 Y N ONLINE

B ClusterService ca-dbsol2 Y N OFFLINE

B ca_oracle1 ca-dbsol1 Y N ONLINE

B ca_oracle1 ca-dbsol2 Y N OFFLINE

B ca_oracle2 ca-dbsol1 Y N ONLINE Online

B ca_oracle2 ca-dbsol2 Y N OFFLINE

B global_oracle3 ca-dbsol1 Y N ONLINE

B global_oracle3 ca-dbsol2 Y N OFFLINE

Page 13: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

-- WAN HEARTBEAT STATE

-- Heartbeat To State

M Icmp ny-dbsolclus2 ALIVE

-- REMOTE CLUSTER STATE

-- Cluster State

N ny-dbsolclus2 RUNNING

-- REMOTE SYSTEM STATE

-- cluster:system State Frozen

O ny-dbsolclus2:ny-dbsol3 RUNNING 0

-- REMOTE GROUP STATE

-- Group cluster:system Probed AutoDisabled State

P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE

5. If the ca_oracle2 service group is online on ca-dbsol1, continue with the next step. If the ca_oracle1 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step

6. Simulate a failure of the ca_oracle2 service group by removing the following lock file: # rm /testfiles/oracle2

Use # hastatus –sum to determine that the service group has failed over from one

system to another. Login to the other system and issue the above command again:

# rm /testfiles/oracle2

Examine the output in the terminal sessions started in step 1, as well as the

visualization in the VOM UI. The ca_oracle2 service group is now completely faulted

and cannot go online. The Firm dependency propagates the failure and brings offline

the ca_websphere2 service group in the tier above. Examine the log output as well as

the VOM visualization.

Page 14: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

7. Clear the resource failure and online the ca_oracle2 service group. This command can be executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from VOM.

# hagrp –clear ca_oracle2

# hagrp –online ca_oracle2 –sys ca-dbsol1

This step simulates that an administrator resolved a non-recoverable issue with the

database, and then starting up the database again.

Examine the output in the terminal sessions started in step 1 and the visualization in the

VOM UI again. The dependent service groups are automatically started by VBS

Understanding the “Restart” service group dependency

1. Open a Terminal Session to “ca-applinux1” and “ca-dbsol1” using SSH. Issue the following command in both sessions: # tail –f /var/VRTSvcs/log/engine_A.log

2. Start the Finance_prod_VBS Virtual Business Service using the VOM UI. Make sure that the “Service Group order” panel is displayed. This panel is displayed automatically after VBS start has been initiated

3. Open a terminal session to “ca-dbsol1”

4. Issue the following hastatus command to locate the global_oracle3 service group

# hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A ca-dbsol1 RUNNING 0

A ca-dbsol2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B ClusterService ca-dbsol1 Y N ONLINE

B ClusterService ca-dbsol2 Y N OFFLINE

B ca_oracle1 ca-dbsol1 Y N ONLINE

B ca_oracle1 ca-dbsol2 Y N OFFLINE

B ca_oracle2 ca-dbsol1 Y N ONLINE

Page 15: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

B ca_oracle2 ca-dbsol2 Y N OFFLINE

B global_oracle3 ca-dbsol1 Y N ONLINE Online

B global_oracle3 ca-dbsol2 Y N OFFLINE

-- WAN HEARTBEAT STATE

-- Heartbeat To State

M Icmp ny-dbsolclus2 ALIVE

-- REMOTE CLUSTER STATE

-- Cluster State

N ny-dbsolclus2 RUNNING

-- REMOTE SYSTEM STATE

-- cluster:system State Frozen

O ny-dbsolclus2:ny-dbsol3 RUNNING 0

-- REMOTE GROUP STATE

-- Group cluster:system Probed AutoDisabled State

P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE

5. If global_oracle3 service group is online on ca-dbsol1, continue with the next step. If global_oracle3 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step

6. Simulate a failure of the global_oracle3 service group by removing the following lock file: # rm /testfiles/oracle3

Examine the output in the terminal sessions started in step 1, as well as the

visualization in the VOM UI. The global_oracle3 service group should be failed over

from one system to another, and the fault is propagated to the application tier, in this

case the ca_websphere3 service group running on the ca-applinux1 cluster node.

This service group will automatically be restarted when the global_oracle3 service

group fails over.

Page 16: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

7. Clear the resource failure of the global_oracle3 service group. This command can be executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from VOM.

# hagrp –clear global_oracle3

Configuring Virtual Business Services for Disaster Recovery

Together with VERITAS Cluster Server and VERITAS Operations Manager, VBS implements end-to-end disaster recovery management. Disaster Recovery can be managed per VBS, or for a set of VBS:s, or for the complete data center. The data center can be recovered on the DR site by executing one single command. The following prerequisites exist for VBS DR operations:

At least one tier needs to be configured with VCS/GCO

Control of data replication should be managed by VCS/GCO. For tiers without data replication requirements, GCO is not mandatory

VBS DR operations are not supported together with VMWare SRM

The service group on the primary site and the secondary site can not be in the same VBS.

Page 17: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

In this lab, the number of VBS tiers between production and DR sites is equal. While this is most common, it is not a requirement. Number of tiers, number of cluster nodes, OS etc, can differ. In this lab, two sites are simulated (California and New York). The database tier is configured with VCS/GCO, while the remaining two tiers are configured for local HA only (note that the number of nodes has been reduced to prevent lab machines from being overloaded) VCS on the database tier will take care about the availability of the database. The following flow is required to configure a VBS for Disaster Recovery:

1. Configure VCS/GCO (already done) 2. Configure replication and global service groups (already done, but simulated for the

stake of the lab) 3. Configure the VBS on the primary site, California (done in earlier labs) 4. Configure the VBS on the secondary site, New York. This will be done in the next lab.

Page 18: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Configuring the DR VBS.

1. In the VOM GUI, click Manage > Business Entities > Virtual Business Services

2. Click Actions, Create Virtual Business Service

3. Type in a Finance_DR_VBS as Name and click Next

4. In the Select Base Object Types panel, select Service Groups and click Next

5. In the Select Base Service Groups panel, select Service Groups the following service groups:

global_oracle3 (Note that two service groups exist with this name. Make sure to select the global_oracle3 service group from the ny-dbsolclus2 cluster)

ny_websphere3

NY_IIS_SG

Click Next

6. In the Virtual Business Service Summary panel, click Finish

7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies

8. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table below. Select Parent and Child, and click Link.

VBS Name

Lowest Tier / Child Service

Group

Dependency Type

Depending Tier 1 / Parent Service

Group

Dependency Type

Depending Tier 2 / Parent Service

Group

Finance_DR_VBS global_oracle3 Restart ny_websphere2 Soft NY_IIS_SG

9. Make sure that the Configure Fault Management checkbox is selected. This checkbox

should be set to true for new VBS deployments.

10. Click Next

11. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish

12. In the Results panel, click OK

Page 19: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

VBS DR Operations for individual VBS objects

Concurrency violation prevention is handled by VCS/GCO. VBS integrates with this feature

and will not allow a VBS to be started if the corresponding site still is up. If the site is

down/unreachable, VBS will allow start-up.

1. Make sure that Finance_prod_VBS is started.

2. At this point, the Finance_DR_VBS is partially started. For the stake of the lab, offline the NY_IIS_SG from the VOM UI:

In the VOM GUI, click Manage > HA-DR > Service Groups

Right-click on the NY_IIS_SG and select Offline. Click OK

3. Open a Terminal Session to “ca-dbsol1” using SSH and issue the following command in each session:

# vbssvc –showplan Finance_prod_VBS

Note that this cluster has authority for the global service group.

4. Open a Terminal Session to “ny-dbsol3” using SSH and issue the following command in each session:

# vbssvc –showplan Finance_DR_VBS

Examine the difference. The VBS on the NY site cannot go online due to the fact that the VBS is online on the CA site.

5. Offline the Finance_prod_VBS by using VOM GUI or by using CLI from ca-dbsol1/2: # vbssvc –stop Finance_prod_VBS

6. Now, execute the #vbssvc –showplan command again:

On ca-dbsol1/2:

Page 20: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

# vbssvc –showplan Finance_prod_VBS

On ny-dbsol3: # vbssvc –showplan Finance_DR_VBS

The VBS is now ready to go online on any site. Online the Finance_DR_VBS site by

using VOM or CLI, for example:

On ny-dbsol3:

# vbssvc –start Finance_DR_VBS

7. To prepare for the next lab, please switch back the VBS to the California site by using VOM or CLI, for example:

On ny-dbsol3:

# vbssvc –stop Finance_DR_VBS

On ca-dbsol1/2:

# vbssvc –start Finance_prod_VBS

Recovery Plan – DR operations for the complete datacenter

In the last lab, you learned how to manage DR for individual VBS:s. While this is fine for

isolated errors, it may not be feasible for a site-wide outage, possible affecting hundreds of

VBS:s.

A VOM 5.0 feature known as the “Recovery Plan” addresses this use case. The Recovery Plan

is a run-book that can be used for disaster recovery as well as server repurposing.

In a Recovery plan, certain tasks can be configured. Tasks are executed in the order specified

in the Recovery plan.

Page 21: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

The Recovery Plan supports three types of tasks:

1. VBS Start/Stop (can be used global failover)

2. Service Group Start/Stop (can be used for global failover)

3. Custom Script Execution (for example to automatically provision extra CPU/Memory in a

disaster recovery scenario)

Each task is configured with a set of parameters:

Type: Type of object to operate on. VBS, Service Group or Custom Script.

Task Name: Name of the task.

Action: What kind of action to trigger. Start, Stop, Execute

Critical: If a task is marked as critical and it fails, then the recovery plan is stopped

during execution and the remaining tasks are not run. If the task is marked

as not critical, then even if the task fails, the recovery plan execution is not

stopped.

Recovery Plan – DR operations for the complete data center

Creating a Recovery Plan

1. In the VOM UI, click Manage, Recovery Plan

2. Click Actions, Create Recovery Plan

3. In the Specify name and description for the Recovery Plan panel, specify “Datacenter_Recplan” as the name for the Recovery Plan and click Next.

4. In the Specify tasks for Recovery Plan wizard panel, you can add tasks for the Recovery Plan. The objective for this Recovery Plan is to switch over the Finance Multi-Tier Business Service from California to New York. Hence you need to add two tasks:

Type VBS Name Action Critical

VBS Finance_prod_VBS stop true

VBS Finance_DR_VBS start true

Page 22: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Click Add and select Type, VBS Name, Action and Criticality according to the table

above. Repeat the step for each VBS. Click Finish

5. In the Result panel, verify that the Recovery Plan has been successfully created. Click

OK.

Running a Recovery Plan

In this exercise, we only have a single VBS DR pair. However, in a real-world configuration,

the Recovery Plan may contain hundreds of VBS’s, Service Groups and/or scripts. Executing

all those tasks manually would take significant amount of time. Here, you will demonstrate the

capability of bringing up the DR site by executing one single click.

1. Click Manage, Recovery Plan

2. Right-click on the “Datacenter_Recplan” Recovery Plan and select Execute Recovery Plan.

3. In the Review tasks for the Recovery Plan panel, confirm the Recovery Plan that you want to run. Click Execute and then quickly OK to get to the Recovery Plan status view.

4. Sit back and relax while the Recovery Plan is bringing up your datacenter on the DR site! On the Results panel, verify that the Recovery Plan was executed successfully. Click OK.

Bonus LAB: Creating shared VBS dependencies

Welcome to the bonus lab! You have deployed the VBS infrastructure, learned how to manage

VBS from VOM and from the command line, understood fault management and also VBS DR.

VBS also support “shared service groups”. A shared service group is shared between two or

more VBS entities. VBS applies a specific logic to shared service groups.

Startup of a VBS with one or more shared service groups:

VBS will start up all service groups in the configured order.

Page 23: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Shutdown of a VBS with one or more shared service groups:

VBS will stop all non-shared service groups. By default, shared service groups will not be stopped. However, there is an option to offline shared service groups from the VOM UI and from the CLI

You will reconfigure the environment according to the picture on the next page.

Removing the Sales_VBS VBS and configuring a new shared VBS

In this lab, we need to remove the Sales_VBS VBS from the configuration and create a new

VBS that shares the bottom-most service group (ca_oracle1).

1. In the VOM UI, click Manage > Business Entities > Virtual Business Services

Page 24: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

2. Right-click on the Sales_VBS VBS and click Delete Virtual Business Service

3. Click Yes to confirm that you want to remove the VBS. Click OK

4. Click Actions, Create Virtual Business Service

5. Type in a PR_VBS as Name and click Next

6. In the Select Base Object Types panel, select Service Groups and click Next

7. In the Select Base Service Groups panel, select Service Groups the following service groups:

ca_oracle1 Note that this will be the shared service group

ca_websphere2

Click Next

8. In the Virtual Business Service Summary panel, click Finish

9. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies

10. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table below. Select Parent and Child, and click Link.

VBS Name Lowest Tier / Child Service

Group

Dependency Type

Depending Tier 1 / Parent Service Group

PR_VBS ca_oracle1

(shared) Restart ca_websphere2

11. Make sure that the Configure Fault Management checkbox is selected. This checkbox

should be set to true for new VBS deployments.

12. Click Next

13. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish

14. In the Results panel, click OK

Start/Stop of a VBS with shared service groups

Page 25: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

In this lab, we will validate start and stop behavior when one VBS has a shared service group

configured.

1. Stop HR_VBS from the VOM UI. Note the warning about shared service groups. Do not select “Offline shared service groups”. Only the non-shared service groups will be stopped now

2. Stop PR_VBS. Select Offline shared service groups (If you prefer CLI operations, you can use # vbssvc –stop –force PR_VBS

3. Start HR_VBS and PR_VBS again and confirm that they online correctly

4. Induce a failure on ca_oracle1 service group (remove the /testfiles/oracle1 lock file as instructed earlier). Note that only the ca_websphere2 service group is restarted.

Lab Configuration Information

Hostname IP Cluster Name

ca-dbsol1 169.254.128.11 ca-dbsolclus1

ca-dbsol2 169.254.128.12 ca-dbsolclus1

ca-applinux1 169.254.128.13 ca-appclus1

ca-webwin1 169.254.128.14 CA-WEBWIN1

ny-dbsol3 169.254.128.21 ny-dbsolclus2

ny-applinux2 169.254.128.22 ny-appclus2

ny-webwin2 169.254.128.23 NY-WEBWIN2

vomserver 169.254.128.31 N/A

Page 26: SA L07 Virtual Business Services Lab Hands-On Lab · 2016. 7. 4. · SA L07 – Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature

Service Group IP Cluster

ClusterService 169.254.128.27 ca-dbsolclus1

ClusterService 169.254.128.28 ny-dbsolclus2

Credentials: vomserver: root/symc4now ca-dbsol1/ca-dbsol2 root/symc4now ca-applinux1/ root/symc4now ca-webwin1 administrator/symc4now ny-dbsol3 root/symc4now ny-applinux2 root/symc4now ny-webwin2 administrator/symc4now