havcs-410-101 a-2-10-srt-pg_3

VERITAS Cluster Server for UNIX, Example Application Configurations HA-VCS-410-101A-2-10-SRT (100-002147)

COURSE DEVELOPERSBilge GerritsSiobhan SeegerDawn Walker

LEAD SUBJECT MATTER EXPERTS

Geoff BergrenConnie EconomouDave RogersJim SenickaPete Toemmes

TECHNICAL CONTRIBUTORS AND REVIEWERS

Billie BachraBarbara CeranBob LucasGene Henriksen

Disclaimer

The information contained in this publication is subject to change without notice. VERITAS Software Corporation makes no warranty of any kind with regard to this guide, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. VERITAS Software Corporation shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this manual.

Copyright

Copyright © 2005 VERITAS Software Corporation. All rights reserved. No part of the contents of this training material may be reproduced in any form or by any means or be used for the purposes of training or education without the written permission of VERITAS Software Corporation.

Trademark Notice

VERITAS, the VERITAS logo, and VERITAS FirstWatch, VERITAS Cluster Server, VERITAS File System, VERITAS Volume Manager, VERITAS NetBackup, and VERITAS HSM are registered trademarks of VERITAS Software Corporation. Other product names mentioned herein may be trademarks and/or registered trademarks of their respective companies.

VERITAS Cluster Server for UNIX, Example Application ConfigurationsApril 2005 Release

VERITAS Software Corporation350 Ellis StreetMountain View, CA 94043Phone 650–527–8000 www.veritas.com

Table of Contents iCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Course IntroductionVERITAS Cluster Server Curriculum ................................................................ Intro-2Guide Prerequisites........................................................................................... Intro-3Guide Objectives............................................................................................... Intro-4Legend .............................................................................................................. Intro-6

Example 1: Clustering ApplicationsApplication Service Overview.................................................................................. 1-3VCS Agents for Managing Applications .................................................................. 1-6The Application Agent ........................................................................................... 1-11Lab 1: Creating an Application Resource.............................................................. 1-15

Example 2: Clustering DatabasesDatabase Service Example..................................................................................... 2-3VCS Database Agents ............................................................................................ 2-5Database Preparation ............................................................................................. 2-9The Enterprise Agent for Oracle ........................................................................... 2-15Database Failover Behavior.................................................................................. 2-30Additional Oracle Agent Functions........................................................................ 2-32Lab 2: Creating Oracle Resources........................................................................ 2-38

Example 3: Clustering NFSNFS Service Overview ............................................................................................ 3-3Preparing NFS for High Availability......................................................................... 3-5Testing the NFS Service ....................................................................................... 3-12Configuring an NFS Service Group....................................................................... 3-16NFS Lock Failover in VCS 4.1 .............................................................................. 3-19Lab 3: Creating NFS Resources ........................................................................... 3-29

Appendix A: Lab SynopsesLab 1 Synopsis: Creating an Application Resource ................................................ A-2Lab 2 Synopsis: Creating Oracle Resources .......................................................... A-6Lab 3 Synopsis: Creating NFS Resources............................................................ A-16

Appendix B: Lab DetailsLab 1 Details: Creating an Application Resource.................................................... B-3Lab 2 Details: Creating Oracle Resources............................................................ B-11Lab 3 Details: Creating NFS Resources ............................................................... B-31

Appendix C: Lab SolutionsLab 1 Solutions: Creating an Application Resource................................................ C-3Lab 2 Solutions: Creating Oracle Resources ........................................................ C-11Lab 3 Solutions: Creating NFS Resources ........................................................... C-35

Appendix D: Design Worksheet: Template

Table of Contents

ii VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Introduction

Intro–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

VERITAS Cluster Server CurriculumThe VERITAS Cluster Server curriculum is a collection of courses that are designed to provide a full range of expertise with VERITAS Cluster Server (VCS) high availability solutions.

VERITAS Cluster Server for UNIX FundamentalsThis course covers installation and configuration of common VCS configurations, focusing on two-node clusters running application and database services.

High Availability FundamentalsThis course covers VERITAS Volume Manager and VERITAS Cluster Server Fundamentals.

VERITAS Cluster Server for UNIX, Example Application ConfigurationsThis guide provides optional supplementary material for VERITAS Cluster Server Fundamentals and High Availability Fundamentals. The instructor may opt to present some of this material in class, depending on time constraints and student interest.

VERITAS Cluster Server Curriculum

Courses

High Availability Fundamentals

VERITAS Cluster Server, Example

Application Configurations

VERITAS Cluster Server Fundamentals

OptionalGuide

Introduction Intro–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

Guide PrerequisitesThis guide assumes that you have mastered the fundamentals of VERITAS Cluster Server. You should also have an administrator-level understanding of one or more UNIX platforms, including the ability to configure systems, storage devices, and networking in multiserver environments. A basic understanding of VERITAS Volume Manager is also required.

Guide PrerequisitesTo successfully use the material in thisguide, you should have the followingexpertise:

VERITAS Cluster Server fundamentalsVERITAS Volume Manager basic configuration System and network device configuration


Guide ObjectivesThe VERITAS Cluster Server for UNIX, Example Application Configurations guide simulates the job tasks you perform to configure common applications in a VCS environment.

Guide ObjectivesAfter completing the VERITAS Cluster Server for UNIX, Example Application Configurations guide, you will be able to:

Configure an Application resource to manage an application in a VCS environment.Configure Oracle and Netlsnr resources and set up detail monitoring of an Oracle database instance.Configure NFS and Share resources to make an NFS service highly available.


Guide ResourcesThis guide contains modules with conceptual and configuration information for each topic. Lab exercises enable you to practice your new skills, and solutions are provided to guide you through the exercises.

Lab materials are provided in three forms, with increasing levels of detail to suit a range of student expertise levels. • Appendix A: Lab Synopses has high-level task descriptions and design

worksheets.• Appendix B: Lab Details includes the lab procedures and detailed steps.• Appendix C: Lab Solutions includes the lab procedures and steps with the

corresponding command lines required to perform each step.• Appendix D: Design Worksheet Template provides a blank design

worksheet.

Additional supplements may be used in the classroom or provided to you by your instructor.

Guide ResourcesParticipant Guide– Modules– Appendix A: Lab Synopses– Appendix B: Lab Details– Appendix C: Lab Solutions– Appendix D: Design Worksheet Template

Supplements– VCS Simulator: van.veritas.com– Tips & Tricks: www.veritas.com/education


LegendThese are common symbols used in this guide.

Symbol Description

Server, node, or cluster system (terms used interchangeably)

Server or cluster system that has faulted

Storage

Application service

Cluster interconnect

Wide area network (WAN) cloud


Client systems on a network

VCS service group

Offline service group

VCS resource

Symbol Description


Example 1Clustering Applications

1–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

IntroductionOverviewThis module describes how agents manage resources in a VCS cluster. In this module you are also introduced to the VCS bundled Application agent.

ImportanceUnderstanding how agents work helps you to determine the best way to manage your application service in a high availability environment.

Outline of Topics• Application Service Overview• VCS Agents for Managing Applications• The Application Agent

Configure the Application agent.The Application Agent

Describe how agents manage applications in a VCS environment.

VCS Agents for Managing Applications

Describe how application services work in an HA environment.

Application Service Overview

After completing this lesson, you will be able to:

Topic

Lesson Topics and Objectives

Example 1 Clustering Applications 1–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

1

Application Service OverviewRecall that application services consist of three basic building blocks or components:• Application—the processes or daemons that handle client requests • Storage—data storage resources • Network—resources that enable clients to access the service from the network

In order to make an application service highly available using VCS, you configure the applicable resources corresponding to the objects that are logically contained within these components.

Lessons in the VERITAS Cluster Server Fundamentals guide focused on configuring the storage and network components in VCS. This module concentrates on the VCS agents you can use to manage the application component.

Note: Before bringing VCS into the environment, ensure that all components are properly configured, as described in the “Preparing Services for High Availability” lesson in the VERITAS Cluster Server Fundamentals Participant Guide.

Application Service Overview

NetworkNetwork

StorageStorage

ApplicationApplicationS1

S2

DaemonsProcessesFile System

VolumeDisk Group

IP AddressNIC


Application RequirementsThe most important requirements for an application to run in a cluster are crash tolerance and host independence. This means that the application should be able to recover after a crash to a known state, in a predictable and reasonable time, on two or more hosts.

Most commercial applications today satisfy this requirement. More specifically, an application is considered well-behaved and can be controlled using a VCS agent if the application meets the following criteria:• There must be a defined start and stop procedure that can be automated.• If the application fails, it must be possible to clean up anything left behind,

such as defunct processes, reserved memory segments, and so on.• It should be possible to check the status of the application using an automated

method. Monitoring an application can be performed in many ways. In general, the closer a test emulates what the users of the application do, the better the test is in discovering problems. However, this may impact the performance of the application. VCS may also detect problems that you want administrator intervention to solve, such as running out of tablespace in a database. Therefore, the monitoring should be carefully balanced to ensure that the application is up while minimizing the overhead and limiting VCS to appropriate monitoring actions.

Application RequirementsIn order to control an application using a VCS agent, the application must:

Have a defined start and stop procedure.Provide the ability to clean up after operational failures.Provide the ability to monitor the health of the application using an automated method.Provide the ability to store its data to external shared disks.Be crash-tolerant; that is, it must be capable of being restarted to a known state after sudden failures.Be able to run on all servers designated as potential hosts, and must have:– No license issues tied to individual servers– No hostname dependencies


1

• If the application has any data that is updated during operation, the data should be stored on an external shared disk. This also means that the application should not keep any data in memory. Because the application is started on another host after operational failures, any data that other hosts cannot reach is lost. This precludes the use of any NVRAM accelerator boards or other disk- caching mechanisms contained in a local host. Disk caching for performance should only be performed on the external array.

• The application must be able to restart to a known state without manual intervention after operational failures, such as a system failure. For example, an Oracle database guarantees this by acknowledging the clients only after writing the changes to a log file on the shared disk (the redo logs) when a commit changes request is received. Anything that has not been committed or acknowledged is the responsibility of the client and is kept there. So when the system fails, the database can be brought up on another system because the other system has access to the redo logs and can determine what has been committed and written to disk. The database can be brought to the exact point of time of the crash by applying these redo logs.

• The application must be host-independent. Some applications are licensed to the host name, which means that the application does not start on another host unless the host name is changed. A highly available application should not have any features that tie it to a specific system in the cluster.


VCS Agents for Managing ApplicationsAgents are programs—combinations of binaries, libraries, scripts—that function as the intermediary between resources and the VCS engine. Every resource type has an agent that interacts with all resources of that type on behalf of VCS.

You have several options for selecting an agent to manage an application. • You can purchase VERITAS Enterprise agent packages as add-ons to the basic

VERITAS Cluster Server product. Some advantages of using Enterprise agents are that they are:– Easy to install and configure– Fully tested and supported for the documented versions of the operating

system and application– Enabled for detail monitoring (most agents) to perform additional

verification of the application status• You can use the Application or Process agents that are bundled with the base

VCS product. Depending on your application and requirements, you may need to write custom code or scripts when using the Application agent. For example, if you want to change the way that the agent monitors your application, you can create your own script and configure the agent to use that script.

• VERITAS has created partnerships with many application vendors to enable them to create agents for managing their software.

• You can modify an existing agent or create a new agent if you have specialized requirements for managing your application in a high availability environment. VERITAS provides separate documentation and training to enable you to develop VCS agents.

VCS Agents for Managing ApplicationsYou can purchase Enterprise agents for popular applications. Examples include:– Oracle, DB2, Informix, Sybase– Sun ONE, Apache, SAP

You can also manage some applications using bundled agents:– Application – Process

Some software vendors provide VERITAS-Enabled agents for their applications, such as:MySQL, NCR, and webMethodsCustomized agents can be developed in-house or by VERITAS Consulting Services.

Oracle

Application

Sun ONE

Process

DB2


1

Key Properties of VCS AgentsKey properties of VCS agents include:• Only one agent process runs on a system for each configured resource type.

That agent manages all resources of that type on that system.• An agent runs a single operation on a resource at one time. This ensures that

agents on different systems cannot be affecting a resource simultaneously on different systems.

• Agents are multithreaded so that operations can be carried out in parallel on multiple resources of the same type. For example, the IP agent can be bringing the IP1 resource online at the same time it is monitoring the IP2 resource.

• A resource cannot be managed without an agent.• If there are no resources of a particular type anywhere in the cluster, the agent

for that type is not started.

VCS agents are located within subdirectories of the VCS bin directory, /opt/VRTSvcs/bin.

Key Properties of VCS AgentsOnly one agent daemon runs on a system for each configured resource type.An agent runs a single operation on a resource at one time.Agents are multithreaded so that operations can be carried out in parallel on multiple resources of the same type.A resource cannot be managed without an agent.If there are no resources of a particular type anywhere in the cluster, the agent for that type is not started.


VCS Agent FunctionsMost VCS agents perform this common set of tasks:• Bring resources online when requested by the VCS engine. • Take resources offline upon request.• Periodically monitor resources and send status information to the VCS engine.• Clean up if a resource fails to go offline properly.

Agents also:• Restart resources when they fault (depending on the resource configuration).• Send a message to the VCS engine and the agent log when errors are detected.

Some agents support multiple levels of monitoring. For example, most database agents can be configured to perform two levels of monitoring:• Basic monitoring: Scans the process table for database processes• Detail monitoring: Performs a transaction on a test table in the database

Agents may also perform other functions, depending on the type of resource they manage. The documentation for each agent describes which functions each one supports. The VERITAS Cluster Server Agent Developer’s Guide provides a comprehensive description of each supported agent function.

For information about resource states and transitions, see the “Job Aid” appendix in this Participant Guide.

VCS Agent FunctionsTo manage an application, the VCS agent must beable to:

Start a specified program.Stop the specified program.Monitor the program.Some agents have multilevel monitoring functions that can be configured.– Basic-level monitoring may only check the process

table for applicable processes.– Second-level monitoring, if enabled, can execute

code, simulating client access to the application.Clean up after a fault.


1

The VCS Agent FrameworkThe modular design of VCS provides an agent framework to enable additional agents to be added without affecting the core engine functions. Agents manage resources by calling the appropriate function or entry point, and HAD manages the agents.

When an event occurs and HAD determines that an action must be taken, it calls the appropriate agent. The agent then invokes the script or function for the applicable entry point, passing a set of input parameters as arguments to perform the resource management task requested. The scripts called by the agent are referred to as entry points into the agent.

Entry points that are common to most agents are:• Online: Starts or creates the resource• Offline: Stops or removes a resource• Monitor: Determines resource status • Clean: Forcibly takes a resource offline

Entry points are determined by the type of resource that an agent is managing. For example, a NIC resource is persistent, so it does not have online and offline entry points. Online-only resources, such as NFS, do not have an offline entry point. However, all agents are required to have monitor and clean entry points.

In most cases, entry points are executed as well-known UNIX shell commands and are easily customizable for developers experienced with creating shell scripts.

The VCS Agent FrameworkAgents consist of an agent binary file plus a collection of scripts or C++ functions that correspond to each agent function.The agent functions are referred to as entry points.Most agents include the online, offline, monitor, and clean entry points.Entry points are often implemented as Perl or shell scripts, in which case they can easily be customized.Agents can also be implemented as compiled executable files.


How Agents WorkWhen the VCS engine requires an action to be taken for a resource—as the result of an administrative command or in response to an event, such as a fault—HAD calls an agent with a set of parameters. The agent takes a set of parameters as input from HAD. These parameters are the values specified by resource attributes in the main.cf file. The order in which they are passed is specified by the ArgList attribute of the corresponding resource type defined in types.cf.

The agent calls the entry point with the name of the resource followed by arguments—values of the resource attributes (main.cf)—in the order specified in the ArgList attribute for the resource type (types.cf).

Note: The diagram shows a conceptual view of the communication and relationships among VCS components. The agent does not read the configuration files directly. The VCS engine has the configuration in memory and passes the configuration information to the agent when it starts or a new resource is created. The agent then stores the configuration information in memory.

HAD

How Agents Work

OnlineEntry Point

$1 AppVolume$2 AppVol$3 AppDG

Volume AppVolume (Volume = AppVolDiskGroup = AppDG

)

main.cfmain.cf

Online AppVolume

vxrecover -g AppDG –bs AppVol

Volume Agentonlineofflinemonitorclean

type Volume (static str ArgList[] = { Volume, DiskGroup }

. . .)

types.cftypes.cf AppVolume

In-memory configuration


1

The Application AgentIf you have an application that does not have an available Enterprise agent, you may be able to use the Application agent to make the application highly available. The Application agent can be used to control more complex applications and can be easily customized. Whereas the Process agent can be used to monitor a single process, the Application agent can manage applications consisting of multiple processes.

Entry Points• Online: Runs StartProgram with the specified parameters in the specified user

context• Offline: Runs StopProgram with the specified parameters in the specified user

context• Monitor: If no MonitorProgram is specified, verifies that all processes

specified in PidFiles and MonitorProcesses are runningIf MonitorProgram is defined, the agent executes the MonitorProgram.

• Clean: Kills all the processes specified in PidFiles or in MonitorProcessesIf CleanProgram is defined, the agent executes the CleanProgram.

The Application AgentThe Application agent:

Simplifies managing applications with multiple processes and enables detailed monitoring.Requires no additional licensing; is bundled with VCS.Is easily customized by creating scripts that the agent uses within its entry points.Manages applications using these entry points:Online Brings an application online using StartProgramOffline Takes an application offline using StopProgramMonitor Monitors the status of the applicationClean Takes the application offline using CleanProgram

or kills all the processes specified for the application


Application Resource DefinitionThe required attributes listed in the slide must be defined to enable the Application agent to manage a resource.

The optional attributes, CleanProgram and User, can be configured to provide additional control of the application.

Application Resource DefinitionRequired Attributes:StartProgram Name of executable to start applicationStopProgram Name of executable to stop application

Plus one or more of the following:MonitorProgram Name of executable to monitor applicationMonitorProcesses List of processes to be monitoredPidFiles List of files that contain the process ID of

the processes to be monitoredOptional Attributes:CleanProgram Executable file that forcibly kills the

application and performs cleanup tasksUser UNIX account used as the execute

environment for entry pointsThe default is root.


1

Application Resource ConfigurationThe example in the slide shows how an Application resource is configured to make the Samba file sharing application highly available.

Note: The MonitorProcess values must match the output displayed by the ps command exactly. For example, if the processes are displayed with full path names, you must include the full path name when specifying the processes to monitor.

Application Resource ConfigurationConfiguration prerequisites:– The application should have its own start and stop

programs.– It must be possible to monitor the application by either

running a program that returns 0 for failure and 1 for success or by checking a list of processes.

Sample configuration:Application samba_app (StartProgram = “/usr/sbin/samba start”StopProgram = “/usr/sbin/samba stop”PidFiles = { “/var/lock/samba/smbd.pid” }MonitorProcesses = { “smbd nmbd” }

)


SummaryThis lesson described how agents manage resources in a VCS cluster. In this lesson you are also introduced to the VCS bundled Application agent.

Next StepsNow that you understand how agents manage applications, you can configure other common types of high availability services.

Additional Resources• VERITAS Cluster Server Bundled Agents Reference Guide

This guide describes each bundled agent in detail.• VERITAS Cluster Server Agent Developer’s Guide

This guide provides detailed information about the agent framework and creating or customizing agents.

• VERITAS Cluster Server Agent Development CourseThis course provides detailed information about the agent framework and creating or customizing agents. The course includes labs to customize existing agents and create new agents.

• VERITAS Cluster Server Fundamentals CourseThis course provides detailed information about configuring VCS service groups and resources.

• http://www.veritas.comA list of agents available from VERITAS and partners is available by selecting Products—>Cluster Server—>Agents and Options link.

Lesson SummaryKey Points – VCS provides a variety of agents to manage all

types of applications.– The Application agent is easily configurable for

basic management of an application.Reference Materials– VERITAS Cluster Server Bundled Agents

Reference Guide– VERITAS Cluster Server Agent Developer's Guide– VERITAS Cluster Server Agent Development

Course– VERITAS Cluster Server Fundamentals Course– http://www.veritas.com


1

Lab 1: Creating an Application ResourceLabs and solutions for this lesson are located on the following pages.

Appendix A provides brief lab instructions for experienced students.• “Lab 1 Synopsis: Creating an Application Resource,” page A-2

Appendix B provides step-by-step lab instructions.• “Lab 1 Details: Creating an Application Resource,” page B-3

Appendix C provides complete lab instructions and solutions.• “Lab 1 Solutions: Creating an Application Resource,” page C-3

GoalThe purpose of this lab is to configure the Application agent to manage a service with multiple processes.

PrerequisitesObtain any classroom-specific values needed for your classroom lab environment and record these values in your design worksheet included with the lab exercise instructions.

ResultsEach student’s service can be started, monitored, and stopped on each cluster system.

Lab 1: Creating an Application Resource

nameApp1

nameMount1

nameVol1

nameDG1

nameProxy1

nameIP1Delete the Process resource and add an Application resource.

Delete the Process resource and add an Application resource.

nameSG1nameSG1

See the next slide for classroom values.See the next slide for classroom values.

Appendix A: Lab SynopsesAppendix B: Lab DetailsAppendix C: Lab Solutions



Example 2Clustering Databases


IntroductionOverviewThis module describes how to manage database services in a VCS environment. An Oracle service group is used as an example showing specific agents and resource types that are provided with VCS enterprise agent for Oracle.

ImportanceDatabase services are a key component of most high availability environments. Understanding how you can use VCS to manage your database software is critical to implementing a complete high availability solution.

Outline of Topics• Database Service Example• VCS Database Agents• Database Preparation• The Enterprise Agent for Oracle • Database Failover Behavior• Additional Oracle Agent Functions

Describe common database service examples.

Database Service Example

Describe VCS agents for managing databases.

VCS Database Agents

Prepare a database for use in a VCS environment.

Database Preparation

Implement additional functionality provided by the Oracle agent.

Additional Oracle Agent Functions

Configure database failover behavior.Database Failover Behavior

Install and configure the VCS Oracle agent.

The Enterprise Agent for Oracle


Topic


Example 2 Clustering Databases 2–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

2

Database Service ExampleAs with most application services, database services generally include three basic components:• Application—the processes providing the service• Storage—data storage resources • Network—resources that enable clients to access the service from the network

The diagram shows a simple example of a database service.

Generic Database Service Example

File System /db

Contains data and log files

File System /db

Contains data and log files

Database/Listener

Database/Listener

IP Address 10.10.21.54

IP Address 10.10.21.54

VxVM Volumedbvol

VxVM Volumedbvol

Network Port NIC_deviceNetwork Port NIC_device

VxVM Disk Group dbdgVxVM Disk

Group dbdg

Network Component

Application Component

StorageComponent


VCS manages database services in the same way it manages any other application service. Resources are contained within the service group for each storage and network and dependencies are defined to ensure that VCS brings the supporting resources online before attempting to start the database.

The type of resource you use to manage the database depends on your database software and your specific management requirements.

Corresponding Database Service Group

/db/db

Database/ListenerDatabase/Listener

10.10.21.54 10.10.21.54

dbvoldbvol NIC_deviceNIC_device

dbdgdbdg

DB/Lsnr

Mount

Volume

DiskGroup

NIC

IP


2

VCS Database AgentsYou have several options for selecting an agent to manage the database application. • You can purchase VERITAS enterprise agent packages as add-ons to the basic

VERITAS Cluster Server product. The advantages of using an enterprise agent are that it is:– Easy to install and configure– Fully tested and supported for the documented versions of the operating

system and database applicationEnterprise agents are the primary focus of this lesson.

• You can use the Application agent that is bundled with the base VCS product. Depending on your database and your requirements, you may need to write custom code or scripts. For example, if you want to change the way the agent monitors your database, you modify the monitor entry point script.

• You can modify an existing agent or create a new agent if you have specialized requirements for managing your database in a high availability environment. VERITAS provides separate documentation and training to enable you to develop VCS agents.

Agents for Managing DatabasesYou can purchase Enterprise agents for:– Oracle– DB2– Informix– Sybase– MySQL

You can also manage some databases using the bundled Application agent or customized agents.

Oracle

DB2

Informix

Sybase

MySQL

Note: Not all agents are supported on all platforms.Note: Not all agents are supported on all platforms.


Agent FunctionsAll agents perform the same basic functions—start, stop, monitor, clean. The enterprise agents for databases each perform similar functions.

The agent starts an instance using the database-specific utility (such as sqlplus for Oracle), usually with a startup profile file name. The instance is likewise stopped with the database shutdown utility.

Two levels of monitoring can be configured:• Basic monitoring: Scans the process table for database daemon processes• Detail monitoring: Performs a transaction on a test table in the database

Primary Monitoring

The instance can be monitored by scanning the process table for the process IDs (PIDs) for critical database processes. The processes monitored vary by database. For example, the Oracle agent monitors the ora_smon, ora_dbw, ora_pmon, and ora_lgwr processes.

Agent FunctionsTo manage a database, the VCS agent must beable to:

Start a specified database instance.Stop the specified instance.Monitor the instance.– Basic-level monitoring may check the process

table for applicable database processes.– Detail monitoring may write to a test table in the

specified database.Clean up after a database fault to prevent possible concurrency violations to enable failover.


2

Detail MonitoringWhen detail monitoring is enabled, the monitor script attempts to write to a test table in a specified database. If this script fails and failover is enabled, the entire service group fails over to the next available system.

Writing to the database ensures that failures, such as full logs, are detected. Reading from the database does not detect this type of error.

In order to write to a database table, you must create the test table in the database that you are monitoring. Then, provide the agent with the access and configuration information needed to write to the table:• The database user account with update privileges• The password for the database user account• The name of the database table • SQL statements used to write to the table

Note: Detail monitoring may detect errors that cannot be handled by failover or restarts, such as a database running out of tablespace. Some agents, such as the Enterprise Agent for Oracle, provide optional methods of handling these types of errors. These additional error-handling methods are described later in this lesson.

Detail MonitoringAgents require this information to access andwrite to the table:

A database user account with update privilegesThe password for the database user accountThe name of the database table The name of the file containing the SQL statements used to write to the table


Monitoring Listener ProcessesSome database products, such as DB2, Sybase, and Informix, have the database network connection mechanism built into the database processes and do not require additional agents.

The Enterprise Agent for Oracle product contains an additional agent for managing the Oracle Net Services (ONS) listener process for Oracle, which controls client network connections to Oracle databases.

The Netlsnr agent starts and stops the Oracle Net Services listener process using the lsnrctl command. Two levels of monitoring can be configured:• Primary monitoring: Scans the process table for the tnslsnr process• Secondary monitoring: Uses the lsnrctl command to test the listener

process

Monitoring Listener ProcessesSome databases have listeners built-in and do not require additional VCS agents for monitoring.The Enterprise agent for Oracle includes an agent to monitor the listener process, named Netlsnr, which provides two levels of monitoring:– Primary: Scans the process table for the tnslsnr

process– Secondary: Uses lsnrctl to test the listener

ClientData

Server

Net8

OracleServer

ListenerProtocol

ONSUser


2

Database PreparationVerifying Software CompatibilityEnsure that each system that can host the database instance meets the operating system and database requirements specified in the agent documentation.

Also, ensure that each system has adequate resources, such as shared memory. Each system in the SystemList of the service group must be properly configured as a database server.

Verifying Software CompatibilityUse the latest database agent available for the VCS version in use.Verify that the agent supports the version of the database used in the cluster by checking the agent release notes.Examples:

Oracle on SolarisOracle on Solaris

Oracle Agent 4.1VCS 4.1Solaris 8, 9, 10Oracle 8.0.x, 8i (8.1.x), 9i, 10g

Oracle Agent 4.1VCS 4.1Solaris 8, 9, 10Oracle 8.0.x, 8i (8.1.x), 9i, 10g

DB2 on AIXDB2 on AIX

DB2 Agent 4.0VCS 4.0AIX 5.2DB2 ESE 7.2, 8.1

DB2 Agent 4.0VCS 4.0AIX 5.2DB2 ESE 7.2, 8.1


Database Program (Executable) FilesThe database binaries can be configured either on the local disks or on the shared storage. Select the location based on your high availability requirements.• Configuring the binaries on the local disks may enable you to minimize

downtime due to maintenance. You can update the database software on one system while the database is running on another.

• Installing binaries on a shared storage disk simplifies configuration, especially if there are many systems that can run the same database instance and participate in the service group. If you have many systems, applying updates on each system can be a significant burden.Note: Some databases, such as Oracle, install updates in a new directory. This directory can be on shared storage, which provides a way to use the advantages of updating one system while the database is running on the other. If binaries are storage on local disks, you have to update each individual system.

Database Program (Executable) FilesThe database program files can be configured either on the local disks or on shared storage.

Local disk:– Provides redundancy; offers additional protection in

case the binaries are damaged– The database must be installed and configured

identically on each system that can run the database service group.

Shared storage:– All systems must have the same mount point directory

for the database home directory.– Simplifies configuration, especially if there are many

systems that can run the same database instance


2

Data FilesYou must locate the data files on shared storage so that each failover target system can access them. Use these guidelines for locating data files in a VCS environment:• If using file system-based data files, the file systems that contain these files

must be located on shared disks. Create the same file system mount point on each system in the cluster.

• If using raw devices, such as Volume Manager volumes, set the permissions for the volumes to be owned permanently by the database account. For example:vxedit –g dgname set group=dba user=dba mode=660

volume_nameNote: Volume Manager overrides UNIX permissions. It is not sufficient to change permissions in UNIX because Volume Manager sets volume permissions when importing a volume (with default owner is root). You must also change the Volume Manager permissions, as shown in the slide.

Data FilesIf using file system-based data files, the file systems that contain these files must be located on shared disks.Create the same file system mount point on each system in the cluster.If using raw devices, such as Volume Manager volumes, set the permissions for the volumes to be owned permanently by the database. For example:vxedit –g dgname set group=dba user=dba mode=660 volNote: You cannot change permission only in UNIX. VxVM sets volume permissions upon import.


Database-Related System FilesAlthough it is not necessary for the system files listed on the slide to be exactly the same on all systems that participate in the service group, you must ensure that any database-related configurations are the same or consistent on all of the systems.

Note: Changes to /etc/system require a reboot to take effect.Solaris

/etc/passwd

/etc/shadow

/etc/group

/etc/services

/etc/system

/etc/hostsAIX

/etc/passwd

/etc/security/passwd

/etc/group

/etc/services

/usr/samples/kernel/vmtune

/etc/hosts

Database-Related System FilesThe following database-related system files must be consistent on all systems listed in the SystemList attribute for the service group:

Database user accountsPasswords for database user accountsDatabase group entries Listener entries Database shared memory settings The database service group (virtual) IP address

HP-UXAIX LinuxSolaris


2

HP-UX

/etc/passwd

/etc/group

/etc/services

/stand/system

/etc/hosts

Linux

/etc/passwd

/etc/shadow

/etc/group

/etc/services

Shared memory settings: – For drivers built into the kernel, append parameters to the kernel command

line using the boot loader. – For kernel modules, use /etc/modules.conf. – For tunable parameters, use sysctl and /etc/sysctl.conf.

/etc/hosts


Network ConfigurationEach database service group requires at least one IP address for client connections. This IP address should fail over together with the database in case of any major faults.

Therefore, you need to use an IP resource (or an IPMultiNIC resource) and configure the host name of the service group IP address in the database. The clients should connect to the host name corresponding to this virtual IP address and not to the local host names of the servers.

Network ConfigurationFor listener service failover, include at least one IP resource in the service group for the database.Use the host name for this service group IP address for the host field in the listener configuration file.

LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=salesDB)(PORT=1521))(ADDRESS=(PROTOCOL=ipc)(KEY=extproc)))). . .

listener.oralistener.ora

. . .10.10.21.54 salesDB. . .

/etc/hosts/etc/hosts

10.10.21.54

IP


2

The Enterprise Agent for OracleInstalling AgentsThe enterprise agents are provided as a standard software package that can be installed using the VERITAS Common Product Installation Utility or using the platform-specific standard installation utility.

Install the agent on each system using this basic installation procedure:1 Log on as root.2 Mount the installation CD.3 Change to the directory containing the agent software packages.4 Unzip and extract the files from the tar archive.5 Type the platform-specific command to install the agent.

For example, to install the Oracle enterprise agent, type:Solaris

pkgadd –d . VRTSvcsorAIX

installp -ac -d /cdrom/VRTSvcsor.rte.bff VRTSvcsor.rteHP-UX

swinstall -s /cdrom/oracle_agent/pkgs VRTSvcsorLinux

rpm -ihv VRTSvcsor-2.0-Linux.i386.rpm

Installing AgentsUse the VERITAS VPI or the platform-specific installation utility to install the VRTSvcs**package:– Solaris: pkgadd– AIX: installp– HP-UX: swinstall– Linux: rpm

Install the agents on each system in the cluster that can run the database service group.

Solaris HP-UXAIX Linux

Sample Oracle Agent Guides

Sample Oracle Agent Guides


Adding Resource Type DefinitionsBefore you can create database resources, you must add the resource type definitions to the VCS configuration.

You can add resource types while VCS is running or take VCS offline and modify the configuration files.

Adding Resource Types to a Running Configuration

These methods are provided for adding resource types while VCS is running:• Using Cluster Manager:

1 Select File—>Import Types.2 Browse to select the file containing the type definition.

• Use hatype to add an include statement to the main.cf file that references the databasetypes.cf file. You can also use hatype commands to add and modify resource type definitions in types.cf.

• Create and run a command file containing hatype commands.

Adding Resource Types to an Offline Configuration

A simple method for adding resource types requires you to stop VCS, edit the main.cf or types.cf file, and restart VCS. This is described in detail in the “Offline Configuration of Service Groups” lesson of the VERITAS Cluster Server Fundamentals course.

Adding Resource Type DefinitionsYou can use an online or offline method to addresource type definitions.

To add types while VCS is running, use Cluster Manager. Select File—>Import Types.To add types offline:1. Copy the databaseTypes.cf file to the

/etc/VRTSvcs/conf/config directory.2. Use the offline configuration process described

earlier in the VERITAS Cluster Server Fundamentalscourse to add this line to main.cf and restart the cluster using the modified configuration files.include “databaseTypes.cf”


2

High Availability Database Configuration OverviewBefore you begin configuring VCS to manage your database services, prepare your environment, including equipment, software, and staff.

Coordinate with the database and systems administrators responsible for the database servers you are adding to the cluster environment. After database services are placed under control of VCS, the databases cannot be managed separately, outside of VCS.

As with any high availability service, configure and test each component on the startup system and each failover system before you perform any VCS configuration of the service.

This procedure applies to configurations where the database binaries are installed locally on each system, which reflects the cluster design decisions assumed for this course. Examples of other configurations are shown later in the lesson. The High Availability Design Using VERITAS Cluster Server course provides a complete discussion of design considerations for VCS environments.

High Availability Database Configuration Overview

Install database.Install database.

Configure database.Configure database.

Test database.Test database.

Stop resources.Stop resources.

Y

N

MoreSystems?

MoreSystems?

Add resource types.Add resource types.

Add service group.Add service group.

Prepare OS.Prepare OS.

Install agent.Install agent.Test service group.Test service group.

Add/test resources.Add/test resources.

YBinaries on Local StorageBinaries on Local Storage


Oracle Service Group Configuration ExampleA service group example for an Oracle database instance contains these types of resources:• Netlsnr: Monitors the listener process• Oracle: Monitors the database and processes• Mount: Monitors one or more file systems containing:

– Oracle application files (if installed on shared storage)– Oracle data files– Oracle log files

• Volume: Monitors one or more volumes for the file systems• DiskGroup: Monitors one or more disk groups for the volumes• IP: Monitors the service group IP for the listener process• NIC: Monitors one or more network interface cards for remote client

connection

The example shown on the slide assumes that the Oracle binaries are located on local storage. The data files are located on a file system (rather than raw volumes). The clients access Oracle services using the service group IP address defined by the IP resource.

Configuration Example: Oracle Service Group

DiskGroup

Volume

Mount

Oracle

Netlsnr

NIC

IP


2

Configuring an Oracle Resource The required attributes for an Oracle resource are shown in the example excerpt of the main.cf file.

Note: In version 4.1 of the Oracle agent, Pfile is not a required attribute. However, Pfile is required for versions prior to 4.1.

Agent Entry Points• Online: Uses the sqldba/svrmgrl/sqlplus startup -f $Pfile

command to start the database The $Pfile argument is the startup profile file for the Oracle database instance and is specified in the Pfile attribute.

• Offline: Uses the shutdown command to stop the database• Monitor: Scans the process table for ora_dbw, ora_smon, ora_pmon, and

ora_lgwr The ora_pmon process monitors all other Oracle processes. If detail monitoring is enabled, as shown in the “Configuring Oracle Detail Monitoring” section, the monitor script attempts to write to the database.

• Clean: Forcibly stops the Oracle database by using the following svrmgrl or sqlplus shutdown abort command:If the process does not respond to the shutdown command, the agent scans the process table and kills processes associated with the configured instance.

• Info: Provides static and dynamic information about the state of the database• Action: Performs predefined actions on a resource

Configuring an Oracle ResourceStarts and stops an Oracle instanceMonitors the process table

OraSG_OraResource Name

/oraHomeoracleOwnerAPSID

Required AttributesOracleResource Type

OraSGService Group NameSample ValueResource Definition

Oracle OraSG_Ora (Sid = APOwner = oracleHome = "/ora")

Oracle OraSG_Ora (Sid = APOwner = oracleHome = "/ora")

main.cfmain.cf

Note: Pfile is a required attribute for versions of the agent prior to 4.1.


Optional Oracle Attributes

The StartUpOpt Attribute

The Oracle agent supports an optional StartUpOpt attribute to enable you to control how VCS starts a database instance.

Examples:• RECOVERDB: Assume that in the event the database node fails for a

nondatabase issue (for example, if the system panics due to a hardware issue), you do not want the database to start up (come online) if any data file recovery needs to take place. If you set StartUpOpt to RECOVERDB and the system running the database faults, VCS starts the database in recovery mode when it fails over to another node.

• CUSTOM: You might want to start the database up in standby mode and apply logs. In this case, you can create an SQL script to perform these actions, and this script is called when you set StartUpOpt to CUSTOM.You must create the script in /opt/VRTSvcs/bin/oracle with the name of start_custom_SID.sql, where SID is the same as the value of the SID attribute.

Optional Oracle Attributes: The StartUpOptAttribute

You can use the StartUpOpt attribute to specifyhow the Oracle instance is started.

ActionValue

Runs the SQL script in a specified location; an example might be to start up in standby mode and apply logs

CUSTOM

Performs a recovery of database files upon instance startup

RECOVERDB

Starts the database in restricted modeRESTRICTED

If the Pfile attribute is empty, run: startupIf Pfile is specified, run:startup pfile=‘pfile_location’

STARTUP(default)


2

The ShutDownOpt Attribute

You can configure the ShutDownOpt attribute for the Oracle resource to control how VCS stops a database instance when the Oracle resource is taken offline.

Note: If CUSTOM is specified, a script must exist in /opt/VRTSvcs/bin/oracle with the name of shut_custom_SID.sql, where SID is the same as the value of the SID attribute for this Oracle resource.

The ShutDownOpt AttributeYou can use the ShutdownOpt attribute to specifyhow the Oracle instance is stopped.

Uses srvctl to stop the databaseSRVCTLSTOP

Runs the SQL script in a specified locationCUSTOM

Performs a transactional stopTRANSACTIONAL

Runs shutdown immediateIMMEDIATE(default)

ActionAttribute


Other Optional Oracle Attributes

The Oracle agent supports three other optional attributes described in the slide. Example main.cf entries showing sample values for these attributes are shown in the following resource definition. For more information, see the Enterprise Agent for Oracle User’s Guide for your platform.Oracle OraSG_Ora (

Sid = AP

Owner = oracle

Home = “/oracle”

Pfile = "/oracle/dbs/initVRT.ora"

EnvFile = "/oracle/.ora_envfile"

AutoEndBkup = 0

Encoding = eucJP

)

The example value for the Encoding attribute sets encoding to the Japanese language set.

Other Optional Oracle AttributesAutoEndBkup: Takes the data files in the database out of the backup mode when coming online (The default is 1, true.)EnvFile: Specifies the location of a file sourced by Oracle agent entry point scripts containing environment variablesEncoding: Specifies encoding for other language sets


2

Configuring Oracle Detail MonitoringThe example configuration shows an Oracle resource with detail monitoring configured.

These optional Oracle attributes are used to configure detail monitoring:• DetailMonitor: A flag to enable and disable detail monitoring (VCS 4.x)

The default is 0, do not perform detail monitoring. You can set this value higher than zero to control how often detail monitoring is performed. For example, if set to 5, detail monitoring is performed every fifth monitor interval.

• User: A database user account with update privileges• Pword: The password for the database user account

You must encrypt the password as described in a later section.• Table: The name of the database table VCS uses for additional monitoring• MonScript: The executable script file containing the SQL statements VCS uses

when writing to the table• EnvFile: The file containing environment variables sourced by the agent

Configuration Prerequisites• Create the database user and password for use by VCS.• Create a test table within the monitored database.• Create an executable script with SQL statements.

In this example, the user scott with the password tiger should be defined in the VRT database with update privileges to the table called testtable. This table should be created in the database before the additional monitoring is enabled.

Configuring Oracle Detail Monitoring

Oracle OraSG_Ora (Sid = APOwner = oracleHome = “/ora”Pfile = “/ora/dbs/initAP.ora”EnvFile = “/ora/.ora_envfile”DetailMonitor = 1User = “scott”Pword = “lgLimHmmKum”Table = “testtable”MonScript = “./bin/Oracle/SqlTest.pl”)

User defined in APdatabase with update privileges for testtable

User defined in APdatabase with update privileges for testtable

The table created inAP database beforeenabling the VCS resource

The table created inAP database beforeenabling the VCS resource

The executable script containing SQL write statements

The executable script containing SQL write statements

main.cfmain.cf

EnableEnable


Encrypting Passwords

You can use the VCS encryption utility to encrypt database passwords before configuring the Pword attribute in the Oracle agent configuration.

Note: The value of Pword is automatically encrypted when you use the Oracle configuration wizard or one of the VCS GUIs to configure the resource.

To encrypt a password, run the vcsencrypt command. For example, type:# root:/opt/VRTSvcs/bin> vcsencrypt -agent


2

Error Handling for Detail Monitoring

You can configure how VCS responds to different types of errors when detail monitoring is configured to prevent the database from failing over in cases where that action is not appropriate.

For example, if the user name and password specified for an Oracle resource is incorrect, an error of the form ORA-01017 invalid username/password; logon denied occurs. This error indicates that the configuration is wrong, but the database may be functioning properly. In this case, you do not want VCS to bring down the database and restart it on another system. The same failure occurs wherever the database is brought online.

Rather, you want VCS to notify the appropriate administrator to fix the configuration problem.

You can customize error handling by modifying the oraerror.dat file in /opt/VRTSvcs/bin/Oracle. Entries in this file have the format:Oracle_error_string:action:confidence_level

This example entry leaves the Oracle resource online and sets the confidence level to 10 when an invalid user name error code is returned.01017:WARN:10

Error Handling for Detail MonitoringYou can configure the optional Oracle error handling file when using detail monitoring to specify VCS response based on the type of database errors. See /opt/VRTSvcs/bin/Oracle/oraerror.dat.

Marks the resource state as ONLINE and notifies the administrator (when notification is configured)

WARN

Marks the resource offline when a serious error occurs; resource faults and SG fails over

FAILOVER

Freezes the service group temporarily and marks the resource OFFLINE

NOFAILOVER

Marks the resource state as UNKNOWNUNKNOWN

Error is not critical; take no actionIGNOREDescriptionAction


Configuring the Netlsnr Resource and AgentThe following main.cf excerpt corresponds to the Netlsnr example resource shown in the design worksheet in the slide.Netlsnr OraSG_Listen (

Owner = oracleHome = "/oracle"TnsAdmin = "/ora/net/admin"Listener= LISTENERLsnrPwd = S2cEjcD5s3CbcEnvFile = "/ora/dbs/envfile“MonScript = "./bin/Netlsnr/LsnrTest.pl")

Note: The Sqlnet resource type manages the listener for versions of the Oracle agent prior to 4.0. The attributes are the same, except LsnrPwd, which is not supported prior to 4.0.

/oracle/dbs/envfileEnvFile

LISTENERListenerS2cEjcD5s3CbcLsnrPwd

Optional Attributes

OraSG_ListenResource Name

./bin/Netlsnr/LsnrTest.pl

Monscript

/ora/net/adminTNSAdmin/oraHomeoracleOwner

Required AttributesNetlsnrResource Type

OraSGService Group NameSample ValueResource Definition

Configuring a Netlsnr Resource

Starts and stops a listener process using lsnrctlMonitors the process table

Detail Monitoring


2

Agent Entry Points• Online: Uses the following command to start the listener process:

lsnrctl -start $LISTENER

• Offline: Uses the following command to stop the listener process: lsnrctl -stop $LISTENER

• Monitor: Scans the process table for tnslsnr $LISTENER and runs MonScript as the optionIf secondary monitoring is enabled, the monitor script runs the command:lsnrctl -stat $LISTENER

• Clean: Scans the process table for tnslsnr $LISTENER and kills it• Action: Performs predefined actions on a resource


Common Oracle ConfigurationsThe diagram shows some common Oracle VCS configurations. • In the resource diagram on the left:

– Oracle data files are written to raw volumes. A Mount resource is not required.

– Oracle binary files are located on shared storage on a file system with a separate volume that is part of the service group configuration, so they are highly available.

– The Oracle instance must be online before the listener is started.• In the resource diagram on the right:

– The Oracle resource must come online last. – The log and data files are located on separate file systems and volumes, but

they use the same disk group.– The binaries are on local storage and so are not part of the service group

configuration.

Note: As a best practice, minimize the number of volumes and disk groups used in database service groups. As a general rule, use one disk group per database. Large numbers of volume management objects complicate administration and can impact performance during startup.

Common Oracle Configurations

Oracle

BinaryVolume

Mount

Netlsnr

NIC

IP

DiskGroup

DataVolume

Oracle

DiskGroup

LogVolume

Mount

Netlsnr

NIC

IP

DataVolume

Mount


2

Alternative Oracle ConfigurationsYou can use variations of resources and dependencies in an Oracle service group to accommodate different configurations. Consider these alternatives:• The Oracle resource is not required to be dependent on the IP resource.

However, there is little purpose in bringing the database online unless the IP address is available and clients can connect to the database.

• Regardless of whether you link the Oracle resource to the IP resources, the Netlsnr resource should have a dependency on the IP resource.

• If Net8 is used, a Netlsnr resource can be configured with no dependency on the database. The Net8 listener can be started independently of the database.


Database Failover BehaviorDatabase Failover ScenariosYou can use resource and service group attributes to control how VCS responds when resources or systems fault. A complete discussion of failover behavior is provided in the “VCS Response to Resource Faults” and “Handling System and Communication Failures” lessons in the VERITAS Cluster Server Fundamentals course.

The scenarios in the slide show the practical application of some of the more common failover configurations.

Database Failover ScenariosIf you want the listener to be restarted and never cause failover if it faults: – Set the listener resource to non-critical.– Do not make the listener a child of a critical resource.– Set RestartLimit = x.

If you do not want VCS to forcibly clean or fail over if the database or listener faults:– Set ManageFaults=None for the service group.– Notification on fault, no clean, no failover

Even if a database resource is non-critical, theclean entry point is run if the resource faults. This shuts down the database and leaves it offline.

Even if a database resource is non-critical, theclean entry point is run if the resource faults. This shuts down the database and leaves it offline.

!


2

Failover During Oracle Hot BackupIf a cluster node fails while running a hot backup of an Oracle database, failover to another node can succeed only if the Oracle resource attribute, AutoEndBkUp, is set to a nonzero value.

Otherwise, the database in the backup mode on the failover system cannot be opened and VCS cannot bring the Oracle resource online.

The following errors are displayed to indicate this condition:$ ORA-1110 "data file %s: ’%s’"

or$ ORA-1113 "file %s needs media recovery"

Before VCS can bring the Oracle resource online on the failover system, you must take the data files in the database out of the backup mode and shut down the database so that it can be reopened. Refer to the Oracle documentation for instructions on how to change the state of the database files.

Failover During Oracle Hot BackupIf a fault occurs while a database is in hot backup mode, you can prevent the service group from coming online elsewhere.Set the AutoEndBkUp attribute to 0 to prevent the database from being opened on another node.By default, AutoEndBkUp is set to 1, which takes data files out of backup mode when the Oracle resource is brought online, enabling failover.


Additional Oracle Agent FunctionsIn 4.x versions, the Oracle agent supports two additional entry points you can use to manage database functions from within VCS:• Info: Checks the status of the instance• Action: Performs specified actions, such as backing up the Oracle database,

changing the database state, and suspending and resuming a database

You can initiate these types of operations using the hares command, as shown on the following pages.

This can be useful for scripting common database administration tasks that can be initiated from the VCS operator or administrator.

Other Oracle Agent FunctionsYou can use the info and action entry points ofthe Oracle agent to perform additional tasks.

Info: Provides static and dynamic information about the state of an online resource (such as a database)Action: Enables you to perform predefined actions on a resourceThese entry points are supported in other resource types as well. Their uses related to Oracle are described in more detail in the following sections.


2

Predefined Actions for OracleThe Oracle agent supports the action entry point, which enables you to perform predefined actions on a resource.

The Oracle agent supports the actions shown in the slide. The Netlsnr agent supports two of these actions:• VRTS_GetInstanceName• VRTS_GetRunningServices

Predefined Actions for Oracle

Resumes a suspended databaseDBResume

Suspends a database (Oracle 8i and later)DBSuspend

Enables RESTRICTED mode for the database sessionDBRestrict

Disables RESTRICTED mode for the database sessionDBUndoRestrict

Places a tablespace in backup modeDBTbspBackup

Retrieves the list of processes monitored by the agentVRTS_GetRunningServices*

Retrieves the name of the configured instanceVRTS_GetInstanceName*

DescriptionKeyword (Action)

*Also applies to Netlsnr resources


Initiating ActionsTo perform an action on a resource, use the hares command with the following syntax:hares -action res token [-actionargs <arg1 ...>] [-sys system]

The keyword represented as token in the command line example, must be one of the keywords shown in the left column on the “Predefined Actions for Oracle“ slide. Example command lines are shown in the slide.

Custom actions can be developed enabling you to perform additional actions. See the VERITAS Cluster Server Agent Development Guide for more information.

Initiating ActionsUse the hares command to initiate predefined actions on a resource.

Example usage:hares -action OraSG_Ora DBRestrict

hares –action OraSG_Ora DBTbspBackup \-actionargs mytblspace

You can also develop custom actions for an agent.


2

Retrievable Information Using the Info Entry PointThe Oracle agent info entry point provides static and dynamic information about the state of the database. As described further in the next section, you can use the hares command to obtain predefined types of information shown in the tables on the slide.

When invoked, the info entry point maps one of these keywords to a SQL statement and runs that statement on the database specified by the Oracle resource.

Retrievable Information Using the Info Entry Point

InstanceRoleThreadParallelStartupTimeHostNameDatabaseNameInstanceNameInstanceNoVersion

Shared Pool Percent free

FreeDBfileSizeTotalDBfileSize

Buffer Hits Percent

DatabaseStatusShutdownPendingLogMode

OpenModeLoginsInstanceStatus

Static Information

Dynamic Information


Retrieving InformationYou can manually invoke the info entry point using the hares command. • The first command line shown in the slide causes the agent to update the value

of the ResourceInfo attribute for the specified resource on the specified system if the resource is online. If the Info entry point is successful, no output is displayed.Note: ResourceInfo is a temporary attribute—its values are never dumped to the main.cf file.

• The second command line displays the value of the ResourceInfo attribute.

As an alternative to manually running hares -refreshinfo, you can set the InfoInterval resource type attribute to a nonzero value. This tells the agent to run the info entry point periodically, according to the value of the InfoInterval attribute.

To set the InfoInterval type attribute, use the following syntax:hatype -modify Oracle InfoInterval seconds

Note: You can add SQL statements to the /opt/VRTSvcs/bin/Oracle/resinfo.sql file to obtain additional information using the info entry point. For information on how to customize this file, see the VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide.

Retrieving InformationUse the hares command to retrieve informationabout an online resource.

Example usage:hares –refreshinfo OraSG_Ora –sys sys_namehares –value OraSG_Ora ResourceInfo

You can also add SQL statements to the following file to customize information retrieval using the info entry point:/opt/VRTSvcs/bin/Oracle/resinfo.sql


2

SummaryThis lesson described how to put a database service under control of VCS. An Oracle example was presented to show the resource types and agent functions that are provided with enterprise agents.

Next StepsNow that you understand how Oracle agents manage exported file systems, you can configure other common types of high availability services.


This guide describes each bundled agent in detail.• VERITAS Cluster Server Enterprise Agent for Oracle, Installation and User’s

GuideThis guide provides detailed information on procedures and concepts for configuring and managing Oracle using the VCS enterprise agent.

• VERITAS Cluster Server Agent Developer’s GuideThis guide describes how to customize agents, including the info and action entry points.

• High Availability Design Using VERITAS Cluster Server CourseThis provides guidance on designing local and global clusters to manage database environments.

Lesson SummaryKey Points – VCS provides a variety of agents to manage

databases.– The Oracle agent has specialized functions for

managing Oracle databases.Reference Materials– VERITAS Cluster Server Bundled Agents

Reference Guide– VERITAS Cluster Server Enterprise Agent for

Oracle, Installation and Configuration Guide– VERITAS Cluster Server Agent Developer's Guide– High Availability Design Using VERITAS Cluster

Server Course


Lab 2: Creating Oracle ResourcesLabs and solutions for this lesson are located on the following pages.

Appendix A provides brief lab instructions for experienced students.• “Lab 2 Synopsis: Creating Oracle Resources,” page A-6

Appendix B provides step-by-step lab instructions.• “Lab 2 Details: Creating Oracle Resources,” page B-11

Appendix C provides complete lab instructions and solutions.• “Lab 2 Solutions: Creating Oracle Resources,” page C-11

GoalThe purpose of this lab is to make an Oracle database service highly available.

ResultsEach student’s database service can be started, monitored, and stopped on each cluster system.


Lab 2: Creating Oracle Resources

nameOracle1

nameMount1

nameVol1

nameDG1

nameProxy1

nameIP1

nameApp1*

nameSG1nameSG1

1. Add an Oracle resource.

2. Add a Netlsnrresource.



nameListener1


*This may be aProcess resource if the Application lab was not completed.



Example 3Clustering NFS


IntroductionOverviewThis module describes a recommended practice to prepare resources and manually test NFS services before configuring an NFS service group. In this lesson, you are introduced to the VCS Share and NFS resource types.

ImportanceAlthough understanding how agents work enables you to use any VCS resource type, you need to have an understanding of the specific application service at hand to successfully configure the application in a high availability environment. Understanding how you can use VCS to manage your NFS shares helps you implement highly available NFS servers.

Outline of Topics• NFS Service Overview• Preparing NFS for High Availability• Testing the NFS Service• Configuring an NFS Service Group• NFS Lock Failover in VCS 4.1

Configuring NFS and Share resources in an NFS service group.

Configuring an NFS Service Group

Configure an NFSLock resource to fail over record locks.

NFS Lock Failover in VCS 4.1

Test the NFS service on each target system.

Testing an NFS Service

Prepare NFS components for the VCS environment.

Preparing NFS for High Availability

Describe how NFS services work in a high availability environment.

NFS Service Overview


Topic


Example 3 Clustering NFS 3–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

3

NFS Service OverviewNetwork File System (NFS) provides file system services to remote clients over the public network. As with most application services, NFS service components can be categorized into three basic components:• Application—the NFS processes (daemons) that handle file system mount

requests (mountd), client requests (nfsd), and the share name configuration• Storage—data storage resources • Network—resources that enable clients to access the service from the network

The diagram shows a simple example of an NFS service.

NFS Service Overview

NFS daemonsShare

Client NetworkRequests

IP AddressNIC

File SystemVolumeDisk Group

NetworkNetwork

StorageStorage

ApplicationApplication


NFS Configuration and Migration Process OverviewUse the process shown in the diagram to prepare and test NFS services on each system before putting them under VCS control. Use the design worksheet to obtain and record information about the service group and each resource. This is the information you need to configure VCS to control these resources.

Details are provided in the following sections.

NFS Configuration and Migration Process

Ready for VCS

Ready for VCS

More Systems?

More Systems?

N

Y

Perform one-timeconfiguration tasks on

each system.

Perform one-timeconfiguration tasks on

each system.

Start, verify, and stop NFS services on one system at a time.

Start, verify, and stop NFS services on one system at a time.

Create or prepare NFS-related resources initially on each system for each NFS service that will be put under VCS control; this is not part of VCS operation.Manually test NFS shared file systems on each system that is a startup or failover target for that service group before puttingthe file systems under VCS control.


3

Preparing NFS for High AvailabilityIdentifying NFS Components and RelationshipsVCS manages NFS services in the same way it manages any other application service. Resources are contained within the service group for components, and dependencies are defined to ensure that VCS brings the supporting resources online in the necessary order.

The diagram shows the resources in an NFS service and their relationships. You can use this diagram as a guide to configure and test NFS resources before bringing them under VCS control.

Notice that unlike many other resource dependency diagrams, the IP resource is the topmost parent. If the IP address came up before the NFS shared file system is available, a client attempting to connect to the share gets a stale file handle and never tries to mount the file system again. However, if the IP address is not configured yet, the client retries to connect to the file system.

Identifying NFS Components and Relationships

DiskGroup

NIC

IP

Mount

Share

NFS

Volume

Shared storageresources:– Disks or components

of a logical volume manager

– File systems to be mounted

Network resources:– IP addresses– Network interfaces

Application (NFS) resources:– NFS (nfsd, mountd)– Share names


Configuring Shared Storage for NFSAfter you create the disk group, volume, and file system to be shared, verify that the major and minor devices number for storage devices are the same on each system.

NFS services use the major and minor number of the block device to identify the exported file system. For the identity of the exported file system to be the same no matter which system in the cluster provides NFS services, the major and minor numbers of the storage device must be the same on each system in the cluster.

Generally, major and minor numbers match. However, the values are affected by circumstances, such as the order in which software is installed. For example, if you install VxVM and VxFS in different orders on different systems, the major numbers for devices can be different.

Configuring Shared Storage for NFSMajor and minor numbers for block devices usedfor NFS services must be the same on each system.

Stale File HandleError

NFSRequest

NFSRequest

NFSResponse

Before failover After failover


3

Major Numbers for Volumes

To display and change the major numbers on the VERITAS Volume Manager (VxVM) volumes on shared storage, use the following platform-specific instructions.

Solaris1 Find the number corresponding to vxio and vxspec in the

/etc/name_to_major file. Type: grep vx /etc/name_to_major

2 Select major numbers that are not in use by any other device by viewing the /etc/name_to_major file on each system.

3 To make the major numbers the same on both systems use the haremajor command to change the major numbers for vxio and vxspec on S1 to match S2. For example, type: haremajor -vx 90 91Note: It may be necessary to change the major numbers on more than one system to make them unique for all systems.

4 Reboot each system on which haremajor was successful changing major numbers.

Repeat this procedure for each volume used for an NFS file system.

Major Numbers for VolumesVerify that each system has the same major number for the block device of the shared volume. Verify that major numbers are also unique within a system.Verify that the major numbers match on all systems.If major numbers do not match, reset one system’s numbers to match the other system.


grep vx /etc/name_to_major

vxdmp 87vxio 88vxspec 89


vxdmp 87vxio 88vxspec 89 S1


vxdmp 89vxio 90vxspec 91


vxdmp 89vxio 90vxspec 91 S2


AIX

Use the lvlstmajor command to find out what the available major numbers are for the file system. On each system:1 Check the vxio driver numbers. Major numbers match the driver number.

# haremajor -v 47

2 Find an unused driver number on all systems.# haremajor –a 50...

3 Change each system to the same driver number.# haremajor -s 50vxio major number changed to 50

4 Reboot each system.# reboot

HP-UX

See the HP-UX system documentation for information about changing LVM device numbers.

Linux

Major and minor device numbers on Linux systems depend entirely on the order of the SCSI devices—internal and external are found during system startup. Therefore, you cannot ensure that major and minor numbers match on different Linux systems.

Because of this platform-specific limitation, your clients may have problems with stale file handles to NFS-mounted file systems after a failover. To mitigate this issue, follow these guidelines:• Ensure that shared storage is on a different controller (than the internal

controller) on each node.• Verify that the SCSI driver for the mentioned controller is loaded after the

internal controller (if such a driver exists).• If both controllers are driven by the same driver then ensure that the internal

controller is on a lower PCI slot.


3

Minor Numbers for Volumes

Use the ls command to determine whether the minor numbers for volumes match on each system. ls –l /dev/vx/rdsk/dgname/volume_name

For example:ls –l /dev/vx/rdsk/datadg/vol01

crw------- 1 root root 99 0x000005 Aug 17 07:30 /dev/vx/rdsk/datadg/vol01

If the minor numbers of the volume devices do not match, change the base minor number for the disk group to a common value on all systems:vxdg reminor dgname new_base_number

For detailed information on using haremajor and vxdg reminor, refer to the VERITAS Cluster Server Installation Guide. You can also refer to the Volume Manager command vxdg manual page for instructions on reconciling the VERITAS Volume Manager minor numbers.


Configuring Networking for NFSIn a high availability environment, the IP address and host name that are used by the client to access the NFS service should not be tied to a specific system because the same service can be provided by any system in the cluster.

Ensure that you have an administrative IP address configured on the network interface that clients will use to access the NFS shared file systems. The administrative IP address must be configured to enable VCS to set up the virtual IP address for the NFS service group.

Set up any other network configuration files needed for your public network environment. Also, configure the name service to enable clients to connect to the shared file system using a virtual host name and IP address that will migrate among the systems that can host the NFS service.

Configuring Networking for NFSAn administrative IP address must be configured during system startup for the interface used by this service. Add entries to the name server to enable clients to access the virtual IP address and host name used to connect to the shared file system.Set up any other applicable network configuration files.


3

Configuring NFS (Application) Components

Checking the Startup Configuration

To share a file system in a high availability environment where the file system can be mounted and shared from any one of the systems, you must prevent the operating system from controlling the NFS resources:

Solaris• The file system mount point should not have an entry in the /etc/dfs/

dfstab file.• The file system should not have an entry in the /etc/vfstab file or the entry

should say no to mount at boot.AIX

• The file system mount point should not have an entry in /etc/exports.• The file system should have an entry in /etc/filesystems with

mount=false.HP-UX

• The file system mount point should not have an entry in /etc/exports.• The file system should not have an entry in the /etc/fstab file.

Linux• The file system mount point should not have an entry in /etc/exports.• The file system should have an entry with noauto in /etc/fstab.

Checking the Startup ConfigurationVerify that the file system is not mounted or shared at system startup to enable VCS control:

Solaris:– No file system mount point entry is placed in /etc/dfs/dfstab.

– The /etc/vfstab entry should say no or manual mount at boot.

AIX: No entry in /etc/exports; a mount=false entry /etc/filesystemsHP-UX: No entries in /etc/exports or /etc/fstabLinux: No entries in /etc/exports; noauto entry in /etc/fstab


Testing the NFS ServiceBefore configuring a service group in VCS to manage an NFS service, test it on each system that can be a startup or failover target for the service group. Following this best practice recommendation ensures that VCS will successfully manage the shared file systems when you configure the service group.

Detailed information about setting up shared storage and network resources was provided earlier in the course. This lesson focuses on the additional steps needed for testing NFS services.

Testing Procedure

Bring up resources.Bring up resources.

Ready for VCS

Ready for VCS

More Systems?

More Systems?


Bring up resources.Bring up resources.

N

Y

Test remote access.Test remote access.

S1

S2…Sn

Test remote access.Test remote access.


Shared storageShared storage

Start up all resources in dependency order.

NFS daemons, exports NFS daemons, exports

Virtual IP addressVirtual IP address


3

Starting NFS Daemons The NFS daemons are usually configured to start during system boot. To verify the appropriate daemons are running, and start them, if necessary:

Solaris

ps -ef | grep nfs mountd statd lockd

/usr/lib/nfs/nfsd -a nserver

/usr/lib/nfs/mountd

/usr/lib/nfs/statd

/usr/lib/nfs/lockdAIX

lssrc –g nfs

startsrc –g nfsHP-UX

ps -ef | grep nfs

/usr/sbin/nfsd -a nserver/usr/sbin/rpc.mountd

Linux

/sbin/service nfs status

/sbin/service nfs start

/usr/sbin/rpc.mountd

Starting NFS DaemonsVerify that the NFS daemons are running or start them if necessary:1. Determine whether the appropriate NFS daemons

are running:Solaris: nfsd, mountd, statd, lockdAIX: nfsd, rpc.mountd, rpc.lockd, rpc.statdHP-UX: nfsd, rpc.mountd, rpc.lockd, rpc.statdLinux: nfsd, mountd

2. Start the NFS daemons, if they are not already running.



Exporting the File SystemAfter you have verified that the file system is mounted and the NFS daemons are running, export (share) the mount point of the file system with specified share options, if any. Then, verify that the share is configured.

Solaris

share mount_pointshare | grep mount_point

AIX

smitty mknfsexp: creates/updates /etc/exports

exportfs –a

showmountHP-UX

/usr/sbin/exportfs [–F fstype] mount_pointexportfs | grep mount_point

Linux

/usr/sbin/exportfs [–F fstype] mount_pointexportfs –a

Note: Whenever you change /etc/exports, you must notify the NFS daemon using either of these methods:• /sbin/service nfs reload• /sbin/exportfs -r

Exporting and Testing the File SystemAfter verifying that the file system is mounted andNFS daemons are running:1. Export the file system.2. Verify that the file system is exported.3. Configure the virtual IP address.4. Mount the file system from a remote client on

the public network.



3

Migrating a ShareBefore you place an NFS service under VCS control, it is recommended that you manually migrate the service between all intended target systems. By doing so, you:• Ensure that your operating system resources are properly configured on all

potential target cluster systems.• Observe the types of tasks VCS performs to bring a service online, monitor the

status of each resource, and switch a service to another system.

When you have completed the preparation of the testing tasks, you can document or verify that your design worksheet is complete, and then configure VCS to manage the NFS services.

Migrating a Share

NetworkNetwork

StorageStorage

ApplicationApplicationS1

S2

NFS daemonsShareFile System

VolumeDisk Group

IP AddressNIC


Configuring an NFS Service GroupUse the online or offline configuration process described earlier in the course to create a service group and add resources after you have manually tested the NFS configuration on each system.

The NFS Resource

Entry Points• Online: Checks whether the nfsd and mountd processes are running

If processes are not running, the online entry point starts them and exits.• Offline: None• Monitor:

– Solaris, HP-UX, Linux: Monitors TCP and UDP versions of the processes by sending RPC calls clnt_create and clnt_call to the RPC serverIf calls succeed, the resource is reported as online.

– AIX: Determines whether processes are active by querying the SRC

Optional Attributes• Nservers: Specifies the number of concurrent NFS requests the server can

handleDefaults: Solaris 16; HP-UX 4; AIX 0, obtains the value from the SRC

• Nproc (Linux-only): Specifies the number of processes of nfsd to start The default value is 8.

• Protocol (HP-UX only): Specifies the UDP or TCP protocol

The NFS ResourceThe NFS agent:

Starts the NFS processes if they are not already runningNo offline; OnOnly resourceChecks for active NFS-related processes (based on platform type)


APExportNFSResource Name

32NserversOptional Attributes

NFSResource Type

APExportSGService Group NameSample ValueResource Definition

NFS APExportNFS (Nservers = 32

)

NFS APExportNFS (Nservers = 32

)

main.cfmain.cf


3

Configuring a Share Resource

Entry Points• Online: Shares an NFS file system• Offline: Unshares an NFS file system• Monitor: If there is an entry in the shared table for the file system specified by

PathName, the resource is reported as online.

Optional Attributes

Options: Specifies options to be used with the share command while sharing the file system. Defaults are:

Solaris, AIX, HP-UX

No options are specified by default.Linux

ro, async, wdelay, root_squash

Note: Linux has an additional required attribute, Client, which specifies the host to which the PathName directory is exported.

Configuration Prerequisites• The file system and share should not be configured automatically during

system startup.• The Mount and NFS resources must be configured in VCS.

Configuring a Share ResourceThe Share agent:

Shares and unshares an NFS file system Reads the share table to check for an entry for the file system


/AP/export1PathNameRequired Attributes

APExportShareResource Name

OptionsOptional Attributes

ShareResource Type


Share APExportShare (PathName = "/AP/export1"

)

Share APExportShare (PathName = "/AP/export1"

)

main.cfmain.cf


NFS Considerations in a VCS EnvironmentNFS-specific issues to consider when configuring a high availability NFS service group include:• Increase RestartLimit to avoid failover when daemons fail sporadically.• Increase the Nservers value if the NFS daemon needs more threads. If you

need to increase this value after the service group is running:a Freeze the service group.b Change the Nservers attribute value.c Stop and restart NFS daemons outside of VCS to restart with the increased

Nservers value.Note: You cannot simply take the NFS resource offline and bring it back online. The NFS daemons are not stopped when you take the NFS resource offline because the NFS resource type is on-only.

d Probe the service group to clear the faulted NFS resource.e Unfreeze the service group.

• Avoid using a cluster system that can be a failover target for the NFS service as a client to mount the shared file system. If you have the file system mounted and the NFS service group fails over to that system, the client is stopped as a prerequisite to starting the NFS server.Also, it is generally good practice to avoid relying on services outside the cluster.

NFS Considerations in a VCS EnvironmentScenarios:

Consider setting RestartLimit to a higher value than the default of 1 so that VCS attempts to restart the NFS daemons before failing over to another system.Increase the value of Nservers (Nproc on Linux) if you need to increase the number of client connections. Note: If you increase this attribute while the resource is online, freeze the service group, stop and restart NFS outside of VCS, probe, and then unfreeze the service group.Best practice recommends that you do not use cluster nodes as NFS clients.


3

NFS Lock Failover in VCS 4.1VCS 4.1 introduced support for NFS lock failover by implementing a new bundled agent, NFSLock. This agent works with the existing recovery mechanisms inherent in the Network Lock Manager component of NFS.

NFS Locking OverviewNFS record locking is handled by lockd and statd, two cooperating daemons, which run on both NFS server and client systems. • lockd

– NFS client: The lockd process on an NFS client accepts locking requests from applications for NFS mounted file systems.

– NFS server: The lockd process on an NFS server accepts locking requests from lockd processes on NFS clients.

• statd– NFS client: The statd process on an NFS client system keeps track of the

list of servers where it holds locks. – NFS server: The statd process on an NFS server system tracks the clients

holding locks on that server. The client information is stored on the server in /var/statmon/sm.

NFS Locking Overview

lockd lockd

statdstatd

/var/statmon/sm

client1client2. . .

NFS Client

NFS Server

RPC lock requests

Status messages


Basic Lock Process

To illustrate how NFS lock works, this example describes locking between one NFS client and one NFS server, where an application running on the client requests a remote lock:1 lockd registers the server name hosting the requested remote lock with the

local statd process.2 statd on the client contacts statd on the server to determine whether the

server is running.3 statd notifies lockd that the server is running.4 lockd on the client then communicates with lockd on the server to request

the lock.5 statd on both the client and server systems periodically checks the status of

each statd process for registered systems


3

Lock Recovery

The lockd and statd processes also provide recovery for locks in the event that either a client or service crashes and reboots.• Client failure: When an NFS client crashes (as determined by statd):

a lockd on the server holds the locks the client established before the crash.b When the client reboots, statd on the client sends new state information

to statd on the server, which passes the information to lockd.c Applications on the client restart and request locks again as needed.Note: If a client with locks on a server crashes and is not restarted, the locks are held indefinitely.

• Server failure: When an NFS server crashes and reboots:a statd on the server sends a message to statd on each registered client

(listed in /var/statmon/sm) indicating that the server has rebooted. b The client statd process informs the client lockd process that the server

has rebooted.c The client lockd process attempts to reclaim the locks from the server.d The server lockd process waits for a defined grace period (default 45

seconds) to grant reclaimed locks before the server revokes the locks and resumes normal operation.

NFS Lock Recovery After Server Reboot

lockd lockd

statdstatd

/var/statmon/sm

client1client2. . .

client2

server1

"Reclaim lock"

"Server rebooted" a

b

cd

Reclaim grace period

(after reboot)


VCS Failover and NFS Lock RecoveryVCS 4.1 includes several cooperating mechanisms that work with the NFS lock recovery mechanisms provided by lockd and statd.

The smsyncd Daemon

The smsyncd process provides two functions:• In oneshot mode, smsyncd copies the list of clients holding locks from a

specified shared storage location to /var/statmon/sm. statd then uses these entries to contact clients for recovery.

• In daemon mode, smsyncd copies entries from /var/statmon/sm to a specified shared storage location once per second. Each online NFSLock resource has a corresponding instance of smsyncd running.

The NFSLock Agent

The NFSLock agent performs these functions:• online: Kills lockd to start the grace period and kills statd to initiate

recovery • offline: Kills lockd to release locks and kills statd to break the client/

server connection• monitor: Determines whether lockd, statd, and smsyncd are running and

restarts them if necessary

VCS Failover and NFS File Lock RecoveryVCS 4.1 provides several cooperating mechanisms to enable a failover system to acquire NFS lock information for recovery:

The system monitor synchronization daemon, smsyncd, copies /var/statmon/sm entries to shared storage once per second.The NFSLock agent:– Kills lockd and statd on the failover system to initiate lock

recovery (online) and release locks (offline)– Monitors lockd, statd, and smsyncd

The nfs_restart trigger:– Runs smsyncd in oneshot mode to copy entries from shared

storage to /var/statmon/sm on the failover system– Restarts nfsd, statd, and lockd and starts smsyncd


3

The nfs_restart Trigger

The nfs_restart trigger is configured by default and runs for every service group when the service group is brought online. The trigger determines whether a service group contains an NFS resource and takes no action if no NFS resources are present.

If a service group has an NFS resource, the nfs_restart trigger runs smsyncd (in oneshot mode) to copy lock entries from shared storage to /var/statmon/sm. Then the trigger restarts nfsd, statd, and lockd and starts smsyncd (in daemon mode).


The NFSLock Resource

The NFSLock resource has one required attribute, PathName, which specifies the root directory of the file system for which the resource is managing record locks.

The resource diagram in the slide shows how the NFSLock resource fits in the service group dependency tree.

The NFSLock Resource

APExportNFSLockResource Name

/AP/export1PathNameRequired Attributes

NFSLockResource Type


DiskGroup

NIC

IP

Mount

Share

NFS

Volume

NFSLock

VCS 4.1 onlyVCS 4.1 only


3

NFS Service Group Startup ProcessThis process describes how an NFS service group is started, showing the relationships between the NFS lock recovery components. The same events occur when an NFS service group is initially brought online, switched from another system, or failed over due to a fault.1 Run the nfs_restart trigger.

For a service group with an NFS resource, the trigger:a Runs smsyncd in oneshot mode to copy entries from shared storage to the

local /var/statmon/sm directoryb Starts nfsd, statd, and lockd if they are not already running

Note: These daemons are usually configured to start during system boot.

c Starts smsyncd in daemon mode, which copies /var/statmon/sm entries to shared storage once per second

2 Bring the DiskGroup, Volume, and Mount resources online.3 Bring the NFS resource online. This starts monitoring nfsd, mountd, statd,

and lockd.4 Bring NFSLock online. The online entry point:

a Kills lockd to start the grace periodAt this point, lockd allows 45 seconds (default) for clients to reclaim locks.

NFS Service Group Startup Process

Run the nfs_restart trigger (runs smsyncd).11

Bring DiskGroup, Volume, and Mount online. 22

Bring NFS online (monitors NFS daemons).33

Bring NFSLock online (kills lockd, statd).44

Bring remaining resources online.55

VCS performs these actions during startup to enable lock recovery.


b Kills statd to initiate recoverystatd on the server contacts statd on all registered clients indicating that NFS has (re)started and locks can be reclaimed.

c After the grace period expires, lockd resumes normal operation and accepts locking requests from any client.

5 Bring remaining resources online, in dependency order.


3

NFS Lock Failover LimitationsConsider these limitations when implementing NFSLock resources in an NFS service group:• If there are multiple service groups online on each node in the cluster, failing

over a single service group disrupts the locking service for all other groups online on both the failing-from and failing-to nodes.

• There is a time period where the virtual IP address is online but locking services are not registered on the server. Any NFS client trying to acquire a lock in this interval would fail and receive an ENOLCK error.

• Every second, the smsyncd daemon copies the list of clients that hold the locks on the shared file system in the service group. If the service group fails before smsyncd has an opportunity to copy the client list, the clients may not receive a notification after the service group is brought up. In this case, the locks cannot be recovered.

• You must provide a fully qualified hostname (nfsserver.princeton.edu) for the NFS server while mounting the file system on the NFS client. If you do not use a fully qualified hostname, or if you use a virtual IP address (10.122.12.25) or partial hostname (nfsserver), NFS lock recovery fails.

• Ensure that the NFS client stores the same information for the NFS server as the client uses while mounting the file system. For example, if the NFS client mounts the file system using fully qualified domain names for the NFS server, the NFS client directory, /var/statmon/sm, should also have a fully qualified domain name after the acquisition of locks.

NFS Lock Recovery LimitationsFailover of one service group with an NFSLockresource disrupts locking for all other online service groups with NFSLock.Clients may receive lock errors during the period when the virtual IP address is up but locking services are not registered on the server.The one-second intervals of smsyncd writes leave a window where a client added during that write period may not be notified after for recovery.Clients must use FQDN host names for the server.


SummaryThis lesson described how to put an NFS service under control of VCS.

Next StepsNow that you understand how NFS agents manage exported file systems, you can configure other common types of high availability services.


This guide describes each bundled agent in detail.• VERITAS Cluster Server User’s Guide

This guide provides detailed information on procedures and concepts for configuring and managing VCS clusters.

• VERITAS Cluster Server Installation GuideThis guide provides detailed information about configuring NFS in preparation for clustering NFS resources.

Lesson SummaryKey Points – You can use the NFS and Share agents to

manage NFS exported file systems.– Prepare and test your NFS-related services

before configuring service groups.Reference Materials– VERITAS Cluster Server Bundled Agents

Reference Guide– VERITAS Cluster Server User's Guide– VERITAS Cluster Server Installation Guide


3

Lab 3: Creating NFS ResourcesLabs and solutions for this lesson are located on the following pages.

Appendix A provides brief lab instructions for experienced students.• “Lab 3 Synopsis: Creating NFS Resources,” page A-16

Appendix B provides step-by-step lab instructions.• “Lab 3 Details: Creating NFS Resources,” page B-31

Appendix C provides complete lab instructions and solutions.• “Lab 3 Solutions: Creating NFS Resources,” page C-35

GoalThe purpose of this lab is make an NFS service highly available.


ResultsEach student’s NFS service can be started, monitored, and stopped on each cluster system.

Lab 3: Creating NFS Resources

nameShare1

nameMount1

nameVol1

nameDG1

nameNFS1

nameProxy1nameProcess1*

nameSG1nameSG1

1. Add an NFS resource.

2. Add a Share resource.



nameIP1

*This may be an Application resource if you completed that lab exercise.




Appendix ALab Synopses

A–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Lab 1 Synopsis: Creating an Application ResourceStudents work separately to modify their own nameSG1 service groups to add NFS and Share resources.

Step-by-step instructions for this lab are located on the following page:• “Lab 1 Details: Creating an Application Resource,” page B-3

Solutions for this exercise are located on the following page:• “Lab 1 Solutions: Creating an Application Resource,” page C-3

Lab AssignmentsObtain the values for your classroom.

Object Values

Location of application script files: loopy_dir


nameApp1

nameMount1

nameVol1

nameDG1

nameProxy1



nameSG1nameSG1




Appendix A Lab Synopses A–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

A

Each student configures their own Application resources.

1 Delete the nameProcess1 resource.

2 Set up the application.

a Copy the scripts used by the application resource to your /name1 file system from the location provided by your instructor:

Script location:

_________________________________________________________loopy_dir

Copy these scripts:

startloopystoploopyloopy

b Verify that the scripts are executable.

Adding an Application Resource

Resource Definition Sample Value Your Value

Service Group nameSG1

Resource Name nameApp1

Resource Type Application

Required Attributes

StartProgram /name1/startloopy name 1/name1

StopProgram /name1/stoploopy /name1

PidFiles /name1/loopypid

Critical? No (0)

Enabled? Yes (1)


c View the startloopy and stoploopy scripts to determine how they are used.

3 Add the Application resource to the service group using the information in the design worksheet and bring it online.

4 Save the cluster configuration.





Required Attributes




Critical? No (0)

Enabled? Yes (1)


A

Link the resources according to the design worksheet.

1 Test the service group by switching it between systems in the cluster.

2 Set the Application resource to critical.

3 Save the cluster configuration and when all students have completed the lab, close the cluster configuration.

4 Fault resources in the group to test failover.

Remove the link from the nameApp1 resource to the nameIP1 resource. An IP address is not strictly needed for the loopy process to run. Removing the link provides a simplified service group for any subsequent labs your instructor may ask you to complete.

1 Unlink the nameApp1 and nameIP1 resources.

2 Save the cluster configuration and when all students using the cluster are finished, close the cluster configuration.

Linking the Resources

Resource Dependency Definition


Parent Resource Requires Child Resource

nameApp1 nameIP1

nameApp1 nameMount1

Testing the Application Service Group

Resetting the Service Group


Lab 2 Synopsis: Creating Oracle ResourcesStudents work separately in most sections of this lab to modify their own nameSG1 service groups to add Oracle and Netlsnr resources.

Step-by-step instructions for this lab are located on the following page:• “Lab 2 Details: Creating Oracle Resources,” page B-11

Solutions for this exercise are located on the following page:• “Lab 2 Solutions: Creating Oracle Resources,” page C-11


nameOracle1

nameMount1

nameVol1

nameDG1

nameProxy1

nameIP1

nameApp1*

nameSG1nameSG1





nameListener1






A

Lab AssignmentsObtain the preparation information for your classroom.

1 Verify that your nameSG1 service group is online on your system; if it is not, switch it or bring it online on your system.

2 Create a link for your SID to the /name1 file system on your system. You are assigned a SID based on your system, as shown in the design worksheet.

For SID ORCL1: /orcl1 —> /name1




Object Sample Value Your Value

Oracle instance name for your service group

train1, train5, train9: ORCL1train2, train6, train10: ORCL2train3, train7, train11: ORCL3train4, train8, train12: ORCL4

Location of Oracle data files

Script used for data file extraction

ORCL1: ORCL1_files.sh (train1, train5, train9)

ORCL2: ORCL2_files.sh(train2, train6, train10)



Preparing an Oracle Instance


3 Create a link for your SID to the /name1 file system on the other system in the cluster.

Note: Be sure to complete this step. If you do not create the link on both systems, you cannot switch the service group later.





4 Change to the directory containing the scripts for extracting Oracle data files. Obtain that location from your instructor and record the directory.

Location of scripts:

________________________________________________________

5 Extract the Oracle datafiles by running the script for your SID, supplied in the worksheet and recorded in the follow space. The script runs tar to copy the Oracle data files to the shared storage volume you have imported.

Script name:_________________________________

6 Modify the /etc/hosts file on each system in the cluster to add an entry for the Oracle SID (ORCL#) corresponding to the IP address used for the nameIP1 resource.

7 Ensure that the virtual IP address is configured.


A1 Start Oracle on your system.

2 After the Oracle instance is started, start the Listener. Your listener name is the same name as your SID.

3 Stop the Listener (as user oracle).

4 Stop the Oracle instance (as user oracle).

5 Switch your nameSG1 service group to the other system and perform the same test.

6 Switch your nameSG1 service group to your system and exit from their system.

All lab participants install the Oracle Enterprise agent on their own systems. If you have no lab partner, install the agent on each system in the cluster.

1 Record the installation directory provided by your instructor.

Installation directory:

________________________________________________________install_dir

2 Copy the agent package to /tmp on your system.

3 Install the VRTSvcsor agent package.

4 If you do not have a lab partner, install the agent on each system.

Testing Oracle and the Listener

Installing the Oracle Agent


.

Work with your lab partner to add the resource types for Oracle to the cluster.

Use the information in the design worksheet to add an Oracle resource to your nameSG1 service group.

Adding a Resource Type Definition

Configuring an Oracle Resource


Service Group nameSG1Resource Name nameOracle1

Resource Type Oracle

Required Attributes

Home /oracle

Owner oracle

Sid ORCL#

Optional Attributes

Pfile /orcl#/admin/ORCL#/pfile/initORCL#.ora

Critical? No (0)

Enabled? Yes (1)


A

Use the information in the design worksheet to add a Netlsnr resource to your nameSG1 service group.

Note: If using a version of VCS earlier than 4.0, skip this section and proceed to the “3.5 Alternate: Adding a Sqlnet Resource” section.

Adding a Netlsnr Resource


Service Group nameSG1Resource Name nameListener1

Resource Type Netlsnr

Required Attributes

Home /oracle

Owner oracle

TnsAdmin /oracle/network/admin

Listener ORCL#

Critical? No (0)

Enabled? Yes (1)


If your class is using a version of VCS earlier than 4.0, use the information in the design worksheet to add a Sqlnet resource to your nameSG1 service group.

Link the Oracle and Listener resources according to the design worksheet.

3.5 Alternate: Adding a Sqlnet Resource



Resource Type Sqlnet

Required Attributes

Home /oracle

Owner oracle


Listener ORCL#

Critical? No (0)

Enabled? Yes (1)



Service Group ClusterService


nameOracle1 nameMount1

nameListener1 nameOracle1

nameListener1 nameIP1


A

Test the service group by switching it between the systems. Then, set the Oracle and Netlsnr resources to critical and test failover.

In this lab section, each student works on their own to remove the Oracle-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.

Note: If you are configuring detail monitoring as shown in the optional lab section, skip this section and return when you have completed configuring and testing detail monitoring.

1 Verify that Oracle is online on a system. Switch the nameSG1 service group to your local system, if it is not already online.

2 Log in as the oracle user on that system and start sqlplus as internal.

3 Create an Oracle user named vcs with the password vcs.

4 Grant create session permissions to vcs while in sqlplus.

5 Create a table named vcs.vcstest and insert a row. This table is used to test Oracle from VCS.

6 Select the data in the vcs.vcstest table to verify that a row is present.

7 Freeze the nameSG1 service group temporarily.

Testing the Service Group


Optional Lab: Detail Monitoring of Oracle


1 Encrypt the vcs password using /opt/VRTSvcs/bin/vcsencrypt. Use the output when setting the Pword attribute in the next step.

2 Modify up the nameOracle1 resource for detail monitoring. Use the values shown in the outlined portion of the worksheet.

Modifying the Oracle Resource


A3 Unfreeze the nameSG1 service group.


5 Start sqlplus and verify that the vcstest table is being updated at the monitor interval.




Required Attributes

Home /oracle

Owner oracle

SID ORCL#

Optional Attributes


User vcs

Pword vcs

DetailMonitor 1

Table vcstest

MonScript ./bin/Oracle/SqlTest.pl

Critical? No (0)

Enabled? Yes (1)


Lab 3 Synopsis: Creating NFS ResourcesStudents work separately to modify their own nameSG1 service groups to add NFS and Share resources.

Step-by-step instructions for this lab are located on the following page:• “Lab 3 Details: Creating NFS Resources,” page B-31

Solutions for this exercise are located on the following page:• “Lab 3 Solutions: Creating NFS Resources,” page C-35

Note: Skip this section if you are working on HP-UX or Linux systems.

1 Determine the block device name that will be shared by looking at the nameMount1 resource attributes in the nameSG1 service group.

2 Verify that the major and minor numbers are the same for each system in the cluster. If the major numbers do not match, change the numbers for vxio and vxspec to make them unique and match.

Preparing an NFS Service


nameShare1

nameMount1

nameVol1

nameDG1

nameNFS1


nameSG1nameSG1





nameIP1





A

Using the information in the design worksheet, add an NFS resource to your nameSG1 service group.

Using the information in the design worksheet, add a Share resource to your nameSG1 service group.

Adding an NFS Resource


Service Group nameSG1Resource Name nameNFS1

Resource Type NFS

Required Attributes

Critical? No (0)

Enabled? Yes (1)

Adding a Share Resource


Service Group nameSG1Resource Name nameShare1

Resource Type Share

Required Attributes

PathName /name1

Critical? No (0)

Enabled? Yes (1)


Link resources, switch the group between systems, and then verify that clients can connect.

In this lab section, each student works on their own to remove the NFS-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.1 Delete the nameShare1 and nameNFS1 resources.

2 Link the nameProcess1|App1 resource (parent) back to the nameIP1 resource (child).


Testing an NFS Service Group




nameIP1 nameShare1

nameShare1 nameMount1

nameShare1 nameNFS1


Appendix BLab Details

B–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Lab 1 Details: Creating an Application Resource B–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

B

Lab 1 Details: Creating an Application Resource


Lab 1 Details: Creating an Application ResourceStudents work separately to modify their own nameSG1 service groups.

Brief instructions for this lab are located on the following page:• “Lab 1 Synopsis: Creating an Application Resource,” page A-2

Solutions for this exercise are located on the following page:• “Lab 1 Solutions: Creating an Application Resource,” page C-3


Object Values



nameApp1

nameMount1

nameVol1

nameDG1

nameProxy1



nameSG1nameSG1





B


1 Verify that the nameSG1 service group is online and running on your system. If the service group is not online, bring it online on your system.

2 Open the cluster configuration.

3 Take the nameProcess1 resource offline.

4 Disable the nameProcess1 resource.

5 Unlink the nameProcess1 resource from nameIP1 and nameMount1.








Required Attributes




Critical? No (0)

Enabled? Yes (1)




Script location:

_________________________________________________________loopy_dir

Copy these scripts:




9 Add the resource to the service group.

10 Set the resource to not critical.

11 Set the required attributes for this resource and any optional attributes, if needed.

12 Enable the resource.

13 Bring the resource online on your system.

14 Verify that the resource is online in VCS and at the operating system level.



B

Each student configures the dependencies for their own service group.

1 Link the resources according to the design worksheet.

2 Verify that the resources are linked properly.






nameApp1 nameIP1

nameApp1 nameMount1


1 Test the service group by switching it to their_sys in the cluster.

2 Verify that the service group came online properly on the other system.

3 Test the service group by switching it back to your system.

4 Verify that the service group came online properly on the original system.


6 Save the cluster configuration and view the configuration file to verify your changes.

7 When all students have completed the lab, close the cluster configuration.




B





4 When all students using the cluster are finished, close the cluster configuration.



Lab 2 Details: Creating Oracle Resources B–11Copyright © 2005 VERITAS Software Corporation. All rights reserved.

B

Lab 2 Details: Creating Oracle Resources


Lab 2 Details: Creating Oracle ResourcesStudents work separately in most sections of this lab to modify their own nameSG1 service groups to add Oracle and Netlsnr resources.

Brief instructions for this lab are located on the following page:• “Lab 2 Synopsis: Creating Oracle Resources,” page A-6

Solutions for this exercise are located on the following page:• “Lab 2 Solutions: Creating Oracle Resources,” page C-11


nameOracle1

nameMount1

nameVol1

nameDG1

nameProxy1

nameIP1

nameApp1*

nameSG1nameSG1





nameListener1






B













2 Create a link for your SID to the /name1 file system on each system as shown.Note: Be sure to complete this step. If you do not create the link on both systems, you cannot switch the service group later.





3 Obtain the location of the scripts provided for extracting Oracle data files from your instructor and record the directory.

Location of scripts:_________________________________script_dir

4 Change to the script directory.

cd script_dir

5 Extract the Oracle data files by running the script for your SID, supplied in the worksheet and recorded in the following space. The script runs tar to copy the Oracle data files to the shared storage volume you have imported.

Script name:_________________________________



B


a Record the IP address value of your nameIP1 resource.

IP address:_________________________________

b Edit the /etc/hosts files on all systems in the cluster to add your Oracle SID as the host name (ORCL#) with this IP address. Coordinate edits with your lab partner.



1 Use the following commands to start Oracle on your system.

su - oracleSelect the instance for your system, such as ORCL1.sqlplus "/as sysdba"startupexit

2 After the Oracle instance is started, start the Listener. Your listener name is the same name as your SID. Use the following command (as user oracle):

lsnrctl start ORCL#

3 Stop the Listener using the following command (as user oracle):

lsnrctl stop ORCL#

4 Stop the Oracle instance using the following commands (as user oracle):

sqlplus "/as sysdba"shutdownexitexit (back to root)

5 Switch your nameSG1 service group to their system.

6 Log in to their system and perform the same test for Oracle on their system.

rlogin their_sys

a Start Oracle.




B

b After the Oracle instance is started, start the Listener.

lsnrctl start ORCL#

c Stop the Listener.

lsnrctl stop ORCL#

d Stop the Oracle instance.


7 Switch your nameSG1 service group to your system.

8 Exit from their system.


All lab participants install the Oracle Enterprise agent on their own systems. If you have no lab partner, install the agent on each system in the cluster.



____________________________________________________install_dir


3 Change to the /tmp directory.

4 Unzip the compressed package file.

5 Extract the compressed file from the tar file.


7 If you do not have a lab partner, install the agents on each system.



B


1 Use the GUI to open the cluster configuration, import the Oracle resource types, and save the configuration.

2 View the main.cf file to see the addition of the include directive for the OracleTypes.cf file.

3 Check the configuration directory for VCS for the presence of the OracleTypes.cf file.



All students configure their own Oracle resources.

1 Add the nameOracle1 resource to the service group.




5 Bring the resource online on your system and verify that the resource is online.






Required Attributes

Home /oracle

Owner oracle

Sid ORCL#

Optional Attributes


Critical? No (0)

Enabled? Yes (1)


B

Each student configures their own Netlsnr resources.












Required Attributes

Home /oracle

Owner oracle


Listener ORCL#

Critical? No (0)

Enabled? Yes (1)


Note: Each student completes this section if your class is using a version of VCS earlier than 4.0.











Required Attributes

Home /oracle

Owner oracle


Listener ORCL#

Critical? No (0)

Enabled? Yes (1)


B

Each student configures their own dependencies.

1 Link the Oracle and Listener resources according to the design worksheet.










Test the service group by switching it between the systems. Then set the Oracle and Netlsnr resources to critical and test failover.

1 Switch the nameSG1 service group to their system and verify that it comes online.

2 Switch the nameSG1 service group to your system and verify that it comes online.

3 Set the nameOracle1 and nameListener1 resources to critical.


5 When all students working in the cluster are finished configuring their Oracle service groups, close the cluster configuration.




B

In this lab section, each student works on their own to remove the Oracle-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.



2 Take the nameOracle1 and nameListener1 resources offline.

3 Disable the nameOracle1 and nameListener1 resources.

4 Unlink the nameOracle1 and nameListener1 resources.

5 Remove the nameOracle1 and nameListener1 resources.






2 Log in as the oracle user on that system.

su - oracleSelect the instance for your system, such as ORCL1.

3 Start sqlplus as internal.

sqlplus "/as sysdba"

4 Create an Oracle user named vcs with the password vcs. Type the following commands from sqlplus:

create user vcs

identified by vcs

default tablespace users

quota 500K on users;

5 Grant permissions to vcs while in sqlplus:

grant create session to vcs;

6 Create a table named vcs.vcstest and insert a row. This table is used to test Oracle from VCS:

create table vcs.vcstest (tstamp date);

insert into vcs.vcstest (tstamp) values (SYSDATE);

7 Select the data in the vcs.vcstest table to verify that a row is present:

select TO_CHAR(tstamp, 'MON DD, YYYY HH:MI:SS AM')tstamp from vcs.vcstest;

exit (from sqlplus)exit (back to root)



B




Use the values in the table to configure the nameOracle1 resource to perform detailed monitoring.





Required Attributes

Home /oracle

Owner oracle

SID ORCL#


Optional Attributes


User vcs

Pword vcs

DetailMonitor 1

Table vcstest

MonScript ./bin/Oracle/SqlTest.pl

Critical? No (0)

Enabled? Yes (1)


B


2 Modify up the nameOracle1 resource for detail monitoring. Use the values shown in the outlined portion of the worksheet.

Note: Use the encrypted string from the previous step when setting the Pword attribute.

3 Unfreeze the nameSG1 service group.




su - oraclesqlplus "/ as sysdba"

select TO_CHAR(tstamp,'MON DD, YYYY HH:MI:SS AM')tstamp from vcs.vcstest;



Lab 3 Details: Creating NFS Resources B–31Copyright © 2005 VERITAS Software Corporation. All rights reserved.

B

Lab 3 Details: Creating NFS Resources


Lab: Creating NFS ResourcesStudents work separately to modify their own nameSG1 service groups to add NFS and Share resources.

Brief instructions for this lab are located on the following page:• “Lab 3 Synopsis: Creating NFS Resources,” page A-16

Solutions for this exercise are located on the following page:• “Lab 3 Solutions: Creating NFS Resources,” page C-35


nameShare1

nameMount1

nameVol1

nameDG1

nameNFS1


nameSG1nameSG1





nameIP1





B



2 Verify that the major and minor numbers are the same for each system in the cluster.

3 If the major numbers do not match:

a Select major numbers that are not in use by any other device by viewing the /etc/name_to_major file on each system.

b To make the major numbers the same on both systems, use the haremajor command to change the major numbers for vxio and vxspec.

Note: You may need to change the major numbers on more than one system to make them unique for all systems.



Add an NFS resource to your nameSG1 service group.





5 Bring the resource online if it is not already online. If the NFS daemons were started by the operating system and running, the resource is marked as online.

Note: NFS is a persistent resource type; however, it is of type on-only.





Resource Type NFS

Required Attributes

Critical? No (0)

Enabled? Yes (1)


B

Add a Share resource to your nameSG1 service group.



3 Set the required attributes for this resource, and any optional attributes, if needed.







Resource Type Share

Required Attributes

PathName /name1

Critical? No (0)

Enabled? Yes (1)



1 Unlink nameProcess1|App1 from nameIP1, if it is not already unlinked.


3 Test the service group by switching it between the systems.

4 Verify NFS functionality from a client system using the trainer system in the classroom, or another system outside of your cluster.

a Log in to the client system.

b Create a mount point on the client system using your name as the mount point.

c Mount the NFS file system on the trainer (or other) system using the virtual IP address of your nameSG1 service group.

d Verify that the file system mounted.

5 In another window, perform a switch of the nameSG1 service group to the other system (their_sys) and verify that the NFS mounted file system remains mounted on the client machine.





nameIP1 nameShare1


nameShare1 nameNFS1


B

6 Switch the nameSG1 service group back to your system and verify that the NFS mounted file system remains mounted on the client machine.

7 Unmount the NFS file system on the client system.

8 Log off of the client system.

9 Set the NFS and Share resources to critical.


11 When all students using the cluster are finished configuring the NFS service group, close the cluster configuration.



In this lab section, each student works on their own to remove the NFS-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.

1 Clear all faulted resources in the service group.


3 Take the nameIP1 and nameShare1 resources offline.

4 Disable the nameShare1 and nameNFS1 resources.

5 Unlink the nameIP1 and nameShare1 resources.

6 Unlink the nameShare1 and nameNFS1 resources.

7 Unlink the nameShare1 and nameMount1 resources.

8 Delete the nameShare1 and nameNFS1 resources.





Appendix CLab Solutions

C–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Lab 1 Solutions: Creating an Application Resource C–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

C

Lab 1 Solutions: Creating an Application Resource


Lab 1 Solutions: Creating an Application ResourceStudents work separately to modify their own nameSG1 service groups.

Brief instructions for this lab are located on the following page:• “Lab 1 Synopsis: Creating an Application Resource,” page A-2

Step-by-step instructions for this lab are located on the following page:• “Lab 1 Details: Creating an Application Resource,” page B-3


Object Values



nameApp1

nameMount1

nameVol1

nameDG1

nameProxy1



nameSG1nameSG1





C


1 Verify that the nameSG1 service group is online and running on your system. If this service group is not online, bring it online on your system.

hastatus -sumhagrp -switch nameSG1 -to your_sys orhagrp -online nameSG1 -sys your_sys


haconf -makerw

3 Take the nameProcess1 resource offline.

hares -offline nameProcess1 -sys your_sys

4 Disable the nameProcess1 resource.

hares -modify nameProcess1 Enabled 0






Required Attributes

StartProgram /name1/startloopy name 1 /name1



Critical? No (0)

Enabled? Yes (1)


5 Unlink the nameProcess1 resource from nameIP1 and nameMount1.

hares -unlink nameProcess1 nameIP1hares -unlink nameProcess1 nameMount1


hares -delete nameProcess1


haconf -dump



Script location:

_________________________________________________________loopy_dir

Copy these scripts:



ls -l /name1



hares -add nameApp1 Application nameSG1


hares -modify nameApp1 Critical 0


C


hares -modify nameApp1 StartProgram "/name1/startloopy name 1 /name1"

hares -modify nameApp1 StopProgram "/name1/stoploopy /name1"

Note: Quotation marks are not required when you use the GUI to configure the StartProgram and StopProgram attributes.

hares -modify nameApp1 PidFiles /name1/loopypid


hares -modify nameApp1 Enabled 1

13 Bring the resource online on your system.

hares -online nameApp1 -sys your_sys

14 Verify that the resource is online in VCS and at the operating system level.

hares -display nameApp1

Check for loopy output displayed on the console.


haconf -dump


Each student configures the dependencies for their own service group.


hares -link nameApp1 nameMount1hares -link nameApp1 nameIP1

2 Verify that the resources are linked properly.

hares -dep nameApp1


haconf -dump





nameApp1 nameIP1

nameApp1 nameMount1


C

1 Test the service group by switching it to their_sys in the cluster.

hagrp -switch nameSG1 -to their_sys

2 Verify that the service group came online properly on the other system.

hastatus -sum

3 Test the service group by switching it back to your system.

hagrp -switch nameSG1 -to your_sys

4 Verify that the service group came online properly on the original system.

hastatus -sum


hares -modify nameApp1 Critical 1

6 Save the cluster configuration and view the configuration file to verify your changes.

haconf -dump view /etc/VRTSvcs/conf/config/main.cf


haconf -dump -makero


kill -9 loopy_pid





haconf -makerw


hares -unlink nameApp1 nameIP1


haconf -dump




Lab 2 Solutions: Creating Oracle Resources C–11Copyright © 2005 VERITAS Software Corporation. All rights reserved.

C

Lab 2 Solutions: Creating Oracle Resources


Lab 2 Solutions: Creating Oracle ResourcesStudents work separately in most sections of this lab to modify their own nameSG1 service groups to add Oracle and Netlsnr resources.

Brief instructions for this lab are located on the following page:• “Lab 2 Synopsis: Creating Oracle Resources,” page A-6

Step-by-step instructions for this lab are located on the following page:• “Lab 2 Details: Creating Oracle Resources,” page B-11


nameOracle1

nameMount1

nameVol1

nameDG1

nameProxy1

nameIP1

nameApp1*

nameSG1nameSG1





nameListener1






C













hastatus -sumhagrp -online nameSG1 -sys your_sysorhagrp -switch nameSG1 -to your_sys

2 Create a link for your SID to the /name1 file system on your system. You are assigned a SID based on your system, as shown in the design worksheet.


ln -s /name1 /orcl1


ln -s /name1 /orcl2


ln -s /name1 /orcl3


ln -s /name1 /orcl4



C

3 On the other system in the cluster, create a link for your SID to the /name1 file system as shown.

Note: Be sure to complete this step. If you do not create the link on both systems, you cannot switch the service group later.


Solaris, AIXrsh their_sys ln -s /name1 /orcl1

HP-UXremsh their_sys ln -s /name1 /orcl1

Linuxssh their_sys ln -s /name1 /orcl1














4 Obtain the location of the scripts provided for extracting Oracle data files from your instructor and record the directory.

Location of scripts:

___________________________________________________________script_dir

5 Change to the script directory.

cd script_dir

6 Extract the Oracle datafiles by running the script for your SID, supplied in the worksheet and recorded in the following space. The script runs tar to copy the Oracle data files to the shared storage volume you have imported.

Script name:_________________________________

ORCL1_files.sh (for train1 train5 train9)





C


a Record the IP address value of your nameIP1 resource.

IP address:_________________________________

hares -display nameIP1 -attribute Address

b Edit the /etc/hosts files on all systems in the cluster to add your Oracle SID as the host name (ORCL#) with this IP address. Coordinate edits with your lab partner.

vi /etc/hosts

192.168.xxx.xxx ORCL# 192.168.xxx.xxx ORCLpartner#


Solaris/AIX/Linux

ifconfig -aHP-UX

netstat -in


1 Use the following commands to start Oracle on your system.


2 After the Oracle instance is started, start the Listener. Your listener name is the same name as your SID. Use the following command (as user oracle):

lsnrctl start ORCL#

3 Stop the Listener using the following command (as user oracle):

lsnrctl stop ORCL#

4 Stop the Oracle instance using the following commands (as user oracle):


5 Switch your nameSG1 service group to their system.


6 Log in to their system and perform the same test for Oracle on their system.

rlogin their_sys

a Start Oracle.




C

b After the Oracle instance is started, start the Listener.

lsnrctl start ORCL#

c Stop the Listener.

lsnrctl stop ORCL#

d Stop the Oracle instance.


7 Switch your nameSG1 service group to your system.


8 Exit from their system.

exit


All lab participants install the Oracle enterprise agent on their own systems. If you have no lab partner, install the agent on each system in the cluster.



__________________________________________________________install_dir


If you do not have a lab partner, install the agents on each system.

cp install_dir/VRTSvcsor.gz /tmp

3 Change to the /tmp directory.

cd /tmp

4 Unzip the compressed package file.

gunzip VRTSvcsor.tar.gz

5 Extract the compressed file from the tar file.

tar -xvf VRTSvcsor.tar


Solarispkgadd -d VRTSvcsor

AIXinstallp -ac -d VRTSvcsor.rte.bff VRTSvcsor.rte

HP-UXswinstall -s pkgs VRTSvcsor

Linuxrpm -ihv VRTSvcsor-2.0-Linux.i386.rpm

7 If you do not have a lab partner, install the agents on each system.



C


1 Use the GUI to open the cluster configuration, import the Oracle resource types, and save the configuration.

Select File—>Open configuration.Select File—>Import types.Browse to the /etc/VRTSvcs/conf directory.Select OracleTypes.cf.Click Import.Select File—>Save configuration.

2 View the main.cf file to see the addition of the include directive for the OracleTypes.cf file.

view main.cf

3 Check the configuration directory for VCS for the presence of the OracleTypes.cf file.

ls /etc/VRTSvcs/conf/config



All students configure their own Oracle resources.

1 Add the nameOracle1 resource to the service group.

hares -add nameOracle1 Oracle nameSG1


hares -modify nameOracle1 Critical 0


hares -modify nameOracle1 Home /oraclehares -modify nameOracle1 Owner oraclehares -modify nameOracle1 Sid ORCLxhares -modify nameOracle1 Pfile /orcl#/admin/ORCL#/pfile/initORCL#.ora





Required Attributes

Home /oracle

Owner oracle

Sid ORCL#

Optional Attributes


Critical? No (0)

Enabled? Yes (1)


C


hares -modify nameOracle1 Enabled 1


hares -online nameOracle1 -sys your_sys hares -display nameOracle1


haconf -dump


Each student configures their own Netlsnr resources.



hares -add nameListener1 Netlsnr nameSG1


hares -modify nameListener1 Critical 0





Required Attributes

Home /oracle

Owner oracle


Listener ORCL#

Critical? No (0)

Enabled? Yes (1)


C


hares -modify nameListener1 Home /oraclehares -modify nameListener1 Owner oraclehares -modify nameListener1 TnsAdmin /oracle/network/adminhares -modify nameListener1 Listener ORCL#


hares -modify nameListener1 Enabled 1


hares -online nameListener1 -sys your_sys hares -display nameListener1


haconf -dump


Note: Skip this section unless your class is using a version of VCS earlier than 4.0.


hares -add nameListener1 Sqlnet nameSG1


hares -modify nameListener1 Critical 0





Required Attributes

Home /oracle

Owner oracle


Listener ORCLx

Critical? No (0)

Enabled? Yes (1)


C


hares -modify nameListener1 Home /oraclehares -modify nameListener1 Owner oraclehares -modify nameListener1 TnsAdmin /oracle/network/adminhares -modify nameListener1 Listener ORCLx


hares -modify nameListener1 Enabled 1


hares -online nameListener1 -sys your_sys hares -display nameListener1


haconf -dump


Each student configures their own dependencies.

1 Link the Oracle and Listener resources according to the design worksheet.

hares -link nameListener1 nameOracle1hares -link nameListener1 nameIP1hares -link nameOracle1 nameMount1


haconf -dump









C

Test the service group by switching it between the systems. Then, set the Oracle and Netlsnr resources to critical and test failover.

1 Switch the nameSG1 service group to their system and verify that it comes online.


2 Switch the nameSG1 service group to your system and verify that it comes online.

hagrp -switch nameSG1 -to your_system

3 Set the nameOracle1 and nameListener1 resources to critical.

hares -modify nameOracle1 Critical 1hares -modify nameListener1 Critical 1


haconf -dump

5 When all students working in the cluster are finished configuring their Oracle service groups, close the cluster configuration.





In this lab section each student works on their own to remove the Oracle-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.



haconf -makerw

2 Take the nameOracle1 and nameListener1 resources offline.

hares -offline nameListener1 -sys your_syshares -offline nameOracle1 -sys your_sys

3 Disable the nameOracle1 and nameListener1 resources.

hares -modify nameOracle1 Enabled 0hares -modify nameListener1 Enabled 0

4 Unlink the nameOracle1 and nameListener1 resources.

hares -unlink nameOracle1 nameMount1hares -unlink nameListener1 nameOracle1hares -unlink nameListener1 nameIP1

5 Remove the nameOracle1 and nameListener1 resources.

hares -delete nameOracle1hares -delete nameListener1


haconf -dump





C


ps -ef | grep smonhagrp -display nameSG1

2 Log in as the oracle user on that system.

su - oracleSelect the instance for your system, such as ORCL1.

3 Start sqlplus as internal.

sqlplus "/as sysdba"

4 Create an Oracle user named vcs with the password vcs. Type the following commands from sqlplus:

create user vcs

identified by vcs

default tablespace users

quota 500K on users;

5 Grant permissions to vcs while in sqlplus:

grant create session to vcs;

6 Create a table named vcs.vcstest and insert a row. This table is used to test Oracle from VCS:

create table vcs.vcstest (tstamp date);

insert into vcs.vcstest (tstamp) values (SYSDATE);



7 Select the data in the vcs.vcstest table to verify that a row is present:

select TO_CHAR(tstamp, 'MON DD, YYYY HH:MI:SS AM')tstamp from vcs.vcstest;



haconf -makerw


hagrp -freeze nameSG1


C

Use the values in the table to configure the nameOracle1 resource to perform detailed monitoring.





Required Attributes

Home /oracle

Owner oracle

SID ORCL#

Optional Attributes


User vcs

Pword vcs

DetailMonitor 1

Table vcstest

Monscript ./bin/Oracle/SqlTest.pl

Critical? No (0)

Enabled? Yes (1)



/opt/VRTSvcs/bin/vcsencrypt -agent vcs

2 Modify the nameOracle1 resource for detail monitoring. Use the values shown in the outlined portion of the worksheet.

Note: Use the encrypted string from the previous step when setting the Pword attribute.

hares -modify nameOracle1 Pword drjPgrGhares -modify nameOracle1 User vcshares -modify nameOracle1 DetailMonitor 1hares -modify nameOracle1 Table vcstesthares -modify nameOracle1 MonScript \

./bin/Oracle/SqlTest.pl

3 Unfreeze the nameSG1 service group.

hagrp -unfreeze nameSG1


haconf -dump




su - oraclesqlplus "/ as sysdba"

select TO_CHAR(tstamp,'MON DD, YYYY HH:MI:SS AM')tstamp from vcs.vcstest;


Lab 3 Solutions: Creating NFS Resources C–35Copyright © 2005 VERITAS Software Corporation. All rights reserved.

C

Lab 3 Solutions: Creating NFS Resources


Lab Solution: Creating NFS ResourcesStudents work separately to modify their own nameSG1 service groups to add NFS and Share resources.

Brief instructions for this lab are located on the following page:• “Lab 3 Synopsis: Creating NFS Resources,” page A-16

Step-by-step instructions for this lab are located on the following page:• “Lab 3 Details: Creating NFS Resources,” page B-31


nameShare1

nameMount1

nameVol1

nameDG1

nameNFS1


nameSG1nameSG1





nameIP1





C



hares -display nameMount1

2 Verify that the major and minor numbers are the same for each system in the cluster.

grep vx /etc/name_to_majorrsh their_sys grep vx /etc/name_to_major

3 If the major numbers do not match:

a Select major numbers that are not in use by any other device by viewing the /etc/name_to_major file on each system.

view /etc/name_to_majorrsh their_sys view /etc/name_to_major

b To make the major numbers the same on both systems, use the haremajor command to change the major numbers for vxio and vxspec.

haremajor -vx 90 91

Note: You may need to change the major numbers on more than one system to make them unique for all systems.



Add an NFS resource to your nameSG1 service group.


haconf -makerw


hares -add nameNFS1 NFS nameSG1


hares -modify nameNFS1 Critical 0


hares -modify nameNFS1 Enabled 1

5 Bring the resource online if it is not already online. If the NFS daemons were started by the operating system and running, the resource is marked as online.

Note: NFS is a persistent resource type; however, it is of type on-only.

hares -display nameNFS1hares -online nameNFS1 -sys your_syshares -display nameNFS1


haconf -dump




Resource Type NFS

Required Attributes

Critical? No (0)

Enabled? Yes (1)


C

Add a Share resource to your nameSG1 service group.


hares -add nameShare1 Share nameSG1


hares -modify nameShare1 Critical 0

3 Set the required attributes for this resource, and any optional attributes, if needed.

hares -modify nameShare1 PathName /name1


hares -modify nameShare1 Enabled 1


hares -online nameShare1 -sys your_system hares -display nameShare1


haconf -dump




Resource Type Share

Required Attributes

PathName /name1

Critical? No (0)

Enabled? Yes (1)



1 Unlink nameProcess1|App1 from nameIP1, if it is not already unlinked.

hares -unlink nameProcess1 nameIP1


hares -link nameIP1 nameShare1hares -link nameShare1 nameMount1hares -link nameShare1 nameNFS1

3 Test the service group by switching it between the systems.

hagrp -switch nameSG1 -to their_syshagrp -switch nameSG1 -to your_system

4 Verify NFS functionality from a client system using the trainer system in the classroom, or another system outside of your cluster.

a Log in to the client system.

rlogin trainer

b Create a mount point on the client system using your name as the mount point.

mkdir /name





nameIP1 nameShare1


nameShare1 nameNFS1


C

c Mount the NFS file system on the trainer machine using the virtual IP address of your nameSG1 service group.

mount 192.168.xxx.xxx:/name1 /name

d Verify that the file system mounted.

mount

5 In another window, perform a switch of the nameSG1 service group to the other system (their_sys) and verify that the NFS mounted file system remains mounted on the client machine.


6 Switch the nameSG1 service group back to your system and verify that the NFS mounted file system remains mounted on the client machine.


7 Unmount the NFS file system on the client system.

umount /name

8 Log off of the client system.

9 Set the NFS and Share resources to critical.

hares -modify nameNFS1 Critical 1hares -modify nameShare1 Critical 1


haconf -dump





In this lab section, each student works on their own to remove the NFS-related resources. This provides a simplified service group for any subsequent labs your instructor may ask you to complete.

1 Clear all faulted resources in the service group.

hagrp -clear nameSG1


haconf -makerw

3 Take the nameIP1 and nameShare1 resources offline.

hares -offline nameIP1 -sys your_syshares -offline nameShare1 -sys your_sys

4 Disable the nameShare1 and nameNFS1 resources.

hares -modify nameShare1 Enabled 0hares -modify nameNFS1 Enabled 0

5 Unlink the nameIP1 and nameShare1 resources.

hares -unlink nameIP1 nameShare1

6 Unlink the nameShare1 and nameNFS1 resources.

hares -unlink nameShare1 nameNFS1

7 Unlink the nameShare1 and nameMount1 resources.

hares -unlink nameShare1 nameMount1

8 Delete the nameShare1 and nameNFS1 resources.

hares -delete nameShare1hares -delete nameNFS1



C


hares -link nameProcess1 nameIP1


haconf -dump




Appendix DDesign Worksheet: Template

D–2 VERITAS Cluster Server for UNIX, Example Application ConfigurationsCopyright © 2005 VERITAS Software Corporation. All rights reserved.

Cluster Interconnect Configuration

First system:

/etc/VRTSvcs/comms/llttab Sample Value Your Value

set-node(host name)

set-cluster(number in host name of odd system)

link

link

/etc/VRTSvcs/comms/llthosts Sample Value Your Value

/etc/VRTSvcs/comms/sysname Sample Value Your Value

Appendix D Design Worksheet: Template D–3Copyright © 2005 VERITAS Software Corporation. All rights reserved.

D

Second system:

/etc/VRTSvcs/comms/llttab Sample Value Your Value

set-node

set-cluster

link

link

/etc/VRTSvcs/comms/llthosts Sample Value Your Value

/etc/VRTSvcs/comms/sysname Sample Value Your Value


Cluster Configuration (main.cf)

Types Definition Sample Value Your Value

Include types.cf

Cluster Definition Sample Value Your Value

Cluster

Required Attributes

UserNames

ClusterAddress

Administrators

Optional Attributes

CounterInterval

System Definition Sample Value Your Value

System

System


D

Service Group Definition Sample Value Your Value

Group

Required Attributes

FailoverPolicy

SystemList

Optional Attributes

AutoStartList

OnlineRetryLimit



Service Group

Resource Name

Resource Type

Required Attributes

Optional Attributes

Critical?

Enabled?


D


Service Group

Resource Name

Resource Type

Required Attributes

Optional Attributes

Critical?

Enabled?



Service Group

Resource Name

Resource Type

Required Attributes

Optional Attributes

Critical?

Enabled?


D


Service Group

Resource Name

Resource Type

Required Attributes

Optional Attributes

Critical?

Enabled?



Service Group


Technology

havcs-410-101 a-2-10-srt-pg_3