PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287

7/25/2019 PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287

1/64

Copyright IBM Corporation 2015. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

PowerHA SystemMirror CommonAdministrator Tasks

Michael HerreraPower Systems Strategic Initiatives Team

[email protected] Coppell, TX


2/64

2IBM Systems Technical University, October 5-9 | Hilton Orlando

Copyright IBM Corporation 2015. Technical University/Symposia

materials may not be reproduced in whole or in part without the prior

written permission of IBM.

Agenda

Deployment Methods

Available methods

Tips & Options

Recovery (snapshots | alt disk install | replication)

Management

Version in use

Start / Stop / Move Operations

Maintenance Tasks

SW Updates

Available Migration Methods

Recommendations

Configuration Optimization

Tunables

Custom Verification Methods

Custom Events

Pager Notification Events

Application Monitoring

CLI Usage

V7 Changes

Monitoring

Troubleshooting

Methods to collect cluster

information

Useful Cluster Logs


3/64





Deployment Methods

PowerHA SystemMirror for AIX


4/64





V7 Deployment Methods

There are a number of different ways to achieve the same result:

smitty sysmirror

Initial | Discovery

Custom

IBM Systems Director Plug-In

Available GUI Interface

clmgr cluster copy !cluster cloning from snapshot

Command Line Interface (clmgr)1.

clmgr add cluster repository= nodes=,

2. clmgr add service_ip network=

3.

clmgr add application_controller startscript= stopscript=

4.

clmgr add nodes=, startup=ohn fallback=nfb service_label=

volume_group=application=5. clmgr sync cluster


5/64





PowerHA SystemMirror: Cluster Snapshots

/usr/es/sbin/cluster/snapshots/ .info

.odm

Cluster Configuration

HACMPcluster

...info

HACMPnodeinfo

HACMPadapter

info.

Cluster Report & CLI output

cllsnode

..

cllscf..

cllsif

..

Snapshot C .odm

clusterODM stanzas

Snapshot files:

Snapshot C .info

cluster reportSnapshot B .odm

clusterODM stanzas

Snapshot A .odm

clusterODM stanzas

Snapshot B .info

cluster reportSnapshot A .info

cluster report

" Snapshots are saved off

automatically any time a

Verify / Sync operation is

invoked

" The .info file is not

necessary in order toable to restore the

configuration

" The snapshot menu willask for a and a

as the only

required fields

" The snapshot upgrademigration path requires

the entire cluster to be

down


6/64





Cluster Copy Function with clmgr

V713 provides new fast-start mechanism called cluster copying

Good for environments with many, similar clusters

Traditional snapshot capture & restore:

# clmgr add snapshot

# clmgr manage snapshot restore \

[ configure = yes | no ] \[ force = no | yes ]

Copy function on new cluster:

# clmgr manage snapshot restore \

nodes=, \repositories=, [] [: [,]] \

[ cluster_name = ] \

[ configure = yes | no ]

[ force = no | yes

Snapshot must bemanually copied onto

new nodes

Service Labels are notpreserved

Will perform a new

discovery but will not

automatically

synchronize the cluster


7/64





Planning for Environment Recoverability

Restore from mksysb | NIM

Cluster snapshots

Rootvg Replication

Cloning a cluster

25GB

25GB

25GB

rootvg

rootvg

Alt disk copy

25GB

Storage

Replication

25GB

StorageReplication

LPAR

LPAR

VSCSI or NPIV


8/64





Removing a cluster software

Removal Options:

Delete CAA cluster

# rmcluster fn

Delete cluster from

one or all nodes

Other Considerations:

/etc/cluster/rhosts

/etc/hosts

ECM VG conversion

Application Monitoring

Custom Pre / Post Events

Steps & Considerations:

smitty remove !cluster.*

Hard Set IPs

Auto-varyon of volume groups

Auto-mount of file systems

Automated Application start

The goal is not have to remove the software, but if you have to


9/64





Removing a partition

that has a CAA cluster configured


10/64





Management



11/64





What version of code do you have installed?

# lslpp l cluster.es.server.rte

cluster.es.server.rte 7.1.3.3 COMMITTED Base Server Runtime

# halevel s

7.1.3 SP3

# lssrc ls clstrmgrES | grep fix

cluster fix level is 3

# clmgr query version

SystemMirror Information:=========================

Version: 7.1.3 SP3Build Level: 1509B_hacmp713 (3/6/15, 10:59:11)

Cluster Type: Multi Site Cluster Deployment (Linked Cluster)

CAA Information:================

May 16 201417:01:161419D_71N

@(#) _kdb_buildinfo unix_64 May 16 2014 17:01:16 1419D_71NCluster Configured: Yes.

Host Information:

=================HOSTNAME: mhoracle1.dfw.ibm.com

IPADDRESS: 9.19.51.211

LOCALHOST: true

HAVERSION: 7.1.3.3VERSION_NUMBER: 15HAEDITION: ENTERPRISE

AIX_LEVEL: 7100-03-03-1415


12/64





So how do you start up Cluster Services ?

smitty sysmirror !System Management !PowerHA SystemMirror Services !Start / Stop

smitty clstart (FastPath)

clmgr start cluster

-clmgr online node nodeA

- clmgr start node nodeA



13/64





PowerHA SM: Cluster Start up Behavior

The cluster manager daemon is now running all of the time

# clshowsrv -v

Status of the RSCT subsystems used by HACMP:

Subsystem Group PID Status

cthags cthags 4980948 active

ctrmc rsct 4063376 active

Status of the HACMP subsystems:


clstrmgrES cluster 4915234 active

clcomd caa 6422738 active

Status of the optional HACMP subsystems:


clinfoES cluster 8847544 active

Settings can be altered within the cluster panels:

# lssrc -ls clstrmgrES | grep stateCurrent state: ST_STABLE

Verify Cluster should be left to true


14/64





Automatic Cluster Services Startup Behavior

What is the Best Practice ?

If entry exists thecluster Services are

set to automaticallystart up on boot up

All currently supported

releases perform acluster verification on

start up and will validate

whether the node canenter the cluster


15/64





PowerHA SystemMirror: Cluster Stop Options

What is the purpose of each option ?

For non-disruptive updatesstop services on only one

node at a time to allow forone node to retain the status

of the cluster resources

" You cannotNon-Disruptively upgrade from pre-version 7.X to newer releases

" Avoid Unmanaging cluster from all nodes at the same time


16/64





Moving Resources between Nodes

clRGmove g n -m

clmgr move rg node=


smitty cl_admin

If multiple RGs are selectedthe operation and

resources will be processedsequentially


17/64





Moving Resource Groups with Dependencies

DependantWorkload

Production

Workload

Resource Groups:

Parent / ChildDependency

ODM Stanza:

PowerHA Support:

Parent / Child Dependencies

Location Dependencies (Priority)

Start After | Stop After

CSPOC ll


18/64





CSPOC allows

for a multitude of DARE operations

The Cluster Single Point of Control options facilitate dynamic operations

Follow these panels to

dynamically add or remove

resources from the cluster or

perform resource group

movements between nodes

There are CSPOC specific

logs in the HA cluster that will

provide details in the event of

a problem

# smitty cl_admin


19/64


20/64





Cluster Report Functions

Prior to version 7 an HTML cluster report was available

Required Online Planning Worksheets (discontinued in V7)

In V7 feature was added in IBM Systems Director plug-in

V713 introduces a native HTML cluster report via clmgr # clmgr view report cluster file= type=html

Only officiallysupported on

IE & Firefox


21/64





Sample HTML Report


22/64





Maintenance



23/64





Updating your Multipath Driver with CAA cluster

Stopping cluster services does not close the CAA private volume group

root@mhoracle1 /> lspv | grep private

hdisk9 00f626d13aa3645a caavg_private active

root@mhoracle1 /> lsvg -l caavg_private

caavg_private:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

caalv_private1 boot 1 1 1 closed/syncd N/A

caalv_private2 boot 1 1 1 closed/syncd N/A

caalv_private3 boot 4 4 1 open/syncd N/A

powerha_crlv boot 1 1 1 closed/syncd N/A

" New option in Version 713 SP1 to stop CAA with it:

root@mhoracle1 /> clmgr stop cluster STOP_CAA=yes

root@mhoracle1 /> clmgr start cluster START_CAA=yes

Use of CAA option typically not required


24/64





Upgrade Considerations

Operating System: Should you do AIX first or HA code?

Should you combine the upgrade

New OS requirements for HA

What is your back-out plan? Alternate disk install

Mksysb

BOS updates will typically require areboot (hence a disruption)

Cluster Software Code:

What type of Migration

Snapshot Migration

Rolling Migration

Non-Disruptive Update

Remove & Reconfigure

Evaluate source to target level

Can you user NDU?

Any new critical requirements?

Any configuration changes required?

There are two main areas that you need to consider OS & HA Software

Change Controls: what is your ability to apply and test the updates ?

Consider things like Interim Fixes locking down the system

Will they need to be reapplied?

Will they need to be rebuilt?


25/64





Flow of Non-Disruptive PowerHA Software Upgrade (NDU)

Active Production Environment

-

HA Version 7.1.3

-

UNMANAGE resources

-

Application is still running

-

smit update_all

-

HA Level & Patches

-

Be mindful of new base filesets

-

smit clstart

-

Start scripts will get reinvoked

Node Running at New 7.2.0 version

-

Application still active

Hypothetical Example 2 Node Cluster HA version 7.1.3 to 7.2.0

Starting Point Standby System

- HA Version 7.1.3

-

UNMANAGE resources

-

smit update_all

-

smit clstart

Node Running Version 7.2.0

We advise against stopping thecluster with the UNMANAGE

option on more than one node ata time. Note that it can be done

but there are various factors toconsider

Common Question: How long can the cluster run in a mixed mode ? What operations are supported ?

Important Note: This process flowassumes that the minimum AIX levels for

target release are already in place


26/64





Migration Scenario to PowerHA SM Version 7.X

Active Production Environment

-

PowerHA SM Version 6.1(OS & CAA pre-reqs must be there to proceed)

- Populate /etc/cluster/rhosts

- Refresh clcomd

Stop Cluster Services with Takeover

Run clmigcheck utility

Install PowerHA V7.X + fixes (2 steps)

Reboot Server

Start Cluster Services (cluster conversion is run)26

Starting Point Standby System

-

PowerHA SM Version 6.1(OS & CAA pre-reqs must be there to proceed)

Perform pre-migration check (clmigcheck)

- option 1

- checks for unsupported elements

- if valid proceed to option 3

- option 3

- specify repository disk

- define multicast address

- Populate /etc/cluster/rhosts

- Refresh clcomd

Stop Cluster Services

Install PowerHA V7.X + fixes (2 steps)Reboot Server

Reintegrate Node into cluster (migration not complete)

Issue rg_move or contine to run on 2ndNode

Resources & applications are interrupted thenresumed after during this fallover operation

At this stage of the migration the cluster willbe temporarily running in a mixed mode


27/64





Migration from V6 to V7 Recommendations

IBM Knowledge Center upgrade recommendations

http://www-01.ibm.com/support/knowledgecenter/SSPHQG_7.1.0/com.ibm.powerha.insgd/ha_install_mig61_plan.htm

V7.1.3 Recommend:

AIX 6.1 TL9 SP4

AIX 7.1 TL3 SP4

APAR Service Pack Level

IV58849 Included in SP4



IV65472 Interim Fix available for download

IV66606 Interim Fix available for download

Ensure thatthese APARs

are installed

Configurations that will not be automatically converted:o

IPAT with HWATo

FDDI, ATM, X.25, token ringo

LVM Split-Site configurations with disks assigned to each site

New clmigcheckutility available for download (additional checks & enhancements)


28/64





Additional Tools & References

PowerHA SystemMirror Internal Wiki

https://w3-connections.ibm.com/wikis/home?lang=en-us#!/wiki/hacmp/page/PowerHA%20for%20AIX

Detailed checklist puttogether by HA

development resources


29/64





Configuration Optimization


Node initiated IP traffic


30/64





Node initiated IP traffic

Default behavior with first alias is to set service IP first in routing table

Firewall

9.20.20.7 (service IP)

En2 9.20.20.10 (base) En2 9.20.20.20

9.20.20.10 (base)

En2 9.20.20.7 (service IP) En2 9.20.20.20

* Example using first alias behavior

Disabling First Alias behavior (V713)


31/64





Disabling First Alias behavior (V713)

This option willprompt you for the

network

The default is for the service IP to beplaced before the base address.

This is ideal for environments with afirewall between the boxes that wantto see node initiated traffic go out of

the service IP and not the baseaddress

# smitty sysmirror !Cluster Applications & Resources !Resources

A t ti C ti V if & S


32/64





Automatic Corrections on Verify & Sync

There are Verify & Sync options in thefirst two paths, however, note that they

do not include the Auto-Corrective option.

You need to follow the Custom Cluster

Configuration Path for that.

The custom path will allow to make

corrective actions only if ALL cluster nodesare not running cluster services. By default

it will not perform any corrective actions.

Cluster Custom Verification Methods


33/64





Cluster Custom Verification Methods

Cluster Verification is made up of a bunch of data collectors

Checks will return PASSED or FAILED

Will often provide more details than what is reported in the smit.log output

CustomVerification Methods may be defined to run during the Verify / Syncoperations

Note: Automatic verify & sync on node start up does not include any custom verification methods

Adding Custom Verification Methods


34/64





Adding Custom Verification Methods

Problem Determination Tools > PowerHA SystemMirror Verification > Configure Custom Verification Method

> Add a Custom Verification Method and press Enter

Output in smit.logand clverify.logfiles:

Currently Loaded Interim Fixes:

NODE mutiny.dfw.ibm.com

PACKAGE INSTALLER LABEL

======================================================== =========== ==========bos.rte.security installp passwdLock

NODE munited.dfw.ibm.com

PACKAGE INSTALLER LABEL

======================================================== =========== ==========

bos.rte.security installp passwdLock

Please Ensure that they are consistent between the nodes!



35/64






Custom methods should be in a common path between the cluster members

-

ie. /usr/local/hascripts/custom_ver_check.sh

The Methods are stored in the cluster ODM stanzas

Script Logic & Return Codes

How fancy do you want to get#!/bin/kshecho "Currently Loaded Interim Fixes:"

clcmd emgr -Pecho "Please Ensure that they are consistent between the nodes!"

PowerHA SystemMirror Tuning Differences


36/64





PowerHA SystemMirror Tuning Differences

Version 7 clusters & CAA changed DMS behavior

" CAA DMS tunable (deadman_mode) allows two different actions

Assert (crash) the system (default behavior)

Generate AHAFS event

..............

Client Failure Scenario

Repository Disk LUN had beenlocked and had not been

responsive for days. Client wasunaware and standby node had

a problem. Primary system wasbrought down when it was

unable to write to repository disk

Tuning the Failure Detection Rate in V7 X


37/64





Tuning the Failure Detection Rate in V7.X

Note that the SMIT menu to alter values was missing prior to HA 7.1.1 SP1

" Checking current settings:

root@mhoracle1 /> clctrl -tune -o node_down_delay

sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_down_delay = 10000

root@mhoracle1 /> clctrl -tune -o node_timeout

sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_timeout = 20000

" Modifying via command line:

clmgr modify cluster HEARTBEAT_FREQUENCY= 10000 GRACE_PERIOD=5000

*** The settings will take effect only after the next sync

Attributesstored in

HACMPclusterobject class

Defining Pre / Post Events


38/64





Defining Pre / Post Events

Pre/Post-Event Commands are NOT the same thing as User Defined Events

A custom Event will neverget invoked unless you

explicitly define it as a Preor Post event command to

an existing Cluster Event

PowerHA SystemMirror: File Collections


39/64





PowerHA SystemMirror: File Collections

Configuration_Files

/etc/hosts

/etc/services /etc/snmpd.conf

/etc/snmpdv3.conf

/etc/rc.net

/etc/inetd.conf

/usr/es/sbin/cluster/netmon.cf

/usr/es/sbin/cluster/etc/clhosts

/usr/es/sbin/cluster/etc/rhosts

/usr/es/sbin/cluster/etc/clinfo.rc

" Introduced in HA 5.2

Ability to automatically push files every 10 min from source node specified Default collections created but not enabled by default

" SystemMirror_Files

Pre, Post & Notification

Start & Stop scripts Scripts specified in monitors

Custom pager text messages

SNA scripts

Scripts for tape support

Custom snapshot methods

User defined events

" Not intended to maintain users & passwords between cluster nodes

File Collections Application script Scenario


40/64





File Collections Application script Scenario

If set to yes fileswill be

propagated every10 minutes

# smitty sysmirror !System Management !File Collections

#!/bin/ksh

Application Start Logic

RED Updates#!/bin/ksh

Application Stop Logic

RED Updates

/usr/local/hascripts/app*

#!/bin/ksh

Application Start Logic

BLUE Logic#!/bin/ksh

Application Stop Logic

Blue Logic

/usr/local/hascripts/app*

Node A Node B

PowerHA SM: User & Group Administration


41/64





p

Can select

Local (files)

LDAP

Select Nodes byResource Group

No selection

means all nodes

# smitty sysmirror !System Management !Security and Users

" Users will be

propagated to all of

the cluster nodes

applicable

" Password

command can be

altered to ensure

consistency across

al nodes

PowerHA SM: User Passwords (clpasswd)


42/64





( p )

Optional List ofUsers whosepasswords willbe propagatedto all clusternodes

passwdcommand isaliased toclpasswd

Functionalityavailable sinceHACMP 5.2

(Fall 2004)

# smitty sysmirror !System Management !Security and Users !Passwords in a PowerHA SystemMirror cluster

Pager Notification Events


43/64





g

As long as sendmail is enabled you can easily receive EVENT notification

smitty sysmirror !Custom Cluster Configuration !Events !Cluster Events !RemoteNotification Methods !Add a Custom Remote Notification Method

Sample Email:

From: root 10/23/2012 Subject: HACMPNode mhoracle1: Event acquire_takeover_addr occurred at Tue Oct 23 16:29:36 2012, object =

Pager Notification Methods


44/64





Action Taken: Halt of Node connor

HACMPpager:

methodname = "Herrera_notify"desc = Lab Systems Pager Event"

nodename = "connor kaitlyn"

dialnum = "[email protected]"

filename = "/usr/es/sbin/cluster/samples/pager/sample.txt"

eventname = "acquire_takeover_addr config_too_long event_errornode_down_complete node_up_complete"

retrycnt = 3

timeout = 45

# cat /usr/es/sbin/cluster/samples/pager/sample.txt

Node %n: Event %e occurred at %d, object = %o

Sample Email:

From: root 09/01/2009 Subject: HACMPNode kaitlyn: Event acquire_takeover_addr occurred at Tue Sep 1 16:29:36 2009, object =

* Sendmail must be working and accessible via the firewall to receive notifications

Application Monitoring within Cluster


45/64





Some are provided in Smart Assistants

ie. cluster.es.assist.oracle !/usr/es/sbin/cluster/sa/oracle/sbin/DBInstanceMonitor

A Monitor is bound to the Application Controller

Example OracleDB

StartupMonitor

ProcessMonitor

CustomMonitor

Confirm thestartup of the

application

Invokes thecustom logic

Checks theprocess table

NewApplication

Startup Mode

in HA 7.1.1

60 secinterval

60 secinterval

Onlyinvoked onapplication

startup

Long RunningMonitors will continue

run locally with the

running application

Application Monitoringwithin the cluster

configuration is optional

Monitoring can beconfigured to perform

restarts | notify | fallover

If the source LPAR

remains ONLINE andonly the application goes

offline - withoutmonitoring the cluster will

not attempt to relocatethe workload/s


46/64





V7 Command Line Interface


Monitoring the Cluster Status


47/64





clcmd lssrc ls clstrmgrES | grep state

Current state: ST_STABLE

Current state: ST_STABLE

clmgr v a name,state,raw_state query node

NAME="mhoracle1"

STATE="OFFLINE"

RAW_STATE="ST_INIT"

NAME="mhoracle2"

STATE="OFFLINE"

RAW_STATE="ST_INIT

clRGinfo p

Cluster Name: sapdemo71_cluster

Resource Group Name: SAP_rg

Node Group State

---------------------------- ---------------

mhoracle1 ONLINE

mhoracle2 OFFLINE

Provided by the CAA packages

Part of new CLI interface in V7 clusters

Legacy cluster command (location of resources)

PowerHA V7.1: lscluster command


48/64





Flags

-i Lists the cluster configuration interfaces on the local node

-n Allows the cluster name to be queried for all interfaces-s Lists the cluster network statistics on the local node

-m Lists the cluster node configuration information

-d Lists the cluster storage interfaces

-c Lists the cluster configuration

root@mhoracle2 /> lscluster -cCluster Name: MH_Oracle_SVC_EE_cluster

Cluster UUID: 3662ad2a-8d22-11e3-8d70-46a6ba546402Number of nodes in cluster = 3

Cluster ID for node mhoracle1.dfw.ibm.com: 2

Primary IP address for node mhoracle1.dfw.ibm.com: 9.19.51.211Cluster ID for node mhoracle2.dfw.ibm.com: 3

Primary IP address for node mhoracle2.dfw.ibm.com: 9.19.51.212Cluster ID for node mhoracle3.dfw.ibm.com: 4

Primary IP address for node mhoracle3.dfw.ibm.com: 9.19.51.198Number of disks in cluster = 2

Disk = UUID = 497a007c-a3e9-fcc4-d362-2db9002bdfbb cluster_major = 0 cluster_minor = 2

Disk = hdisk9 UUID = d3ce4fd5-3003-ac21-9789-6d9a590242fd cluster_major = 0 cluster_minor = 1Multicastfor site Site2: IPv4 228.19.51.212IPv6 ff05::e413:33d4

Multicastfor site Site1: IPv4 228.19.51.211IPv6 ff05::e413:33d3

Communication Mode: unicastLocal node maximum capabilities: HNAME_CHG, UNICAST, IPV6, SITE

Effective cluster-wide capabilities: HNAME_CHG, UNICAST, IPV6, SITE

Example of 3 node EE clusterusing Sites & SVC replication

Validating SANCOMM Heartbeat Link


49/64





lscluster i

Interface number 2, sfwcomIFNET type = 0 (none)

NDD type = 304 (NDD_SANCOMM)

Smoothed RTT across interface = 7

Mean deviation in network RTT across interface = 3

Probe interval for interface = 100 ms

Interface state = UP

lscluster mInterface State Protocol Status

-----------------------------------------------

dpcom DOWN none RESTRICTED

en0 UP IPv4 none

sfwcom UP none none

lscluster -sstorage pkts sent: 168493709 storage pkts recv: 82575360

New in version 7.1.3

In version 7 clusters improperly

configured storage framework

communication interfaces could

be perceived as being

operational

clras sancomm_status

NAME UUID STATUS

nodeA.dfw.ibm.com | e9b4d6a4-5e71-11-e2-af42-00145ee726e1 | UP |

Monitoring interface traffic


50/64





# lscluster i

Node mhoracle1.dfw.ibm.com

Node UUID = 1dfc2d5a-6ab0-11e2-ace9-46a6ba546402

Number of interfaces discovered = 3Interface number 1, en0IFNET type = 6 (IFT_ETHER)NDD type = 7 (NDD_ISO88023)MAC address length = 6MAC address = 46:A6:BA:54:64:02Smoothed RTT across interface = 7Mean deviation in network RTT across interface = 3Probe interval for interface = 100 msIFNET flags for interface = 0x1E080863NDD flags for interface = 0x0021081BInterface state = UPNumber of regular addresses configured on interface = 2

IPv4 ADDRESS: 9.19.51.211 broadcast 9.19.51.255 netmask 255.255.255.0IPv4 ADDRESS: 9.19.51.239 broadcast 9.19.51.255 netmask 255.255.255.0Number of cluster multicast addresses configured on interface = 1IPv4 MULTICAST ADDRESS: 228.19.51.211

Interface number 2, sfwcomIFNET type = 0 (none)NDD type = 304 (NDD_SANCOMM)Smoothed RTT across interface = 7

Mean deviation in network RTT across interface = 3Probe interval for interface = 100 msIFNET flags for interface = 0x00000000

NDD flags for interface = 0x00000009

Interface state = UPInterface number 3, dpcom

IFNET type = 0 (none)

NDD type = 305 (NDD_PINGCOMM)Smoothed RTT across interface = 750Mean deviation in network RTT across interface = 1500Probe interval for interface = 22500 msIFNET flags for interface = 0x00000000NDD flags for interface = 0x00000009Interface state = UP RESTRICTED AIX_CONTROLLED

# lscluster s

Cluster Network Statistics:

pkts seen: 7441874 passed: 1698499IP pkts: 6026783 UDP pkts: 5768975gossip pkts sent: 1154441 gossip pkts recv: 2304475cluster address pkts: 0 CP pkts: 5743449bad transmits: 0 bad posts: 33

Bad transmit (overflow): 0Bad transmit (host unreachable): 0Bad transmit (net unreachable): 0Bad transmit (network down): 0Bad transmit (no connection): 0short pkts: 0 multicast pkts: 5449950

cluster wide errors: 0 bad pkts: 0dup pkts: 185857 pkt fragments: 5112fragments queued: 0 fragments freed: 0pkts pulled: 0 no memory: 0rxmit requests recv: 292 requests found: 250requests missed: 66 ooo pkts: 40

requests reset sent: 66 reset recv: 48remote tcpsock send: 0 tcpsock recv: 0rxmit requests sent: 332alive pkts sent: 0 alive pkts recv: 0

ahafs pkts sent: 14 ahafs pkts recv: 4nodedown pkts sent: 0 nodedown pkts recv: 0

socket pkts sent: 11673 socket pkts recv: 11711cwide pkts sent: 462885 cwide pkts recv: 463555socket pkts no space: 0 pkts recv notforhere: 0

Pseudo socket pkts sent: 0 Pseudo socket pkts recv: 0Pseudo socket pkts dropped: 0arp pkts sent: 3 arp pkts recv: 1stale pkts recv: 0 other cluster pkts: 2

storage pkts sent: 2814171 storage pkts recv: 2722069disk pkts sent: 3115 disk pkts recv: 3267unicast pkts sent: 203803 unicast pkts recv: 318887out-of-range pkts recv: 0IPv6 pkts sent: 0 IPv6 pkts recv: 0

IPv6 frags sent: 0 IPv6 frags recv: 0Unhandled large pkts: 0

Updates to clras command


51/64





Options to monitor the health of the heartbeat networks

Cluster Repository disk heartbeating:

SANCOMM heartbeating:

Notice that this

output doesntshow both nodes

Dump Repository Disk Information:

# clras dumprepos

Dump Repository Disk information


52/64





The cluster configuration is stored in ODM stanzas and in the repository disk!

# clras dumprepos

New Command Line Interface clmgr


53/64





Several useful references:

Rapid Deployment Guide

IBM Systems Magazine article

IBM Redbooks

Documentation

MAN page

An much more robust extension of the original clvt command

The command is the back-end to the IBM Systems Director plug-in

Uses:

CLI Cluster Deployment

Cluster snapshots | Cluster Copy functions

Cluster Reporting

Start | Stop | Move resource operations

Dynamically adding / removing resources

Detailed MAN page:

CAA clcmd command


54/64





Introduced in bos.cluster.rte

I typically probeclstrmgrES to get

the cluster state

Think of the many uses:

Identify hdisk#s for differentPVIDs or volume groups

Compare file set levels betweenall of the cluster members

Check scriptlocations and

permissions

Leveraging new V7 Command Line Interface


55/64





The clmgr command can be used for many different operations:

Dynamic LVM changes (sample syntax) Description

clmgr modify vg add= Add physical volume to VG

clmgr add lv type=jfs2 vg logical_partitions= Add new logical volume

clmgr add fs type= lv

lv_for_log=inline inline_log_size=

Add file system to existing

logical volume

Cluster Operations (sample syntax) Description

clmgr move rg node= Relocate RG

clmgr start | stop cluster Start / Stop services cluster wide

clmgr online | offline node Start / Stop on individual node

clmgr sync cluster Verify & synchronize clusterclmgr replace repository Dynamically replace CAA repo disk

clmgr query cluster Query cluster attributes / status

Creating & Managing Application Monitors


56/64





Earlier releases did NOThave a command to suspend / resume monitors:

- clmgr manage application_controller suspend test_app1

- clmgr resume application_controller resume test_app1

"

Active application monitors will have a corresponding clapp_mon process

" Creation can be performed via SMIT, Director or CLI:

clmgr add mon appA_mon TYPE=Custom APPLICATION=appA MONITORINTERVAL=60FAILUREACTION=fallover STABILIZATION=300 RESTARTINTERVAL=1200

CLEANUPMETHOD=/usr/local/hascripts/appA_cleanup.sh RESTARTMETHOD=/usr/local/

hascripts/appA_restart.sh RESTARTCOUNT=3 MONITORMETHOD=/usr/local/hascripts/appA_monitor.sh


57/64





Troubleshooting


Capturing Cluster Information


58/64





Quick Troubleshooting:

Topology & Resource configuration

(netstat in | ifconfig a)

Individual Logs

(Error report | hacmp.out file | cluster.log)

Cluster snapshot

# clmgr add snapshot

Data Detailed Analysis:

PowerHA Information !snap e

/tmp/ibmsupt/snap.pax.Z

RSCT logs

/usr/sbin/rsct/bin/phoenix.snap !collects the RSCT information & logs

Includes everythingincluding RSCT logs,

PowerHA config/log

Consider using cllsif | clshowres orvariations using the clmgr command

Review the .info report file & the .odmcluster configuration stanzas

Troubleshooting Cluster Problems


59/64





Most widely used logs:

Log Files Description

/var/hacmp/adm/cluster.log High level cluster EVENTs

/var/hacmp/adm/hacmp.out Detailed EVENT processing

/var/hacmp/log/clstrmgr.debug Detailed information from clstrmgrES daemon

/var/hacmp/log/clutils.log Generated by cluster utilities

/var/adm/ras/syslog.caa CAA logging & Troubleshooting

Log Files Description

/var/hacmp/clverify/clverify.log Detailed verification check output

/var/hacmp/clcomd/clcomd.log Troubleshooting communication issues

/var/hacmp/log/cspoc.log.long Detailed information from CSPOC

/var/hacmp/log/autoverify.log Generated by Nightly Verification

/var/hamp/adm/history/cluster.mmddyyyy Cluster history files generated daily

Troubleshooting specific areas:

Automatic Nightly Cluster Verification


60/64





By Default the cluster will run a nightly Verification check at midnight

"

The clutils.log file should show the results of the nightly check

On V7 clusters check /var/hacmp/log/autoverify.log

Summary


61/64





Goal of the session:

-

Highlight useful information for new & experienced users- Cover scenarios an Administrator may run into

V6.1 is now officially out of Support

- Review migration options and attend appropriate sessions

There are a number of useful features for HA Administrators

- File collections

- Pager Notification

- Custom Events

- Custom Verification Checks

- Application Monitoring Framework

V7 Clusters are significantly different:

- Deployment Methods

- Command Line Interface

- Monitoring

Please fill out an evaluation!


62/64





!"#$ &'$() *'+,$-)" .$ /"01

@ IBMtechU


63/64


Copyright IBM Corporation 2015. Technical University/Symposiamaterials may not be reproduced in whole or in part without the prior

written permission of IBM.63

Questions?

Thank you for your time!

Continue growing your IBM skills


64/64


Copyright IBM Corporation 2015. Technical University/Symposiamaterials may not be reproduced in whole or in part without the prior


ibm.com/trainingprovides acomprehensive portfolio of skills and careeraccelerators that are designed to meet allyour training needs.

Training in cities local to you- where andwhen you need it, and in the format you want

Use IBM Training Searchto locate public training classes

near to you with our five Global Training Providers

Private training is also available with our Global TrainingProviders

Demanding a high standard of quality view the paths to success

Browse Training Pathsand Certificationsto find thecourse that is right for you

If you cant find the training that is right for youwith our Global Training Providers, we can help.

Contact IBM Training at [email protected]

64

Global Skills Initiative

Documents

PowerHA SystemMirror for AIX Common Administrative Tasks ABA3287