© 2011 IBM Corporation PowerHA SystemMirror for AIX IBM Power Systems

© 2011 IBM Corporation

PowerHA SystemMirror for AIX

IBM Power Systems

© 2011 IBM Corporation2

Virtualization without Limits Drive over 90% utilization Dynamically scale per demand

Dynamic Energy Optimization 70-90% energy cost reduction EnergyScale™ technologies

Resiliency without Downtime Roadmap to continuous availability High availability systems & scaling

Management with Automation VMControl to manage virtualization Automation to reduce task time

Workload-Optimizing Systems

AIX - The Future of UNIX

Total Integration with i

Scalable Linux ready for x86 Consolidation

Systems for a Smarter Planet

+

Power Your Planet


PowerHA SystemMirror

Power Systems High Availability Solution For mission critical application availability through planned and unplanned outage events

Editions targeted at data center and/or multi-site deployments, value grows in feature and function with each new release

Shared Storage Clustering Technology designed for automation and minimal IT operations.

Deep Integration for simplicity and reliability


PowerHA SystemMirror Strategy

Deep Integration – Cluster Aware AIX– Kernel based event processing– Centralize cluster topology management

Ease of deployment & ease of use– Systems Director based management– Discovery based deployment– Cluster wide security

Multi-Site & Disaster Recovery– Differentiate with IBM storage– Heterogeneous storage support

Solution Package Optimization – Standard Edition, Enterprise Editions


PowerHA SystemMirror Editions

PowerHA SystemMirror Standard Edition

Cluster management for the data center– Monitors, detects and reacts to events– Establishes a heartbeat between the systems– Enables automatic switch-over

IBM shared storage clustering – Can enable near-continuous application service– Minimize impact of planned & unplanned outages– Ease of use for HA operations

PowerHA SystemMirror Enterprise Edition

Cluster management for the Enterprise– Multi-site cluster management – Includes the Standard Edition function


Client Need: Data Center

Requirements– Near continuous application service – Recovery Time Objective (RTO) measured in minutes– Recovery Point Objective (RPO) is Zero (no data loss)– Eliminate affects of planned outages

Solution Strategy: – PowerHA SystemMirror Standard Edition


Requirements– HA/DR clustering for data center & multi-site operations– Automated role-swap operations – Readily demonstrate remote recoverability compliance

Solution Strategy: – Unified clustering solution for data center & multi-site resiliency– PowerHA SystemMirror Enterprise Edition

Data Center

Remote Site

Client Need Today: Unified HA/DR Solution


PowerHA SystemMirror 7.1 for AIX

Standard Edition PowerHA

AIX IT operations skill base

AIX Technical skill support base

AIX Shared Storage management

AIX Volume manager/Mirroring

AIX File System

AIX SMIT Management Tools

AIX Event/error management

AIX DLPAR HA/DR management

Application agents

Director based user interface

AIX Cluster Aware

Kernel based event management

Centralized cluster repository

Triple redundant multi-cast heartbeat

SAN based communications

HA coverage for root Vg storage

Sap LiveCache Hot Standby

Federated Security

PowerHA SystemMirror is differentiated by integration with AIX Cluster Aware

Kernel based event management

PowerHA skills are an extension AIX skills

WW IBM and BP community support PowerHA


PowerHA SystemMirror 7.1 for AIX – Key differences

■ Integration with Cluster Aware AIX

■ Heartbeat mechanism

■ Cluster Repository Disk

■ Cluster Command Line, clmgr

■ Systems Director Management Interface

Let’s take a closer look at these differences…


Deprecated features in v7.1

All non-IP network types– RS-232,diskhb,mndhb,TMSSA,TMSCSI

IP Network types ATM, FDDI, Token Ring Non-ECVGs Hostname changes IPAT via replacement

– HWAT Heartbeat over Aliasing Two-Node configuration assistant WebSMIT

– Online planning worksheets– Cluster configuration html report

Additional Limitations– Site Support– Cross-site LVM mirroring– IPV6 support– 16-node tested cluster restriction as per the README7.1.0.UPDATE


Cluster Aware AIX

AIX-Level Cluster – AIX functionality not PowerHA Provides cluster framework, heartbeating, disk fencing but no failover Available in AIX 6.1 TL6 and AIX 7.1 - bos.cluster.* PowerHA 7.1 hands over heartbeating (previously topology services) to

CAA CAA Cluster definition – on CAA Repository Disk PowerHA Cluster definition – in PowerHA ODM Different commands to query AIX and PowerHA cluster CAA Storage Framework replaces PowerHA non-IP network Also used by

– VIOS SSP

– DB2 PureScale


Existing – without CAA


grpsvcs daemon now called cthags

Topology services no longer used

With CAA


Cluster Aware AIX - differences

■ Repository disk– Minimum recommended size 10GB– Does not store PowerHA cluster configuration – Requires 4GB or 8GB FC adapters– Documented procedure for changing repository disk – recommended to avoid by proper

planning– Must be highly available / SAN RAIDed / Multipathed– 7.1 does NOT support 3rd party Multipath eg. PowerPath but 7.1.1 (Dec 11) adds

support for EMC PowerPath and Hitachi HDLM■ Sites

– Does not support sites■ Hostname

– CAA requires hostname and nodename must be the same


Cluster Aware AIX – differences continued■ 7.1 Requires specific vg and lv naming to be available# lsvg -l caavg_privatecaavg_private:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

caalv_private1 boot 1 1 1 closed/syncd N/A


caalv_private3 boot 4 4 1 open/syncd N/A

fslv00 jfs2 4 4 1 open/syncd /clrepos_private1

fslv01 jfs2 4 4 1 closed/syncd /clrepos_private2

powerha_crlv boot 1 1 1 closed/syncd N/A

■ 7.1.1 does NOT require specific vg and lv naming to be available as we no longer have repository disk filesystems

# lsvg -l caavg_privatecaavg_private:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT



caalv_private3 boot 4 4 1 open/syncd N/A

powerha_crlv boot 1 1 1 closed/syncd N/A


Cluster Aware AIX – differences continued

■ vSCSI support issues for repository disk– 6.1.6/7.1 does not check disk type

– 6.1.6 SP1 checks and will not allow vSCSI

– 6.1.6 SP2 checks and verifies vSCSI device is indeed SAN based

– NPIV / Fibre OK

■ Uses multicast for monitoring– Requires infrastructure to allow forwarding multicast packets

– Uses SRM (Scalable Reliable Multicast)

– Can test using mping AIX command

■ Heartbeat rates are not tunable– No failure detection rate settings (FDR)

– Completely different to heartbeat over IP used in previous versions

■ Uses internal algorithm to determine health– Does consider things like network/system load


“Monitoring” in CAA

Now in kernel space not user space– Higher priority, allows for messages to be sent from node experiencing

extreme resource contention - no DMS

SAN-based communication interface (SFWCOM)– Replaces disk heartbeat

– Uses target mode on HBA

– Also requires creation of VLAN 3358 to create sfwcomX devicedocumentation APAR IV03643 (google finds it)

CAA Repository Disk also used for monitoring, only if all else fails Monitoring is more advanced – many more stats

– Self-adjusting algorithms take into account load and network conditions

– Cannot adjust Failure Detection Rate

Can see all of this in lscluster command


Network

SAN

Simple Setup

Multiple channels – Network– SAN– Central Repository

Triple redundant communication pipe

Heartbeats

First line of Defense

Second line of Defense

Third line of Defense Cluster Repository

Default Multi Channel Health Management:


SMIT Menu updates


Move cursor to desired item and press Enter.

Cluster Nodes and Networks Cluster Applications and Resources

System Management (C-SPOC) Problem Determination Tools

Custom Cluster Configuration

Can't find what you are looking for ?

Not sure where to start ?



Cluster Nodes and Networks Cluster Applications and Resources

System Management (C-SPOC) Problem Determination Tools

Custom Cluster Configuration



HACMP for AIX


Initialization and Standard Configuration

Extended Configuration

System Management (C-SPOC)

Problem Determination Tools



HACMP for AIX


Initialization and Standard Configuration

Extended Configuration

System Management (C-SPOC)

Problem Determination Tools



New fastpath: smit sysmirrosmit sysmrsmit hacmp (v6.1)

Fastpath hacmp is still valid


User Defined Resources

Presently, A user can introduce a new resource type by creating a Application Server.

– PowerHA requires a start/stop/monitor for that resource

However, PowerHA follows a strict pre-known order to handles the resources.

– Volume Groups will be handled first – Application Servers will be handled last.

This approach is not flexible for end users

What’s new in the 7.1 release– User develops “bundle” which includes attributes/verifications/order for

PowerHA etc.– Framework accepts Methods to verify/start/stop/monitor/cleanup/restart

the user defined resource– A xml file can be supplied as input which will be having definition for the

user defined resource type– PowerHA allows to create instances of the user defined resource type

and these instances can be added into RG as resources.

DISK

VOLUME GROUP

FILE SYSTEM

SERVICE IP

APPLICATION

User Defined Resource

Acquisition O

rder

Rel

ease

Ord

er


New Dependencies

■ Previously, PowerHA supports only parent-child dependency among Resource Groups.

– Child RG starts after all of its parents start

– Child RG stops before its parents stop

■ Not sufficient to support complex applications like SAP/FileNet– FileNet App Server requires to be started only after database, but no need to be stopped if the DB is

down for sometime

■ What’s new– Two new dependencies introduced in PowerHA 7.1

• STARTAFTER

• STOPAFTER

– Affects how RG started or stopped (New)

– Simplified UI for managing these relations in both SMIT and Director


Dynamic Node Priority new Adaptive Fallover

■ Present PowerHA support static/dynamic failover policies– Static: New failover node will be the next node from the node participation list

– Dynamic Node Priority : allows the failover node to be having certain degree of free CPU/memory/IO activity

■ This is not sufficient in some specific cases like SAP Enqueue replication, where RG for Enqueue server need to failover where Enqueue Replication is running.

What’s new in 7.1– Introducing a new dynamic failover policy where, it asks user to supply a script which can dictate

the failover behavior.•The supplied script will be executed on all nodes by PowerHA to know whether the given node can be used as host for failing over RG•Since it is a script, no of checks can be performed by user to let PowerHA know whether the node can be used as a host on a dynamic basis.

The DNP feature is enhanced to support two more policies. The return code of a user-defined script is used in determining the destination node:

– cl_lowest_nonzero_udscript_rc– cl_highest_udscript_rc


Rootvg System Event

■ New kernel level monitoring■ Monitors the loss of rootvg■ Defaults response is to log event and reboot causing fallover to

occur■ Smitty sysmirror->Custom Cluster Configuration->Events->System

Events

Change/Show Event Response

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Event Name ROOTVG +

* Response Log event and reboot +

* Active Yes +


Clmgr – cluster command line

■ Much asked-for feature

■ Very complete, verbose, documented

■ All output logged automatically in clutils.log

■ -l flag does trace logging various levels

■ Can shorten keywords to aliases

■ Can do (almost) everything possible in smitty

■ Can save a huge amount of time in build, config, PD of clusters

■ Used by Systems Director plugin

■ Replaces previous clvt


Clmgr – cluster command line

■ Supported actions

– add

– delete

– manage

– modify

– move

– offline

– online

– query

– recover

– sync

– view

Supported object classes

• cluster

• site

• node

• interface

• network

• resource_group

• service_ip

• persistent_ip

• application_controller

• application_monitor

• dependency

• file_collection

• fallback_timer

• volume_group (incomplete coverage)

• logical_volume (incomplete coverage)

• file_system (incomplete coverage)

• physical_volume (incomplete

coverage)

• method (incomplete coverage)

• report

• snapshot

• tape


Clcmd – cluster distributed commands

■ /usr/sbin/clcmd

■ Provided by CAA

■ Distributes command to all cluster nodes

– Ease of use

– Reminiscent of dsh from SP2 days

■ Example

– clcmd cat /etc/cluster/rhostsNODE jessica.dfw.ibm.com

-------------------------------

jessica

jordan

-------------------------------

NODE jordan.dfw.ibm.com

-------------------------------

jessica

jordan


State-of-the-art interface

– No charge plug-in– Masks complexity– Central management – Real-time status– Smart Assist integration– Deployment wizards

IBM Director: PowerHA Management Interface


1 Websphere

2 IBM DB2

3 Oracle Database

4 Oracle Application Server

5 IBM HTTP Server

6 Tivoli Directory Server

7 SAP

8 FIleNet

9 Tivoli Storage Manager (3 Smart Assists)

10 Lotus Domino Server

11 MAXDB

12 Websphere MQ

13 AIX Print Server

14 AIX DHCP

15 AIX DNS

Simplified application availability

– Included in the Standard Edition

Advanced PowerHA agent

– Discover, Configure, and Manage

Support to define and deploy HA policies

– Exploit advanced relationships– Flexible failover policies and resource order

policies– Out of the box start, stop and health monitoring

for common workloads

PowerHA 7.1 Standard Edition Smart Assists


PowerHA SystemMirror Announce Highlights October annouce 7.1.1 TL

Deliver new services fasterSAP LiveCache hot standby– Fast failover for SAP

Deliver services with superior economicsEditions grow in value with each product release

– XIV and v7000 support with PowerHA 6.1 EE on AIX– SVC and v7000 support with PowerHA 7.1 EE on IBM I– EMC PowerPath and Hitachi HDLM multi-pathing based

disks supported as repository disks

– 5 year life cycle for PowerHA System Mirror 7.1 on AIX

Deliver higher quality servicesFederated Security– Centralized cluster wide security management


Disaster Recover Trends and Directions

LEGACY OPERATIONS Tape based back-up recovery Annual restore from tape exercises

ADVANCED OPERATIONS Data center clustering and Geographically dispersed clustering Semi-annual compliance testing

INTERMEDIATE OPERATIONS Geographic dispersion Data replication Optional compliance testing

RPO minutes RTO hours Manual compliance testing Data-centric recovery

RPO hours if not days RTO hours if not days Manual compliance testing Data-centric recovery

RPO minutes RTO less than an hour Automated compliance testing Application-centric recovery

Storage ReplicationFlash CopyReboot/Restore

No ReplicationRecover from Tape

PowerHA Storage ReplicationFlash CopyAutomation


PowerHA SystemMirror 6.1 Enterprise Edition Storage Options for your Enterprise

Sync mode Async mode Active Active Consistency groups

GLVM Yes Yes

XIV Metro Mirror Global Mirror

Storwize® V7000 Metro Mirror Global Mirror

SVC Metro Mirror Global Mirror

DS8800 Metro Mirror Global Mirror

EMC SRDF SRDF

Hitachi TrueCopy

Universal Replicator

TrueCopy


Hitachi VSP Truecopy


Truecopy


HP Storage Works P9500

Continuous Access Software replication

Continuous Access Software replication

Solutions for disaster recovery– Major vendors, popular options– IBM storage options that span the market


Two Data Centers

Systems remain active

Disaster recovery for out of region interruptions

Multi-site recovery with no data loss

Two Data Centers

Systems remain active

Enterprise Edition (6.1)

Disaster Recovery within a Metropolitan Region

PowerHA with

GLVM or Metro Mirror

SRDF, Truecopy

RPO=0 & RTO<1 hr

Standard Edition

Continuous Availability within a Data Center

Continuous access to data in the event of a storage

subsystem outage

Single Data Center

Applications remain active

PowerHA with

LVM Mirrors

RPO=0 & RTO=0

PowerHA with

GLVM or Global Mirror

SRDF, Truecopy

RPO secs & RTO <1 hr

Enterprise Edition (6.1)

Disaster Recovery Unlimited Distance

PowerHA SystemMirror Data Resiliency

© 2011 IBM Corporation39 39

DS8000 – IBM’s Flagship Enterprise Class Storage

Superior DR capabilities

Tight integration with IBM i

Reduced bandwidth costs

Lower data loss risk

XIV – Enterprise Class Storage Reinvented

Unique self optimizing grid architecture with extreme ease of use

Enterprise class storage at lower total costs

Metro and Global mirroring and other advanced features at no extra costs

IBM SAN Volume Controller (SVC) – Enterprise Class Virtualized Storage

Market leading Storage Virtualization

Ease of integration / leverage of existing storage infrastructure

Lower storage costs (30% utilization improvement)

Storwize V7000 - New Standard for Midrange Storage

Mid-range storage with enterprise capabilities

Built in storage virtualization for ease of integration

Incredible ease of use

Summary – Best Fit options

© 2011 IBM Corporation41 © Copyright IBM Corporation 2011

PowerHA Webcast V1.0

PowerHA Website– www.ibm.com/systems/power/software/availability/

HACoC (Availability Factory) [email protected]– Contact your IBM representative or an IBM Business Partner.

IBM Technology Service Offering for PowerHA SystemMirror XD deployment – http://www-935.ibm.com/services/us/index.wss/offering/its/a1000032

Redbooks– PowerHA on AIX (SG24-7739-00)– SG24-7841 IBM PowerHA SystemMirror Enterprise Edition – SG24-7845 IBM PowerHA v7.1 – Education: PowerHA for AIX Implementation, Config and Administration AN410 – Go to IBM.com/services/learning (search for AN410 or PowerHA)

Education: Lab Services AN44 Extended Distance and Disaster Recovery– http://www- 304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course_list&subChapter=194&subChapterInd=S&region=

us&subChapterName=AIX+high+availability&country=us – http://lpar.co.uk/HA/home.html

GLVM white paper– www.ibm.com/systems/resources/systems_p_os_aix_whitepapers_pdf_aix_glvm.pdf

IBM storage virtualization offerings– www.ibm.com/systems/storage/virtualization

SAP consulting services for POWERHA and POWERVM– [email protected]– [email protected]

Wiki– http://www.ibm.com/developerworks/wikis/display/WikiPtype/High%20Availability

Edison Group Paper– http://theedison.com/pdf/2011_Samples_IBM_PowerHA.pdf

Risk Self Assessment:– www.ibm.com/smarterplanet/us/en/business_resilience_management/overview/index.html?re=2brf24

PowerHA for AIX Resources


This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area.

Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions.

IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice.

IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment.

Revised September 26, 2006

Special notices


IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, Active Memory, Balanced Warehouse, CacheFlow, Cool Blue, IBM Systems Director VMControl, pureScale, TurboCore, Chiphopper, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Parallel File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, POWER7, System i, System p, System p5, System Storage, System z, TME 10, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.

A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.AltiVec is a trademark of Freescale Semiconductor, Inc.AMD Opteron is a trademark of Advanced Micro Devices, Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).UNIX is a registered trademark of The Open Group in the United States, other countries or both.

Other company, product and service names may be trademarks or service marks of others.

Revised December 2, 2010

Special notices (cont.)

Documents

© 2011 IBM Corporation PowerHA SystemMirror for AIX IBM Power Systems