IBM Software Group © 2008 IBM Corporation Business Continuity Advances with IBM Tivoli System Automation Family Dr. Norbert [email protected]

IBM Software Group

© 2008 IBM Corporation

Business Continuity Advances with IBM Tivoli System Automation Family

Dr. Norbert Lenz [email protected]

IBM Software Group

October 2008

IBM Software Group | Tivoli software

© 2008 IBM Corporation2 October 2008Business Continuity Advances - IBM Tivoli System Automation Family

Copyright and Trademarks

© Copyright IBM Corporation 2008

The following names are trademarks of the IBM Corp. in USA and/or other countries and may be used throughout this presentation:

AIX, CICS, DB2, eLiza, Enterprise Storage Server, ESCON, FICON, FlashCopy, GDPS, HyperSwap, IBM, IBM eServer, IMS, MVS/ESA, MQSeries, NetView, OMEGAMON, Parallel Sysplex, RMF, RACF, S/390, Syspex Timer, Tivoli, VTAM, WebSphere, z/OS, z/VM, System z, System p, System i

Other company, product and service names may be trademarks or service marks of others.



Agenda

IBM Tivoli System Automation – Product family and basics

IBM Tivoli System Automation for Multiplatforms

GDPS/PPRC Multi Platform Resiliency for System z (xDR)

IBM Tivoli System Automation Application Manager

Business Continuity Process Manager



Business Continuity Issues IT challenges

• Downtime unaffordable– Planned and unplanned

• Heterogeneous environments• Complexity

Customer pressures• Application availability • Operations complexity and costs• Automation implementation and

maintenance costs• Education requirements related• Rapid change of I/T infrastructure

• Loss of business• Loss of customers – the competition is just a mouse click away• Loss of credibility, brand image and stock value

Reasons for planned downtime• Maintenance• Tests

Reasons for unplanned downtime• Operator errors• Application failures• Environmental failures

– OS failures– HW failures– …– Disasters

40%Operations

Errors

40%Application

Failure

20% Environmental

Factors, HW, OS, Power,

Disasters

20% Environmental

Factors, HW, OS, Power,

Disasters

Source: Gartner Group, 2007



Customer Value of Business Continuity

BC reduces the risk of a business interruption•Ensures Continuous Availability (CA) of applications

–Continuous Operation (CO)

Minimizes impact of planned outages–High Availability (HA)

Minimizes impact of unplanned outages

BC helps to stay in business when a severe interruption occurs

•Supports Disaster Recovery (DR)

HAHA

Site Mgmt

Data Mgmt Data Mgmt

Site Mgmt

DR DR

Replication



Vision For a Business Continuity SolutionBusiness Continuity…

Provides both Continuous Availability and Disaster Recovery for heterogeneous distributed IT business applications

• "Recover my business rather than my platform technology“

Ensures successful recovery via automated processes

• Pre-canned, pre-tested

• Can be handled by less-skilled operators

Allows management on the “business impact” level

• Granular

• Customer flexibility on desired RTO, RPO quality of service

Leverages existing customer/IBM investments and technology

System z is the Business Continuity management hub



Tivoli System Automation EvolutionNetwork Edge/Web Application Data/Transaction

Leading the Enterprise

Step I: Componentize SA z/OS

SA z/OS

SA MP

Step II: Leverage components across platforms

Step III: Integrate operational capabilities end-to-end

Automation Engine

SA z/OS

e2eSA App Man *

e2e

ISMBCPM

Step IV: Business Continuity: Leverage automation technology in IBM Service Management

SA MP on Linuxon System z,x or p

MSCSVCS

HACMP

SA MP on AIXOr SUN Solaris

SA z/OS

SA MP on Win

SA MP on AIX

SA MP on Linux(x,p,z)

SA MP on Windows

* SA Application Manager is the new name for SA MP End to End Automation

SA MP on SUN Solaris

NewV3.1

NewV7.1



Business Continuity Solution Architecture

Change Management

Configuration Management

Release Management

Storage Process Manager

Clustered Applications / Middleware /

OMP / Storage Systems

End-to-End Multi-Tiered

Application and Storage System

Process Management

Process Management Infrastructure

AvailabilityProcess Manager

IBM Service Management Process Runtime InfrastructureCMDB Maximo Integration Modules

IBM Tivoli System AutomationApplication Manager

AIX / LinuxWindows / Solaris z/OS

Business ContinuityProcess Manager

New inV3.1

New V7.1



System Automation Basic Principles

Coordinated

Restart

& Failover

Automation

Agent

Event

Automation

Engine

Operator

Administrator

•Restart & FailoverRules•Resource Groups•Relationships and Dependencies

Scheduler

Orchestration

mySAPPolicy

WASPolicy

CustomerPolicies

…MQPolicy

IMSfeature

CICSfeature

Operator

DB2Policy

• Capability to automatically start and stop resources with knowledge of…• Resource groups, relationships, backup resources, …

• Provide Continuous Availability of IT resources• Applications, processes, IP addresses, file systems, …

IT Manager



Goal-driven - Automation in a Nutshell

Request ChangesMonitor Events

Resources & Relationships & Rules

Evaluate relationship correlation rules

Apply state change

Find actions to Submit Action

Translate event to state

Translate Action

Resources

Decide

Monitor and Control

M

A

P

E

Resource DefinitionDesired Operational StatesStart/Stop SequencesGroupingFailure Actions

Automation Policy

reach goal



What is the Power of an Automation Policy? Easy definition

• Pre-defined automation for common applications

• Faster time-to-value

• No script writing

– Elimination of coding errors– Easy to maintain

• Easy to build ‘business view’

– Group nesting, arbitrary abstraction level

• More efficient use of scarce

– ‘people’ resources

Consistent, reliable, automation actions • Testing abnormal condition actions is difficult and sometimes incomplete with scripts

• Policy definitions can be cloned for similar requirements elsewhere in the enterprise

• Management of entire business applications, rather than individual resources

Resources

Aggregated Resources

Business System



IBM Tivoli System Automation for z/OS Proven solution for Automation of IBM z/OS Sysplex

• Best of breed mainframe automation

• More than 1000 applications in sysplex with complex dependencies

• Automated Operation and High Availability

• Grouping and Relationships described in Automation Policies

• Integration with Performance Monitoring (TEP, OMEGAMON XE, CICSPLEX SM)

CF

AgentAgent

Agent

AutomationManager

z/OS Sysplex

IBM Tivoli System Automation

for z/OS

New inV3.2



Agenda








IBM Tivoli System Automation for Multiplatforms (SA MP) Automated Operation and High Availability of Applications on Linux, AIX and Windows

• Automates failover, stop, start, restart in place and move operations for applications

• Supports AIX, Linux on System z, x, p, i, Windows and SUN Solaris

• Monitors and controls

– Applications– Network (TCP/IP)– File Systems

• Automated Operations & High Availability with best of breed Automation Manager from IBM Tivoli SA for z/OS

• Based on Automation Policies

Linux, AIX or Windows

ResourceManager

ResourceManager

ResourceManager

AutomationManager

RSCT

IBM Tivoli System Automationfor Multiplatforms




Event: On Start of RG_WebRG_Web already Online?Yes exitNo continue

All members Offline?No Problem, exitYes continue

DB startable on node1?No goto StartNode2Web startable on node1?

No goto StartNode2

Start DB on node1

Wait until DB is OnlineStart Web on

node1

Wait until Web is OnlineSet group

status to OnlineExit

StartNode2:DB startable on node2?No goto StartNode3 or

exit Web startable on node2?

No goto StartNode3 or exit

Start DB on node2

Wait until DB is OnlineStart Web on

node2

Wait until Web is OnlineSet group

status to OnlineExit

Event: On failure of DBRunning on node1?

Stop Web on node1Wait until Web is OfflineGenerate Event: On Start

of RG_WebExit

Running on node2?Stop Web on node2

Wait until Web is OfflineGenerate Event: On Start

of RG_WebExit

Event: On failure of WebRunning on node1?Restart on node1 possible?

Yes Start Web on node1

Wait until Web is Online

ExitNo Stop DB

Wait until DB is OfflineGenerate Event:

On Start of RG_WebExit

Running on node2?Restart on node2 possible?

Yes Start Web on node2

Wait until Web is Online

ExitNo Stop DB

Wait until DB is OfflineGenerate Event:

On Start of RG_WebExit

Web

DBVIPA

DependsOnDependsOn

No programming required Simple policy with

• Resource information

• Relationships

• Groups of resources

• Incremental policy activation / deactivation

• Template Processing

Allows operators to control applications on

business level

• Frees operators from remembering application

components and relationships

New resources or systems can be added

without re-writing scripts

Automation Policy Reduces Implementation Time, Coding and Support Effort

Web

DBVIPA



NodeNodeNodeNode

AppServer Inst

replicate ERS

NodeNode

ERS

ES MS

VIPA

replicate

conn

ect

ES MSVIPA

connect

Example: mySAP Enqueue Server Recovery

SA Policies: ES, MS, VIPA collocated ERS starts after ES ERS is anti-collocated to ES ES collocated to ERS if online and ES offline

SA Actions:1. Ensure correct start-up sequence /

node of ES and prereq’s

2. Ensure correct start-up sequence / node of ERS

3. Recognize Node 1 failure

4. Failover ES and prereq’s to Node 2

5. Connect SAP App Server to VIPA on Node 2

6. Wait for information transfer from ERS to ES thru shared memory

7. Move ERS to Node 3

8. Resume SAP operations



Pre-canned Automation Policy Templates (Linux / AIX) Data Management

•DB2 v9 (ESE, ESE DPF, HADR) (Note: SA MP is shipped with DB2 on Linux and AIX)•DB2 BCU•DB2 8.x (ESE, ESE DPF, HADR)•DB2 7.x WE, EE•Oracle 10g•Oracle 9i•Oracle 8i

Tivoli Products•Tivoli Provisioning Manager 5.1.1•ITM 6.2•CCMDB V7.1•TADDM 7.1•CCMDB V1.1.1 (this includes TADDM, WebSphere Application Server, WebSphere Portal Server, DB2, IBM HTTPS, and TDS)•Tivoli System Automation for Multiplatforms (SA MP) (note: for the End-to-End Automation Component)•Tivoli Workload Scheduler•Tivoli Storage Manager (TSM) (includes Client, Server and Admin Server)•Tivoli Enterprise Console 3.8

SAP •SAP Replicated Enqueue environment•SAP Application Server•SAPDB•(Note: SA MP is shipped with DB2 on SAP DVD and integrated in SAP‘s installation procedure)

WebSphere•WebSphere Application Server 6.0•WebSphere MQ

Shared File Systems•NFS Server•NFS Client•Samba

Groupware•Sendmail 8.11

Replication•GLVM (AIX)•Metro Mirror with DS4000

Web Servers•Apache Web Server•IBM HTTP Server

Currently under development / planned:•Tivoli Directory Server (TDS)

new



Weber Automotive Drives its SAP Software with Linux on POWERWeber Automotive GmbH, based in Markdorf, Germany, manufactures drive components and fiber reinforced plastic parts for the automotive industry.

To promote the high availability that Weber Automotive needs from its production systems, the company also chose to deploy IBM Tivoli System Automation for Multiplatforms (Tivoli SA). The software manages the SAP application clusters and uses policy-based self-healing technologies to analyze and fix most problems automatically – whether they occur in the application, operating system, or middleware layer.

"Tivoli SA comes with a plug'n play automation module for SAP applications, so it can deal with the majority of problems that occur in SAP software environments with minimal customization," says Günter Dürringer, Head of IT at Weber Automotive. "By understanding the relationships between application components, Tivoli SA finds and resolves issues quickly, and minimizes the need for IT staff to intervene."He adds: "With Tivoli SA, we are able to maintain very high availability for our SAP software environment, ensuring that systems remain online and accessible to users at all times."



SA MP Towards a Distributed World

Linux on System z

Solarison SUN-servers

z/OS on System z

AIX and Linux on System p

HP-UXon HP-servers

Linuxon BladeCenter

Linux on iSeries

Linux on PCs

(System x and non-IBM PCs)

Windows on PCs

(System x and non-IBM PCs)

SA MP SA MP

SA MPSA MP

SA MP

SA MPSA z/OS

System z„classical“ UNIX ServersPC Systems

SA MP

New inV3.1



Agenda








GDPS Overview Continuous Availability and Disaster Recovery solution

•GDPS/PPRC (Peer to Peer Remote Copy (PPRC) - Synchronous)

– PPRC – capable disk subsystems are supported•Multisite Sysplex (fiber distance between sites 100 km - max)

•No or limited data loss in unplanned failover - user policy

•Planned and Unplanned reconfiguration support

Disaster Recovery solution •GDPS/XRC (eXtended Remote Copy (XRC) - Asynchronous)

•Supports any distance

•Production systems in Site 1

•Limited data loss to be expected in unplanned failover

•GDPS initiates restart of production systems in Site 2

Common functions (GDPS/PPRC and GDPS/XRC)•GDPS solution extended to tape resident data

•Point-in-time copy created (FlashCopy™)

•Management of System z Operating Systems (Linux® on System z, z/VM™ , VSE/ESA™ )

1 4

3

2

ESCON

PPRC

1 4 3 2

XRC

SDM

UNIX, NTz/OS

Virtual Tape Controllers

Virtual Tape Controllers

Primary Site

Secondary Site

TCDBTMC

Catalog

TCDBTMC

Catalog

PPRC

PtP VTS



Re-IPL in Place If

• GDPS detects that a z/OS system is no longer active

• Auto IPL has been enabled

• The threshold of the number of IPLs in the predefined time window has not been exceeded

• No planned action is active Then

• GDPS automatically re-IPLs the system in place

• Brings it back into the Parallel Sysplex

• IBM Tivoli System Automation for z/OS (SA z/OS) restarts the workload

z/OS

Check Conditions

Re-IPL



Site takeover

Reset LPARs

Start workloadon site 2

When GDPS detects a complete site failure, it can offer a takeover of the workload on site 2• If additonal capacity is needed on the backup site, Capacity Backup Upgrade (CBU) can be used

CBU

Site 2Site 1



HyperSwap

Site 1 Site 2

PPRCPPRC

capable disks

I/O

z/OS z/OS

I/O

When GDPS detects an disk outage, access to primary disks can be switched to secondary disks

Can swap large number of devices very fast Includes all volumes attached to the system Non-disruptive – applications keep using same logical device addresses Planned (Maintenance) and Unplanned (High Availability) Scenario

PPRC capable

disks




Business Continuity Solution for z/OS and Linux on System z on z/VM

• Based on GDPS

• Leverages existing and proven solutions

– GDPS – SA z/OS– IBM Tivoli System Automation for

Multiplatforms (SA MP)

• Provides coordinated cross platform business resiliency for OSs running on System z hardware

• Integration point of z/OS and Linux on System z

GDPS

SA MP

SA z/OS

NetView

z/OS

Linux on System z

z/VM

System z Hardware




xDR extends GDPS to support not only z/OS but also Linux on System z• Disk error detection

• Heartbeat for sanity checks

• Re-IPL in place

• Coordinated Site Takeover

• Coordinated HyperSwap

xDR is not a Product• In order to implement xDR, the following products are needed:

–GDPS

–System Automation for Multiplatforms

–Linux on System z

–z/VM, if Linux is running on z/VM



GDPS/PPRC Multi Platform Resiliency for System z Scenarios Examples:

• WAS, Portlet servers running on Linux on System z under z/VM, CICS and DB2 running on z/OS sysplex• mySAP app servers running on Linux on System z under z/VM, mySAP DB server running on z/OS• Other Linux on System z applications, e.g. mail servers running on Linux on System z under z/VM• Communication Controller for Linux on System z

ECKD

GDPS K system on

z/OS

LPAR4

NetView

SA z/OS

ECKDPPRC

Site 1

CICS,

DB2 on z/OS

LPAR2

SA z/OS

NetView

mySAP

DB Serv on z/OS

LPAR3

SA z/OS

NetView

Expendable work

load

LPAR6

Expendable work

load

LPAR5

z/OS Sysplex

CBU

Site 2

Capacity Upgrade on

Demand

GDPS Site Takeover / Failover

HyperSwap

LPAR1

z/VM

S

S

S

L

A

e

A

I

P

r

N

v

M

U

App

s

P

X



xDR Customer Value

Industrial strength business continuity solution for Linux on System z based on GDPS•Pretested solution, enables lower skilled operators to perform DR

High customer value for coordinated Linux on System z – z/OS business continuity•Single point of control to manage cross platform business continuity

•Coordinated HyperSwap

-E.g. because storage subsystems are used by both, Linux on System z and z/OS

•Coordinated site takeover / failover

Business continuity augments the benefits for Server Consolidation on Linux on System z



CeBIT Demo: Unplanned HyperSwap Scenario

PPRC

GDPS HyperSwap

EIF

Site 2

z/OS

NetView

GDPS

Autom.

Table

SA z/OSSite 1

z/VM

Production

Linux

xDR Proxy

CLI

LOGRDR

SA MP

CPINT

Web

Online

Shop

z/OS

Production z/OS

SA z/OS

FTPD

z/OS Sysplex

ESS ESS



GDPS/PPRC Multi Platform Resiliency for System z GDPS/PPRC Multi Platform Resiliency for System z (xDR)(xDR)

Coordinated near-continuous availability and DR solutionNew support: two K Systems, z/VM CSE data sharing

CF1

K1z/OS

P1z/OS

Linux z/VM P2z/OS

K2z/OS

SITE 1 SITE 2CF2

S SPL PL

L

k-sys

LP

LS

VMLinux Guests

Linux Guests

CDSCDS

LL

k-sys

Linux GuestProxy

GDPS automates

Linux startup

CBU

Planned &

Unplanned HyperSwap

Boom!

Requires Tivoli System Automation for Multiplatforms

LL



GDPS/PPRC Multiplatform Resilience for System z (aka xDR) – active / active - HyperSwap Experience

z/OS PPRC Volume Pairs

PPRC Pairs per z/VM Cluster

Planned HS RESYNCH

UIT

Planned HS SUSPEND

UIT

Unplanned HyperSwap

UIT

1,200 20 40 sec 8 sec 4 sec

UIT = User Impact Time

z/VM Linux

z/OS ... z/OS

CF

z/OS ... z/OSz/VM Linux

CF

P

10-way Parallel Sysplex (CICS, WebSphere, DB2)

HyperSwap

CF Duplexing

S

Business Requirements No loss of committed data (RPO=0) Continuous data availability for z/OS and

Linux hosted by z/VM Supporting site maintenance without

application outage Coordinated disaster recovery for

heterogeneous System z applications (RTO < 30 minutes)

5 km

z/VM Cluster

z/VM Cluster



GDPS/PPRC Multi Platform Resiliency for System z GDPS/PPRC Multi Platform Resiliency for System z (xDR)(xDR)

Coordinated near-continuous availability and DR solutionNew support: two K Systems, Linux swap disks mirrored with mdraid

CF1

K1z/OS

P1z/OS

Linux z/VM P2z/OS

K2z/OS

SITE 1 SITE 2CF2

S SPL PL

L

k-sys

LP

LS

Native Linux

Native Linux

CDSCDS

LL

k-sys

Native LinuxSwap

GDPS automates

Linux startup

CBU

Planned &

Unplanned HyperSwap

Boom!

Requires Tivoli System Automation for Multiplatforms

Native LinuxSwap

LL

mdraid



Agenda








Actual Application Architecture for Consumer Electronics Company

SA Application Manager is Key for Business Continuity

AutomationManager

System Automation

Manages applications and resources with complex relationships

Automated operations

High availability

Rapid error detection

Clustered environments

SA Application Manager

Cross cluster

Cross platform




Enterprise Service Group

Windows Linux ClusterSUN Solaris

startsAfter

First Level Automation

End-to-End Automation LevelApp. ServerWEB Server Database


My Business Application

Customers need to automate operations for n-tier applications – Enable operators to work on End-to-End level without having Operating System -

specific knowledge – Reduce labor cost associated with maintaining availability levels– Reduce cost of downtime

– shorten repair/maintenance windows– reduce operator errors

– Based on best of breed Automation Manager and Automation Policies



Automation Operation ConsoleDomain Hierarchy

-SA MP Automation Clusters (Linux or AIX)-SA z/OS Automation Sysplexes (z/OS)-HACMP (AIX)-MSCS (Win)

Pageable Resource Table- Hierarchy of Automated Resources

Monitor & Control all Resources

-Name, Owner & SupportInfo-Resource State-Relationships-Start and Stop Operation

The end-to-end automation domain supports the capability to suspend automation for resources• Use this feature e.g. when applying service



Tivoli System Automation Product Family


IBM Tivoli System Automationfor Multiplatforms

IBM Tivoli System Automation

for z/OS

Automation Adapter

Automation Adapter

Automation Adapter

Other Clustering/HAProducts:

• IBM HACMP• MSCS• Veritas

Linux on x SeriesLinux on p SeriesLinux on z SeriesAIXWindows

SUN Solaris

z/OS

Linux on x SeriesLinux on p SeriesLinux on z SeriesAIXWindows

z/OS

Linux AIXWindows SUN Solaris

Available AdapterIBM HACMPMSCSVeritas Cluster Services

New inV2.3 New in

V3.1

New inV3.1



SA Application Manager - Graphical Automation Policy Editor

Graphical editing of policies•SA Application Manager

•SA for Multiplatform

Create, visualize, extend policies

Web application •Runs in Integrated Solution Console (ISC) together with SA Operations Console

Integrated policy checker

XML specific details of policies hidden from the end user

New inV3.1



SA Application Manager Measurement and Reporting

New inV3.1



PPRC

z/OS SysplexGDPS

SA AppMan

ClusteredApplications

GDPS K-System

Site 1 Site 2


SA Application Manager – Integration with GDPSDistributed Disaster Recovery (DDR)

New inV3.1



SA Application Manager – Integration with GDPS Scenario: Rolling Disaster with Unplanned Site Takeover

Site 1 Site 2

SA App Man GDPS K System

Linux, AIX, Win, …

local clusters per site or stretched clusters

PPRC

SA App Man GDPS K System

z/OSspanned sysplex

Operator

STOP

START

Win, AIX, … Linux

KILL

Takeover Prompt

KILLSTOP

START

A Potential Disaster is Detected

Decision for a Site Takeover Made

Site Takeover CompleteStop Discretionary Workload

at Recovery SiteStart Production Workload at

Recovery SiteNew in

V3.1



SA Application Manager - CCMDB Integration Provides data integration with Change and Configuration Mgmt Database (CCMDB) Supports Reconciliation of already discovered Configuration Items (CIs)

New inV3.1



New tool for end-to-end automation value!

Uses client‘s own data:• Operations staff

• Automation staff

• Education and personnel costs

• Number of composite applications

• Optional input like downtime and customization effort saved

Assumptions:• 2 junior operators per shift

• 20% productivity gain for senior operators

• 50% productivity gain for automation team

• 8h/year less downtime

• Based on client observations, best practices and expert knowledge

Downtime saved in h/yearComposite End-to-End applications 10correct & fast startup 3recovery with cross platform dependencies 2cross cluster/platform move 1Faster problem determination 2Total hours 80

Savings per year Persons K€ /yearTotal

K€/yearjunior level operators 6 75 450senior level operators 3 90 270Automation team 5 110 550Application customization and automation savings 0,56 100 56

hours K€ / hourSavings from higher application availability 80 10 800Savings from less education efforts 71

Total 2196



More Value

Rating Details

High Reporting of startup, recovery and shutdown timesCan be used to identify and eliminate problems which further

increases uptimegeneric reporting to check SLAs etc.

High Population and currency of CCMDB automatic

Priceless Faster detection of rolling disasters by GDPS (SA AppMan DDR)

High Higher availability of composite applications in disasters due to cross-site move initiated by GDPS (SA AppMan DDR)



Agenda








AutomationManager

End to end toolsL

abo

r C

ost

Time

StorageExperts

And tools

Application experts and

tools

Databaseexperts and

tools

Securityexperts and

tools

Networkexperts and

tools

Availability Management

Security and Compliance Management.

Change Management

Release Management

Mainframe experts and

tools

Information Mgmt.

UnixExperts

And tools

IT Dilemma: Managing Cost and Responsiveness Across IT Silos

Integration People, Processes, Tools (end to end), Information

Sense Isolate Diagnose Take Action

Evaluate

Many Businesses Struggle to Manage Composite Applications



IBM Tivoli Service Management Center for System z Completing System z and Completing your Enterprise

Tivoli Business Service Manager• Improve the alignment of IT objectives and business objectives

Tivoli Application Discovery & Dependency Manager• Improve operational visibility

Tivoli Service Request Manager• Improve end user satisfaction and control over incidents

Tivoli Change & Configuration Management Database• Improve business control

Tivoli Business Continuity Process Manager (BCPM)• Improve business resilience

Tivoli Security Management• Improve business integrity

NEWNEW

NEWNEW

NEWNEW

NEWNEW

NEWNEW

NEWNEW



Business Continuity Process Integration

IT Service Continuity

Install, configure RTO and RPO

Identify the Business Requirements

Service LevelManagement

ExposeSLA, OLA

Incident Management

Investigate & Diagnose Incidents

Prioritize Incidents by

business impact

Incident Resolution &

Recovery

Incident Management

Investigate & Diagnose Incidents

Prioritize Incidents by

business impact

Incident Resolution &

Recovery

Create and Maintain Service Continuity Strategy

Create and Maintain Service Continuity Plan

Prepare IT Service Continuity capabilities

Execute IT Service Plan

Outage Analysis

Plan Backup Capacity

CapacityManagement

Change for IT Redundancy

ChangeManagement

Identify the IT processes for

the critical business

(automated) Recovery

ChangeManagement

Estimate risk and vulnerability

Identify technical options (RTO and RPO per platform

Estimate the impact of a loss or degradation

Approval to declare a Disaster

Identify products, and services for

RTO and RPO configurations

Understand Define Plan Install/Configure/Test Monitor Execute

Verify RTO and RPO

Evaluate e2e RTO/RPO

Understand dependencies

RTO and RPO simulation and verification

Automated Recovery

Declare RecoveryCompletion

Declare a Disaster



Peo

ple

Workflow

Pla

nContext

IBM Tivoli BCPM

Infrastructure

CI

StartOutageAnalysis

Approval Appr.? Stop

DeclareDisaster

Verify Recovery

ExecutePlan

DeclareDisasterFinished

Stop

Auto-Approve

?

Auto-Approve

StartOutageAnalysis

Approval Appr.? Stop

DeclareDisaster

Verify Recovery

ExecutePlan

DeclareDisasterFinished

Stop

Auto-Approve

?

Auto-Approve

Disaster Recovery Workflow

Analyst

Approver

Manager

Planer

Recovery PlanHistory

adapts...



DR testing Challenges Test Coverage

• Processes (IT and non IT)

• IT Redundancy and Recovery Setup

– Storage/Data, CFs, CDS, Servers, Network, Databases, Application– Sites

• IT Management System

– (GDPS, HACMP etc. Monitoring, Automation/Scripts) Challenges

• How can we test/simulate disasters (significant outages)

– to get a reasonable test coverage– and minimize the impact to the active production

Reasons• Audit the efficiency of the DR plan

• Benchmark BC recovery

• Rehearsal of BC recovery

• Document compliance to regulations

• ….

How can I test IT recovery without executing it ??



DR Test Cycle Test Planning

• Define Outage Scenario– What is broken and why is it broken (e.g. Site because of fire)

• Define Target/Objectives– Test target, (Process (People) , IT infrastructure, IT Management System) – How do measure success (e.g. time, data quality)

• Define Test Type and Rules– Paper only, Human part only, IT test transactions, FlashCopy restarts – Adhere strictly to planned procedure– Document required deviations

• Define People/Role (Administration, Observation, Documentation)• Define Schedule (Date/Time/Duration)

Test Execution• Decide/Approve/Declare the Test Start/End• Step thru

Test Results• Document results• Plan the changes• Feed experience for next test cycle



GEO112E

NetView GDPS

Context CI

IT Infrastructure

Test CI Data

CI

History

SAIOM

Simulation / Test History

TestPlan

RecoveryPlan

Workflow

ApproveproposedRecovery

Notifyimpactedpeople

VerifyRecovery

CompleteRecovery

AnalyzeOutage

Collaboration

Crisis management - based on CCMDB / BCPM



SA Integrated Operations Management (SA IOM )

Automatic Alert forwarding to the person(s) „in charge“

• Flood, fire, abended application Comprehensive set of notification

methods• SMS, voice, pager, email, script

API Interface Built-in escalation if person does

not acknowledge in time Web-based UI to

• define escalation policy

• Acknowledge alerts

• Watch active alerts



Sample Customer Roadmaps towards BC

1) SA z/OS2) SA MP, HACMP3) SA AppMan4) custom DR scripts5) CCMDB6) BCPM

SA z/OS

CCMDBBC PM …

e2e data

mgmt

SA AM – GDPSIntegr.

xDR

TSMTPC-R SA AppMan

GDPS

VCS, MSCS

HACMP

SA MP

OPS/MVS

…AvailPM

…

cust scripts

…

SA z/OS

CCMDBBC PM …

e2e data

mgmt


xDR

TSMTPC-R SA AppMan

GDPS

VCS, MSCS

HACMP

SA MP

OPS/MVS

…AvailPM

…

…

SA z/OS

CCMDBBC PM …

e2e data

mgmt


xDR

TSMTPC-R SA AppMan

GDPS

VCS, MSCS

HACMP

SA MP

OPS/MVS

…AvailPM

…

cust scripts

…

1) OPS / MVS2) GDPS with SA z/OS3) xDR with SA MP

1) SA z/OS2) HACMP, VCS, MSCS3) GDPS4) xDR with SA MP5) SA AppMan6) SA AM – GDPS integration7) CCMDB8) BCPM

custscripts

Examples for incremental growth of HA/DR functionality



End of Presentation

Thank you very much for your attention

http://www-306.ibm.com/software/tivoli/partners/index.jsp



Links• Home pages

– SA z/OS http://www.ibm.com/software/tivoli/products/system-automation-390/

http://www-03.ibm.com/servers/eserver/zseries/software/sa/– SA MP http://www-306.ibm.com/software/tivoli/products/sys-auto-multi/– SA AppMan http://www-306.ibm.com/software/tivoli/products/sys-auto-app-mgr/– BCPM http://www-306.ibm.com/software/tivoli/products/business-continuity-process-mgr

• SA MP Data sheet: http://www.ibm.com/common/ssi/fcgi-bin/ssialias?infotype=pm&subtype=sp&appname=SWGE_TI_PA_USEN&htmlfid=TID10427USEN&attachment=TID10427USEN.PDF

• SA MP Interactive Flash Demo: http://www-306.ibm.com/software/tivoli/library/demos/sa-linux.html

• SA MP documentation: http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.1.html

• All SA MP STE webcasts: http://www-950.ibm.com/search/SupportSearchWeb/SupportSearch?action=search&pageCode=SBSXP&brand=tivoli&sortBy=3&pageNumber=1&searchTerms=tste_webcast&products=&addSearchTerms=SAMP&submit.x=8&submit.y=10

• Download SA MP Plug’n Play Automation Modules: http://catalog.lotus.com/wps/portal/tsa

• SA MP Forum: http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1175

• Data sheet: http://www.ibm.com/common/ssi/fcgi-bin/ssialias?infotype=pm&subtype=sp&appname=SWGE_TI_PA_USEN&htmlfid=TID14019USEN&attachment=TID14019USEN.PDF

• Redbook „End-to-end Automation with IBM Tivoli System Automation for Multiplatforms” http://www.redbooks.ibm.com/abstracts/sg247117.html?Open

• SA AppMan documentation: http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforApplicationManager3.1.html

• Webcast: “High Availability and end-to-end automation of AIX, Windows, Linux and z/OS applications using Tivoli SA”: https://cpc.on.intercall.com/confmgr/view_stored_doc.jsp?docId=9179101159285987505108513435&docType=recording

• SA AppMan self-running demo: http://www-01.ibm.com/software/tivoli/library/demos/tivoli-system-automation.html

• SA AppMan forum: http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1304

Documents

IBM Software Group © 2008 IBM Corporation Business Continuity Advances with IBM Tivoli System Automation Family Dr. Norbert [email protected]