40
© 2010 IBM Corporation © 2010 Rocket Software Information Management IMS Cloning Tool Faster, Automated, IMS Cloning and Refresh Operations Ron Bisceglia 15 June 2010

Dfw ims cloning-tool_dfwug

  • Upload
    ibm-ims

  • View
    740

  • Download
    2

Embed Size (px)

Citation preview

Page 1: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

IMS Cloning ToolFaster, Automated, IMS Cloning and Refresh Operations

Ron Bisceglia

15 June 2010

Page 2: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

2

Agenda

Trends and directions

Cloning terminology

Use cases

Traditional Copy/Cloning Methods

Options for copying data

Clone IMS systems by volume

Refresh IMS Databases by data set

Page 3: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

3

Large DB2 and IMS systems require high availability– Fast and non-intrusive backup and cloning facilities are required

– Fast recovery and cloning capabilities are required to minimize downtime and promote high

availability

– Most backup, recovery and cloning solutions do not leverage storage-based fast-replication

facilities

Storage-based fast-replication facilities are under-utilized– Tend to be used by storage organizations

– Tend not to be used by database administrators (DBAs)

Storage aware database products– Allow DBAs to use fast-replication in a safe and transparent manner

– Provide fast and non-intrusive backup and cloning operations

– Simplify recovery operations and reduces recovery time

– Simplify disaster recovery procedures

The amount of data is increasing

– More data needs to be processed

Database and Storage Administration Trends and Directions

Page 4: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

4

Database and Storage Integration

Mainframe

Database

Systems

Storage Aware

Database Tools

Application and

Database Management

Domain

Storage Administration

and

Business Continuity

Domain

• Organizational Integration

• New Backup Methods

• New Recovery Strategies

• Business Recovery Monitoring

• Cloning Automation

• Disaster Restart Solutions

Source

DatabaseBackup,

Clone,

DR

Page 5: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

5

Database and Storage Integration Operational Advantages

Reduce backup, recovery, and cloning administration costs

Reduce host CPU and I/O resource utilization

Perform backups and create clone copies instantly

Fast restore and parallel recovery reduces recovery time

Simplify disaster recovery operations and procedures

DBMS and storage-based fast-replication integration– Leverage storage processors and fast-replication investments

• IBM, EMC, HDS, STK

– Expose fast-replication capabilities to the DBAs safely and transparently using “storage-aware”

database utilities

Provide a sophisticated infrastructure and metadata to manage the DBMS and storage processor coordination

Page 6: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

6

DBA’s use traditional database backup and recovery tools for cloning– Hard to integrate new cloning methodologies

– Uncomfortable with solutions for system cloning and database refresh operations

Lack of database and storage administration coordination– Storage processor fast-replication facilities are not well understood by application and DBA personnel

• DBA’s don’t trust storage technologies

– Database applicability of fast-replication not well understood by storage administrators

• Storage groups don’t trust DBA’s

Storage-aware database utilities resolve these issues

Database and Storage Integration New Solutions for DBAs to Consider

Page 7: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

7

Cloning Terminology

A clone is an exact replica– Clone an IMS system by volume

– Clone an IMS database by data set

IMS system cloning and database refresh– The act of replicating the data, making the replica accessible, and then using the replica in lieu of the

original data

IMS system cloning automation– Clones a complete IMS system including all its databases

IMS database refresh automation– Refreshes specific databases

Page 8: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

8

Use Cases for Cloning IMS Systems or Refreshing IMS Databases

Offload business reporting to an IMS clone– Improve production performance

– Run pointer checker utility from the clone

Create or refresh test, development, or quality assurance environments – Development refresh

– To stage data-warehouse loads

Aid in problem determination– Troubleshoot a production problem

– Test new releases of IMS

– Apply maintenance and verify integrity before applying to production

Refresh databases into a previously cloned IMS skeleton

Page 9: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Why is Test Data Management Important?

Improving business critical applications

– Improve application reliability

– Increase functionality

– Shorten time to market

– Reduce development and testing costs

Sharing resources and skills for development and testing

– Setting up a test environment usually involves:

• Systems programmer

• DBA

• Application developer

• Tester

– Test databases are typically shared

Meeting technical requirements

– Application testing done in a production replica

– Technical changes tested before production implementation

Meeting regulatory requirements

– Protection of personal or sensitivity data

9

Page 10: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

10

IMS System Cloning Today

Production LPAR

User

Catalogs

IMS001

IMS System

Data Sets

IMS002

Application

Libraries

IMS003

RECON

Data Sets

IMS004

Data Base

Data Sets

IMS005

Test LPAR

User

Catalogs

IMS001

IMS System

Data Sets

IMS002

Application

Libraries

IMS003

RECON

Data Sets

IMS004

Data Base

Data Sets

IMS005

Page 11: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

11

IMS Database Cloning Today

Production IMS

Application

Libraries

RECON

Data Sets

Data Base

Data Sets

IMS System

Data Sets

Test IMS

Application

Libraries

RECON

Data Sets

Data Base

Data Sets

IMS System

Data Sets

Unload Load

Image

Copy

Recovery

Index Rebuild

???

z/OS

Page 12: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

What is Fast Replication?

An instant copy of a volume/data set at a specific point in time

Data Movement (CPU and I/O) offloaded to storage processor

– Frees up resources on host processor

A relationship is established between a source and a target -> Pair

– Geometrically similar devices

Consistency Groups

– Group of volumes copied at exactly the same point in time while maintaining the order of

dependent writes

12

Page 13: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Variations of Fast-Replication

NOCOPY

– Data read from source

– COPY and WRITE occurs on source updates

– Useful for situations where data only needed for temporary amount of time

BACKGROUND

– COPY from source may occur on reads to target

PERSISTENT

– Relationship between source and target maintained

INCREMENTAL

– Only changed tracks copied

– Requires PERSISTENT relationship to be maintained

Space Efficient or Virtual Devices (available 2007)

– IBM Flashcopy SE

– EMC Symmetrix Virtual Devices

– Reduces disk space for target devices

– Mappings or pointers to source data maintained in storage processors

13

Page 14: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Space Efficient Usage Economics Enable Many Clone Copies

14

The Snap target is accessible when the copy session is activated

The first time a track on the source device is written to: – Original data on the source device is copied to a save device (pool)

– Pointer on the VDEV device is changed to point to the save device

– The host write is then written onto the track of the source device

The track on the source device is then updated

Unchanged data stays in place on the source device

Source Save Device

Writeto

Track

VirtualDevice

OriginalTrackVDEV

Page 15: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

15

Fast Replication Data Copy Options

Volume based fast replication options for IMS

system cloning– FlashCopy (IBM,EMC,HDS)

– SnapShot (IBM,STK)

– TimeFinder/Clone Volume Snap (EMC)

– TimeFinder/Snap (EMC)

– Mirror processes•PPRC (IBM,EMC,HDS)

•TimeFinder/Mirror, SRDF (EMC)

•ShadowImage HUR (HDS)

Data set based fast replication options for IMS

database refresh– Data Set FlashCopy (IBM,EMC,HDS)

– Data set SnapShot (IBM,STK)

– TimeFinder/Clone Data set Snap (EMC)

Fast copy processes offloaded to the storage processor

No host CPU or I/O resources

Fast Replication

Commands from

z/OS

Page 16: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

16

Host Based Data Copy Options

Volume copy options for IMS system cloning– TDMF (IBM)

– FDRPAS (Innovation Data Processing)

– DFSMSdss (IBM)

– FDR (Innovation Data Processing)

Data set copy options for IMS database refresh– Any traditional data set copy processes

Data copy processes use host based CPU and I/O facilitiesSlower than storage-based fast replication

Host-based

Copy Process

Page 17: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

17

Challenges to Data Access On the Same or Shared LPAR

IMS system cloning is best done using volume replication

– Large amounts of data

– Volume replication much faster than data set replication

• No allocation

• Assumes volumes somewhat utilized

Volume data is replicated fast and easy but…– How do you access the cloned data?

Inherent Problems:– VOLSERs may have the same volume names as the source

– Volume VTOC, VTOCIX and VVDS may be the same or different

– Data set names are the same on source and target volumes

– If you want to access the data from a common LPAR, how do you access the data?

Page 18: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

18

SYS1.VTOCIX.PIMS01

VTOC

SYS1.VVDS.VPIMS01

Source

PIMS01

SYS1.VTOCIX.PIMS01

VTOC

SYS1.VVDS.VPIMS01

Target

PIMS01 (offline)

Result:

1. Source PIMS01 keeps its original VOLSER on the target volume

2. VTOC, VTOCIX, and VVDS reflect PIMS01

3. The target volume is offline because it has the same VOLSER as the source. It could be

brought online on another non-sharing system to use the clone

4. Target volume can‟t be used on the same or shared disk LPAR without re-labeling the

volume.

Challenges to Data Access On the Same or Shared LPAR - Volume ID Issues

Page 19: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

19

SYS1.VTOCIX.PIMS01

VTOC

SYS1.VVDS.VPIMS01

Source

PIMS01

SYS1.VTOCIX.PIMS01

VTOC

SYS1.VVDS.VPIMS01

Target

TIMS01 (online)

Result:

1. Source PIMS01 becomes new VOLSER TIMS01 on the target volume

2. VTOC, VTOCIX, and VVDS reflect PIMS01

Challenges to Data Access On the Same or Shared LPAR - Volume ID Issues (2)

Page 20: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

20

A.DSN1

A.DSN3

A.DSN2

VTOC

Source

PIMS01

Target

TIMS01

VTOCIXVVDS

ICF

User

Catalog

A1.CAT

A.DSN3

A.DSN2

A.DSN1

VTOCVTOCIX

VVDS

Result:

1. Data sets on the volume are copied, but keep their original name

2. Only the source data sets are cataloged; even if the catalog is on the cloned volumes, it

isn‟t connected to the system‟s master catalog

Challenges to Data Access Data Set Name and Cataloging Issues

Page 21: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

21

Solution – IMS Cloning Tool

— the Key to Data Access

– Provides access to data sets on target volumes

• Renames the VTOC, VTOCIX, and VVDS to match the target volume

• Renames and catalogs all data sets to a new HLQ

• Adjusts target IMS system to accommodate and accept the cloned data

– Solution to enable the use of the data sets on the target volumes on the same image

• No more complicated cloning

• No more requirement for a separate image

IMS Cloning Tool Provides Data AccessOn the Same or Shared LPAR

Page 22: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

22

Production Database

IMS

CloneIMS

Cloning Tool

Cloning

Automation

SourceDatabase

Volumes

CloneDatabase

Volumes

Performs IMS system cloning automation– Takes an existing IMS system (complete installation and

system generation process completed) and creates a new, or cloned, IMS system from it without having to repeat the entire installation and system generation processes

– Simplifies IMS cloning processes– Reduces cloning time and administration costs

Leverages fast replication facilities to clone data– Data can be cloned while online or offline

Performs rapid volume reconditioning and data set renaming on cloned volumes to solve the data access challenges

– Target volumes retain their target volume label– Renames the VTOC, VTOCIX, and VVDS to match the

target volume – Renames and catalogs all data sets to a new HLQ

Adjusts target IMS to accommodate and accept the cloned data

– IMS RECONs, PROCLIB, JOBS, JCL, MDA members

Clone IMS Systems Using IMS Cloning ToolIMS System Cloning Automation

Page 23: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

23

Pre-IMS System Cloning Steps One Time Set Up - Target IMS System

User creation of target subsystem

– Define target IMS to zOS

– Determine what components and configuration options you want

– Build dependent region PROCs

• Create target PROCLIB members in source PROCLIB

– Build IMS.JOBS members

• Create target IMS.JOBS members in source IMS.JOBS data set

By default, because the clone is an exact replica of the source, the clone

would have the same components as the source

Page 24: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

24

IMS System Cloning Steps

Production IMS„Source‟

IMSClone

Target IMS

6

Start cloned IMS

1 IMS volume

selection

2

Volume copy3

A. Suspend IMS

B. Consistency

Group

4 Resume IMS if 2A

7

Rename 5

IMS

Source

Database

Volumes

Clone

Database

Volumes

Update cloned IMS systems data

Page 25: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

25

Status of In-Flight Transactions

Status of In-Flight Transactions

– An online cloning solution often results in transactions in flight

– These in-flight transactions, cloned to the target system result in the same

target system action that would happen on the source system if it were to have

died at that same time and then restarted

– The target restart is essentially an emergency restart of a failed system

Page 26: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

26

Cloned IMS System Updates

RECONs data sets

– Data set names, IMS subsystem IDs, and VOLSERs are updated in the following

RECON records: header record, database data set records, online log records, and

back-out records

Optionally, the following RECON records are updated if they were on volumes that

were cloned:

– Image copy records, change accumulation records

– System log data set (SLDS) records

– Recovery log data set (RLDS) records

Page 27: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

27

Cloned IMS System Updates

IMS PROCLIB and JOBS and user JCL libraries

– New values for IMSID, VOLSERS, and data set names in the JCL members within these

libraries

MDA (MVS Dynamic Allocation) members for databases or system data sets, are

updated to reflect the new data set names. Updates include:

– The RECON data sets

– Online Log data sets (OLDS)

– Write-ahead data sets (WADS)

If IMS data sharing is involved, each additional IMS data sharing member is updated

Page 28: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

28

IMS Environments

IMS Active– IMS offline

– IMS online

• Hardware Consistency

• Software Suspend

• ERE for transaction consistency

Data Sharing– Many to less members

– Data sharing to non-data sharing

– Data sharing members may need to be restarted prior to being removed

Page 29: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

29

IMSSystem Volumes

IMS

Target

IMS Clone

Volumes

IMS

Source

IMS Cloning Tool

Database Refresh

Automation

Performs automated IMS database refresh operations

– Fast refresh of IMS databases– IMS DB support (FF, HALDB, DEDB)

Verifies source and target database compatibility

IMS data copied using storage-based data set fast-replication

– Target takes up the same amount of space as the source

Performs target system meta-data management– Updates DBRC information for target databases

Refresh IMS Databases Using IMS Cloning ToolDatabase Refresh Automation

Production Database

Page 30: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

30

Refresh IMS Databases Using IMS Cloning Tool

Production IMS„Source‟

IMSTarget

Target IMS

8

2

Source Job

7

Target Job

Create target IMS databases

if they do not exist

Data Set Copy

Process

Start Target Databases

Update DBRC

for Target DatabasesDatabase Selection

Stop Source and

Target IMS

databases

Start Source IMS databases

3 Compatibility Check

4

5

6

1IMS

Page 31: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

31

IMS Database Refresh to a Cloned IMS Database Skeleton

IMS

IMS System and Database Volumes

IMSClone

2 Perform IMS System

Cloning Automation

Using System Data

Set Volumes Only

5

Refresh IMS

Database A

3 Create Cloned IMS

System Without

Cloned Database

Data

4

6 Run Offload

Application on

Refreshed IMS

Databases

Production IMS„Source‟

Refresh IMS Databases

Target IMS

IMS

System

Data

Sets

IMS

Skeleton

System

1 IMS System

Volume

Selection Refresh IMS

Database B

A

B

A

B

Page 32: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

32

Database Selection

Specify database names– Finds the IMS subsystem name and whether it is active

– Finds the source and target databases and indexes

– Determines data set names for each database and index

– Verifies their existence

IMSDBREFRESH -

IMS-SSID(ssids,ssidt) -

DBD(dbdname1,dbdname2) -

INDEXES(Y) -

DATA-MOVER(PGM(ADRDSSU),FASTREP(PREF)) –

AUTO-START-SOURCE-DB(Y) -

AUTO-STOP-TARGET-DB(Y) -

AUTO-START-TARGET-DB(Y) -

GLOBAL NOFEOV

Page 33: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

33

Compatibility

Verify compatibility between the source and target IMS– Gets attributes of IMS databases and indexes from source and target IMS RECON, ACBLIB, and

MDALIB data sets

Object type Attributes that must be identical for source and target object

Database Type, Access Method, Number of segments, data set groups, Blocksize,

Randomizer Parameters

Partitioned Database Number of Partitions, Partition Selection Routine, Keystrng

Segment Length, Key start and length, Uniqueness, Pointers, Edit Routine

Index CIsize, record length, DDATA, source and target segment codes, Sparse

routine

Partitioned Index Number of Partitions, Keystrng

Page 34: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Multi-threaded Data Set Copy

ADRDSSU PARALLEL Option

DSS_COPY_COMMANDS

– Specifies number of COPY tasks

DSNS_PER_COPY

– Specifies number of data sets to copy per task

34

ADRDSSUPARALLEL

mode

Page 35: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Test Data Management

Optim reduces and scrubs data from IMS, DB2, VSAM, and sequential files

Cloning Tool creates target environments, refreshes databases and tablespaces

35

IMS data

VSAM and sequential

DB2 data

Cloning

ToolOptim

IMS/DB2Target A

IMS/DB2Target A

IMS/DB2Target A

IMS/DB2Target A

Reduceddata

Page 36: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

36

DBA’s have to balance the value IMS pointer checker utility analysis provides with

the adverse affect it can have on running production applications

The questions that database administrators must ask are: – What IMS pointer checker utility functions need to run?

– How often do they need to be run?

– How much impact will the utility functions have to production data?

– How can the impact to production data be reduced?

Running pointer checker on the clone preserves the availability and performance

of running IMS applications

Run Pointer Checker Utility on the Clone

Page 37: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

37

Copying HALDB databases can be complex and can affect database

availability

– IMS Cloning Tool makes copying HALDB databases fast and easy. • Uses storage-based fast-replication to copy the data instantly

• Updates the target IMS RECON to reflect the same partition and reorg numbers as

the source RECON

• Primary and secondary indexes and ILDS can be copied to eliminate index and ILDS

rebuild time

• Manages partition number and gap sequencing

Copying HALDB Databases

Page 38: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

38

IMS Cloning Tool Session Summarization

IMS Cloning Tool simplifies and automates database administration tasks• Integrates and coordinates database and storage activities

Promote high availability solutions• IMS Cloning Tool clones IMS systems fast and effectively

• IMS Cloning Tool provides fast and effective refresh operations

Integrates storage-based fast replication• Provides fast and non-intrusive cloning operations

Offloads copy to the storage processor– Reduces host CPU and I/O resource utilization

Page 39: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

Sources and Additional References

Simply and Improve Database Administration by Leveraging Your Storage System

– By Ron Haupert, Rocket Software

– http://www.mainframezone.com/it-management/simplify-and-improve-database-

administration-by-leveraging-your-storag

IBM System Storage DS8000 Series: IBM FlashCopy SE

– http://www.redbooks.ibm.com/redpapers/pdfs/redp4368.pdf

EMC Timefinder/Snap: Multiple Snaphots with Oracle 11g Databases

– http://www.emc.com/collateral/software/white-papers/h5635-timefinder-snap-multiple-

snapshots-wp.pdf

Using Hitachi ShadowImage and Hitachi TrueCopy software with the IBM DB2

Universal Database Enterprise Server Edition Version 8.1

– http://www.hds.com/assets/pdf/wp_150_db2.pdf

39

Page 40: Dfw   ims cloning-tool_dfwug

© 2010 IBM Corporation

© 2010 Rocket Software

Information Management

40

Q & A