Backup Methods For a Hot Site Dieter W. Storr Los Angeles Times 23 August 2005

Backup Methods For a Hot Site

Dieter W. Storr Los Angeles Times

23 August 2005

23 August 2005 Dieter W. Storr -- www.storrconsulting.com

B/R Methods Existing Backup Method Experiences Mirroring or Replicating Fast Copy of Data Proposals and Costs Future Technology Lessons learned

Existing Backup Method

•From disk (databases)

•Copy to

•3490 / 3590-1 / VTS

•Then, copy to

•3590-1 (cartridge)

ADABAS 6.2.2 Back-up at LA Times

W e e k l y 2 1 : 0 0 - 2 1 : 3 0

A D A P n B K F O n l i n e S A V E

A D A P n P L C F E O F P L

A D A P n P L C P L O G S w i t c h

D F D S S F u l l V o l u m e B a c k - u p

A D A P n B K O C o p y O n l i n e S A V E s

B R M / A B A R S S e v e r a l J o b s

P D S , G D G s , e t c . D i s k P o o l

2 : 0 0 3 : 0 0 8 : 0 0 - 1 1 : 0 0

P i c k - u p b y R e c a l l

2 1 : 3 0 - 1 : 1 5

J o b 3 4 9 0 t a p e s ( 3 5 9 0 - 1 )

D A P 1 B K O 2 ( 1 ) A D A P 2 B K O 3 5 ( 1 ) A D A P 3 B K O 1 6 ( 1 ) A D A P 4 B K O 8 ( 1 ) A D A P 5 B K O 4 ( 1 ) 6 5 ( 5 ) D F D S S / o n e t a p e p e r v o l u m e

5 9 ( ? )

B R M / A B A R S 2 2 ( ? ) T O T A L 2 1 1 ( ? ) ( O n l y f o r A D A B A S )

S t a t u s : 2 7 J a n 2 0 0 5

B/R Methods

Source: http://www.drj.com/articles/spr02/1502-07.html

“Companies that relied on

tape or on third-party provider

found in many cases they had

difficulty meeting their recovery

time objectives.”

B/R Methods

Source: 15 Apr 2004 | SearchSecurity.com

“Flaws in tape-based data backup may be

leaving enterprises without key information

and could lead to legal exposure under

emerging laws such as Sarbanes-Oxley, say

data backup and recovery experts. “

B/R Methods In a survey of 500 IT departments completed …

found that as many as 20% of routine, nightly backups fail to capture all data.

40% of IT managers had been unable to recover data from a tape when they needed it

More than 23% sought to use data stored on tape backups more than 20 times in a year

Source: 15 Apr 2004 | SearchSecurity.com

B/R Methods

Are tapes really so bad?LA Times experiences?

Tape Problems

1 November 2002:

Six tape drive errors

Tape Problems

24 March 2003:

Only two channel paths per

tape controller were provided

Slow restore time

Tape Problems

5 October 2003:

3590 tape drives were not

defined to DFSMS (SMS)

ADABAS restore and

application test cancelled

Tape Problems

6 December 2003:

VTS problems with GDG

datasets

End-user functions

couldn’t be tested

Tape Problems

5 August 2004:

Restore jobs had to wait for an input

tape that was being used by another

restore job

Tape Problems

30 October 2004:

Packages didn’t arrive in time,

due to a thunderstorm that

affected FedEx delivery

Major delay

Tape Problems30 October 2004:

Automated tape library experienced

unit address problems during the

restore process

Tape Problems

30 October 2004:

VTS logical tapes were not shipped

to Wood Dale (HSM level 2, SAR

level 2)

Tape Problems30 October 2004:

Confusion about

when to load DRP1

and DRP2 tapes,

before or after IPL

Tape Problems

30 October 2004:

ICIS libraries were not

backed up to tape

Application tests were not

possible

Tape Problems

8 December 2004:

Load problems

Tapes were loaded before IPL and

not after IPL

Major delay

Tape Problems

8 December 2004:

Experienced problems when

trying to restore MIG1 data,

e.G. DRADABC0 job

Major delay

Tape Problems

8 December 2004:

Recall sent by FedEx tapes to SunGard

One damaged package arrived without

tapes Restored DATA one generation back (-1)

System was generation (0)

Tape Problems

21 March 2005:

Level 2 tapes for VTS not

being sent off-site (but have

been on the list)

Application team couldn’t

test all data

Tape Problems

5 August 2005:

3590-1 cartridges ejected,

not found DSS8370W - TMS SHOWS TAPE

N00318 OUT OF AREA “DRP1”,SLOT

Time Warner employee data missing

May 2, 2005: 5:51 PM EDT

NEW YORK (CNN) - Time Warner Inc. said Monday that data on 600,000 current and former employees stored on computer backup tapes was lost by an outside storage company and that the Secret Service is now investigating.

Lost Backup Tape Held Ameritrade Client Data

Wednesday, April 20, 2005 - LA Times

… package was damaged during shipping between vendors ….. fourth tape is still missing…… The tapes may have included customers’ Social Security numbers …..

Info On 3.9M Citigroup Customers Lost

Monday, June 6, 2005 – CNN.COM

Citigroup, the nation's biggest financial services company, said that UPS lost the tapes while shipping them to a credit bureau in Texas.

Costs for Tape Backups SunGard recovery services Offsite tape storage Tape handling Shipping per test Special extra pick-ups

Yearly $150,000

Not capable to restore one day $$ ???

Last December: 2 weeks to rebuild manually (?) customer tables

Does it make sense to restore more than 2 days back ??

Example:

20 employees x $140 per day x 10 days

= $28,000

And they couldn’t work on other projects

$140 is based on $51,100 yearly income

Quantitative Risk Analysis Single Loss Expectancy

SLE = Single Loss Expectancy EF = Exposure Factor, for

example 50% or .50 AV = Asset Value, for example

$1,000,000

SLE = EF * AV

SLE = .5 x $1,000,000 = $500,000

B/R Methods

Reducing tapes

B/R Methods

Reducing tapes Stacking datasets to

3590-1 cartridges Using Delta Save Facility

from ADABAS

B/R Methods

Reducing tapes Using Forward Index

Compression (FIC) from ADABAS

Using larger block size for 3590 tapes = 256K, supported by ADABAS

ADASAV

Delta Save

changed blocks

NUCLEUS

DDPLOGR1

DATADATA

Buffer Pool Delta Log (RABN) changed RABN

ADARES

PLCOPY

DDPLOGR2

PLOG copy

DDSAVE1

DDDSIM

DSF=YES

DDSIAUS1

DSF=YES

Dual Protection Log

Extracted

Blocks

Delta Save Facility (DSF)

Delta SaveADASAV

RESTORE

DDDSIM

DSF=YES

DATADDDELT1-8

DDREST1

Full Image

Online/Offline

Online Images

extracted

from PLOG

Delta Save Facility

B/R Methods

Forward Index CompressionRochester Gas & Electric

Space savings: Normal Index: 37% - 55% Upper Index: 21% - 69%

Within an index block the part of the index value that is identical to the forward part of the previous index value is suppressed.

B/R Methods

IBM Magstar 3494 / Virtual Tape Server (VTS)

LA Times

SunGard

B/R Methods

VTS problemsLA Times: Completion code A78 RC 18 We switched from VTS to 3590-1

cartridges

B/R Methods

VTS problemsVirginia Information Technologies Agency: Ran 2003/2004 into the same problem system

completion code A78 RC 18 We … converted … to 3490/3590 physical

tapes Problem solved

B/R Methods

Disk to Disk Mirroring

Hardware Software

Replicating Software

B/R Methods – Enterprise Server

Enterprise Server

NT / 2000 / XP

Hot Site

B/R Methods – Open System

Hot Site

B/R Methods

Marty StewartDisaster Recovery Manager

AnMed Health:

“…we’d rather have a server that’s running slower than having no server at all.”

Disk Mirroring

Benefits Asynchronous disk mirroring can

provide better physical protection by supporting extended physical distances.

No loss of committed transactions in synchronous storage (mirroring/RAID) on a CPU failure

Limitations No protection from data corruption Secondary site is not guaranteed to be

transitionally consistent, in the case of asynchronous mirroring.

Client application must be re-started after failure and need to be aware of failure

Disk Mirroring

Limitations Synchronous mirroring and RAID

devices can add overhead to application performance.

Redundant/specialized high availability hardware/software can be expensive and restricted to use for backup purposes only.

Limitations Secondary copy of data is not

available for use – low hardware utilization.

Need to replicate everything on disk, no selectivity of data replication

Disk Mirroring

Example For Disk Mirroring

S/390 UNIX

12-15 miles

OC-3 link

EMC 5700

SRDF remote mirroredsynchronized

Back Up / Hot Site

SRDF remote mirroredsynchronized

Main Platform

B/R Methods Can we buy used

Enterprise Servers? Yes…..and inexpensive OP system is free for D/R

Search for “selling used mainframes,” for example:

http://www.used-line.com/fdc3236-find-dealer.htm

http://www.azure.co.uk/

Dedicated line broadband speeds and prices

T-1 - 1.544 megabits per second (24 DS0 lines) Ave. cost $400.-$650./mo.

T-3 - 43.232 megabits per second (28 T1s) Ave. cost $6,000.-$16,000./mo.

OC-3 - 155 megabits per second (100 T1s) Ave. cost $20,000.-$45,000./mo.

OC-12 - 622 megabits per second (4 OC3s) no price OC-48 - 2.5 gigabits per seconds (4 OC12s) no price OC-192 - 9.6 gigabits per second (4 OC48s) no price

Source: http://www.infobahn.com/research-information.htm

prices updated: 12 May 2005

Peer-to-Peer Remote Copy Extended Distance (PPRC-XD)

PPRC = 60 miles - PPRC-XD = continent

ESS Shark

- IBM ESS DASD - HDSalso support PPRC

ESS Shark

FlashCopy

Also see TimeFinder from EMC

External Back-up Systems

Fast Copy of Data Snapshot

No data movement A virtual copy by copying pointers

Copy Process Physical copy async. from the log. copy No impact on applic. on the original data

Fast Copy of Data Specific Hardware Required

Software works only with the hardware Work on Volume Level

Some snapshot only tools work also on dataset level

External Back-up Systems

Snapshot & Physical Copy

IBM Hardware: Enterprise Storage Server Software: FlashCopy

http://www.share.org/proceedings/sh98/data/S3087.PDF

Hardware: Symmetrix Remote Data Facility Software: EMC TimeFinderhttp://www.emc.com/interactive_center/media/timefinder/tf_noRC.html

Flash Copy

How It Works

Read / update

PhysicalBackup

SnapshotSnapshot

Read / updateRead only

Pre-defined time window

Suspend Resume

SourceData

Read only: update requests are queued

Source: SAG

ADADBS TRANSACTIONS SUSPEND,TTSYN=60,TRESUME=120

Replication

Benefits Warm standby systems can be

configured over a Wide Area Network, providing protection from site failures.

Ability to more quickly swap to the standby system in the event of failure, as backup database is already on-line.

Data corruption is typically not replicated as transactions are logically reproduced rather than I/O blocks mirrored.

ReplicationBenefits Automatic switch over for clients using

a switching mechanism, no client restart needed.

Originating applications are minimally impacted as replication takes place asynchronously after commit of the originating transaction.

The warm standby database is available for read-only operations, allowing better utilization of backup systems.

Benefits Ability to resynchronize and easily

switch back to primary system when it becomes available without loss of data.

Replication

Limitations Warm standby system will be out-of-

date by transactions committed at the active database that have not been applied to the standby.

Protection is limited to components supporting Warm Standby (e.g. DBMS data sources may be protected but file systems may not be supported).

Replication

Entire Transaction Propagator The Entire Transaction

Propagator allows for

asynchronous data

replication.

Replicated data can be

updated and

synchronized with

master data at user

specified intervals.

ADABAS Data Replication Logical dissemination of ADABAS Data to

homogeneous or heterogeneous targets Near real time propagation Event driven at the Transaction level Implemented at the Database/file level for Store,

Delete and Update commands Define Replication rules through subscriptions Minimal Impact on normal nucleus activity Strategic for Enterprise Data Sharing Replace Entire Transaction Propagator

ADABAS Data Replication

Origin

File File

Target

Tablez/OS

Image B

Server D

Image Cz/OS Image A

Possible Hot Site Solutions

Enterprise Server Los Angeles

Own Enterprise Server Hot Site

OC3OC3

Converter ESCON

Fiber Optic

Costs for Tape Backups SunGard recovery services Offsite tape storage Tape handling Shipping per test Special extra pick-ups

Yearly $150,000

Costs for Real Disaster SunGard Declaration Fee D/R Site Daily Usage Fee Office Space Daily Usage Fee Work Group Declaration Fee Work Group Daily Usage Fee LAN Bridge Declaration Fee LAN Bridge Daily Usage Fee

30 Days $475,000

Costs for Own Hot Site Used IBM Z800-0X2 Mainframe Used IBM 2105-F20 Shark Storage Used IBM 3494 Library, VTS and Tape Drives 3rd Party Next Day HW Maintenance Printer and Terminal Controller Re-location Costs 3490 Tape Drive and Controller Re-location Costs Other Costs

1st Year $520,000After 5 Years Total $735,000

Costs for Own Hot Site

5 Years SunGard = $750,000

30 Days Real Disaster = $475,000

5 Years Own Facility = $735,000

Restore Times (Min)

IPL done DB restore

IPL done 0 330 425 540 365 420 420 330 270 360

DB restore 229 129 165 221 0 104 63 90 78 136

Feb-02

Jun-02

Nov-02

Mar-03

Oct-03

Dec-03

Oct-04

Dec-04

Mar-05

Aug-05

Benefits of Own Hot Site Financial savings > $150,000

annually providing an almost 5% ROI

Reduced recovery time Reduced impact due to road and airport

closures Elimination of reliance on external vendors Mainframe and open system can use the

same facility

Grid Computing

virtual machine

virtual memory

virtual storage

virtual I/O

Grid Computing

ResourceBroker

ReplicaCatalog

Information Service

ComputerElement(s)

StorageElement(s)

HardwareSoftware

Locations

Interface

Passkey

Grid Computing

Grid Computing BBC builds distributed

grid for content sharing

(Gridcast).

62.73.167.57/publicdocs/ppt/prompeg307.pptfile:///C:/My%20Documents/Dieter/my%20presentations/prompeg307.ppt

Grid Computing For Backup?

Intra or Extra Grid?

Pull or Push?

Grid Software

http://www.gridcomputing.com/

http://www.gridforum.org/ggf_grid_understand.htm

http://gridcafe.web.cern.ch/gridcafe/animations.html

Backup Methods

Mostly used by other companiesSource: DRJ Magazine

VTS Disk to disk

Is more and more common for enterprise storage servers and AIX server technology, for example.

Source: @server Magazine

B/R Methods

Problems for other companies High third-party hot site costs, approx.

$10,000 - $70,000 per month Restore time

24-30 hours

How Far is ‘Far Enough?’(http://www.drj.com/articles/spr03/1602-02.html)

Alternate Facility

Offsite Storage

Facility

Answer = 105 miles

…so the survey

Lessons Learned (http://www.drj.com/articles/spr02/1502-07.html)

Distance is keyStreets, bridges, tunnels, airports are closed

Tape recovery is not effective All applications are critical Inconsistent back-up is no back-up at all

Lessons Learned (http://www.drj.com/articles/spr02/1502-07.html)

People-dependent processes do not suffice

Two sites are not enough People are hard to replace but

information is irreplaceable

…..we should have an excellent HOT SITE!

Backup Methods For a Hot Site Dieter W. Storr Los Angeles Times 23 August 2005

Documents

ADABAS Little Known Features Natural Conference in Philadelphia, PA October 17-20, 2006 Dieter W. Storr info@storrconsulting.com

Dieter W. Storr Efficient Usage of Adabas Replication978-3-8348-8214-1/1.pdf · Efficient Usage of Adabas Replication . Dieter W. Storr Efficient Usage of Adabas Replication A Practical

Mechamical Metallugy Dieter

storr board

Climbing Old Man of Storr, Isle of Skye, Scotland - Photos

Chart Tell 25-Yeas Storr y of Clu Businesb s - Home | MSU …archive.lib.msu.edu/tic/golfd/article/1947jun23.pdf · · 2011-11-02Chart Tell 25-Yeas Storr y of Clu Businesb s

Anthony Storr - bukla.si

mechanical metallurgy DIETER

Dieter Schlingloff Memorandum

Anthony Storr Solitude

Dieter Roth

Dieter rams

Network Dieter W. Storr January 2002

Process Modeling with ARIS Why students around the world like to learn BPM based on ARIS Dieter W. Storr – 12 May 2011

Storr - Music and the Mind

Dieter Henrich - Konstellationen

Dieter W. Storr Efficient Usage of Adabas Replication3A978-3... · My special gratitude goes out to Larry Frazin, who supported me with repli cation tests. John Donnelly, Albert Stetson,

ANTHC)NY STORR - Cabrillo Collegeewagner/WOK Eng 2/Storr - Music and the Mind.pdfANTHC)NY STORR . MUSIC AND THE MIND . Iffil . THE FREE PRESS . A Divisioll of Macmillan, ille. NEW

DIETER MARKETPLACE II · RIVER OAKS PROPERTIES GEORGE DIETER ALE ANO A DEL SOL D I E T E R MARKETPLACE 1485 George Dieter El Paso, Texas 79936 1485 George Dieter El Paso, Texas 79936

Best Practices Natural Conference in Philadelphia, PA October 17-20, 2006 Dieter W. Storr info@storrconsulting.com What it means to be a DBA