SAC01 Archive with TSM for Compliance.pdf

8/10/2019 SAC01 Archive with TSM for Compliance.pdf

1/50

2008 IBM Corporation

Archive with TSM forCompliance and Noncompliance

Harley PuckettStorage Consultant

Session ID: SAC01


2/50

2


Archive with TSM for Compliance and Noncompliance



New Enterprise Forum: Information Infrastructure - IBM System Storage and Storage Networking Technical Symposium

15/08/2008

Disclaimers

IBM customers are responsible for ensuring their own compliance with legalrequirements. It is the customer's sole responsibility to obtain advice of competentlegal counsel as to the identification and interpretation of any relevant laws and

regulatory requirements that may affect the customer's business and any actions thecustomer may need to take to comply with such laws.

IBM does not provide legal advice or represent or warrant that its services or

products will ensure that the customer is in compliance with any law.

The information contained in this documentation is provided for informational

purposes only. While efforts were made to verify the completeness and accuracy ofthe information provided, it is provided as is without warranty of any kind, express orimplied. IBM shall not be responsible for any damages arising out of the use of, orotherwise related to, this documentation or any other documentation. Nothingcontained in this documentation is intended to, nor shall have the effect of, creatingany warranties or representations from IBM (or its suppliers or licensors), or alteringthe terms and conditions of the applicable license agreement governing the use ofIBM software.


3/50

3






15/08/2008

The Impact of Growth is Being Felt

Replicated data

Unstructured dataTraditional Structured data

Structured data growing at 32%(databases for transactional

workloads)

Unstructured data growing at 63%

(such as user files, medical images,web and rich media content)

Replicated data growing at 49%

(including backup, archive, business

analysis, discovery and businesscontinuance)

Sources: IDC worldwide enterprise disk in Exabytes from

Changing Enterprise Data Profile, December 2007

25.0

20.0

15.0

10.0

5.0

Exabytes

Information Infrastructure is doubling every 18 months


4/50

4






15/08/2008

Common Challenges with File System Growth

Contributors to Growth

New types of digital content (ie. Blogs, video, Images)

Richer content results in larger files Ease of creating and sharing digital content

Exponential growth in number of files

Regulatory requirements

Keeping information longer

Effects of Growth

Out of space conditions

Increased management cost and complexity

Larger backup and restore windows

Longer searches and scans Resizing LUNs impacts file systems

Increased power and cooling requirements


5/50

5






15/08/2008

Deconstructing the Symptoms of Growth

When trying to identify growth factors, it isimportant to understand that often there is

a multiplying effectData containers can be files, folders,

databases, volumes, disk arrays, tapelibraries, drives, cartridges

Average per container can be bytes, watts,cost, floor space, or hours

Number of Containers (35% growth)

Average

percontainer(25%g

row

th)

63%Gro

wth


6/50

6






15/08/2008

The Two Faces of File Systems Archiving

Space Management

move old, inactive files

to less expensivestorage, resulting inreduced backupwindows and reducedstorage costs

File SystemsTivoli Storage Manager (TSM) HSM for

Windows and Space Management for Unixand Linux, TSM Archive Client, SSAM Client Data Retention

Protect your data for thelong term with non-

erasable, non-rewriteablestorage solutions anddemonstrate complianceto regulations


7/50

7






15/08/2008

Definition of Backup and Archiving

Backup is for recovery from hardware failure and/or recentdata corruption

Archive is for space management and long term retention

Archiving is an intelligent process for managinginactive or

infrequently accessed data, that still has value, while providing theability to preserve, search and retrievethe information during aspecified retention period.


8/50


9/50

9






15/08/2008

IBM System Storage Archive Manager Overview

System Storage Archive Manager (SSAM)

Is a special TSM server

Is an archive/retrieve only solution Archive: copy data or file from client to preserve for later use

Facilitates compliance with regulatory requirements

Has special data retention protections

Accepts data from TSM client API or the SSAM client

SSAM client = Archive only version of TSM B/A client

SSAM performs storage management

Manages data placement based on policies

Manages migration of data within storage hierarchy


10/50

10






15/08/2008

IBM System Storage Archive Manager Overview

SSAM is licensed by data capacity rather than number of processors

Primary storage pools only

System Storage Archive Manager provides archive management for IBMs DataRetention 550 (DR550)

TSM support for WORM media, including Network Appliance SnapLock

EMC Centera

3592 WORM tape

LTO WORM tape


11/50

11






15/08/2008

System Storage Archive Manager (SSAM) clientnodes and client software

A Client node is a Application server, Workstation, or File server registered with

the SSAM server TSM client software is installed on the client node

Client software:TSM API client:

Archive function calls usable with SSAM Open API that allows business applications to use the server functions

Content Manager and FileNet are example API applications

TSM Backup/Archive client:

Users can archive files for long-term storage, optionally delete, and retrievethem when necessary

Supported platforms:http://www-306.ibm.com/software/tivoli/products/storage-mgr/platforms.html


12/50

12






15/08/2008

TSM Policy Management Concepts

Domain 2

TSM ServerPolicy Domain 1

Backup Copy Group

Archive Copy Group

HSM Policy

Policy Domain 2Backup Copy Group

Archive Copy Group

HSM Policy

Domain 1 Centrally defined policesCentrally defined polices

What data?What data?

Where to store it?Where to store it?

How long to keep it?How long to keep it?

FileFile--Level granularityLevel granularity

SSAM only uses the Archive PolicySSAM only uses the Archive Policy

DB

Disk pools

Tape pools

Optical pools


13/50

13






15/08/2008

Policy Domain A set of policies and a group of nodes managed by those policies as

defined by the policy sets. A node (TSM client) may only be defined to

one policy domain (per TSM server).Policy Set A collection of Management Class (MC) definitions.

A Policy Domain may contain a number of Policy Sets.Only one Policy Set in a domain can be active at any time.

Management Class (MC) A collection of management attributes describing backup, archive, and

HSM characteristics.

A set of attributes is called a copy group, and there is a backup copy

groupand an archive copy group. They are always named Standard

TSM Policies Management Concepts (continued)

With SSAM, only archive copy groups are used


14/50

14






15/08/2008

SSAM Storage Management Concept

Data is managed as objectsby policies

Or simply a block of data from an appl.such as Content Manager

Files or directories

Each object is bound to managementclass (policy) that defines

retention period,

initial storage pool, etc.

These objects exist in the TSM storage hierarchy on disk or tape in storage

pools

Objects can be migrated (moved) to another storage pool within the storagehierarchy

freeing up storage space on higher performance devices such as disks

yielding attractive Total Cost of Ownership (TCO)

Objects can (and should) also be copied to copy pools


15/50

15






15/08/2008

SSAM Data Retention Protection

SSAM includes the following capabilities

Archive object protection

Archive object cannot be deleted until retention period is expired

Event Based retention policies Allows starting retention period of objects based on an event

Deletion Hold / Release

Hold overrides retention time countdown

Hold & release controlled by API or SSAM/BA client

Following operations do not delete objects:Client request to explicitly delete a object

API or SSAM/BA client

SSAM administrator commands:

DELETE FILESPACE

DELETE VOLUME DISCARDDATA=YES

AUDIT VOLUME FIX=YES

Additional Data Retention Protection:Decreasing retention time of archived objects is not possible

Deleting retention policies (management classes) is not possible


16/50

16






15/08/2008

Data Retention Protection

Disallows explicit data deletion, prior to the retention criteria, through TSMinterfaces

TSM API e.g. dsmDeleteObj

TSM Administration e.g. DELETE FILESPACE

Leverages Archive Copy Group retention policy

No changes to automated expiration process

Day 0

Data storedin TSM

AutomatedExpiration

Data deleted in

TSM

X

API Deleteissued

DELETEFILESPACE

Date Retention Criteria

Archive Copy Group Retention PolicyDeletion Hold/Release


17/50

17






15/08/2008

X

Chronological Based

Event Based with Fixed Protection Periods

Fixed Period

MinimumFixed Period

X

Event

Dispose after fixed periodfrom creation date

Dispose after fixed periodfrom event date

Day 0

Day 0

Chronological and Event Based Retention

MinimumFixed Period

Event Based with no Fixed Protection Period

XEvent

Dispose after eventDay 0


18/50

18






15/08/2008

Event-Based Archive Retention Policy

Not ApplicableRETMIN=days Clock starts when data is stored in TSM

RETMIN Minimum #days to retain

RETVERGreater of (#days to Event)+RETVERor RETMIN

Automated ExpirationEligibility

RETVER=daysorNOLIMIT

RETVER=daysor NOLIMITRETVER #days to retain after the clock

starts

RETINIT=CREATION Clock starts when data is

stored in TSM

RETINIT=EVENT Clock starts when API application invokes

dsmRetentionEvent

RETINIT When to start RETVER clock

Chronological RetentionEvent-Based RetentionArchive Copy Group

Day 0

Data storedin TSM

RETMIN

API Eventissued

AutomatedExpiration

Data deleted inTSM

RETVER

X


19/50

19






15/08/2008

Deletion Hold / Release is a client API function

Issued by clients against individual objects

Once issued, server will not delete a object until hold is released

Deletion hold can be released by client

Deletion hold has to be issued by client who owns object

Works with both Chronological and Event based retention

Some Rules:

If hold exists and event based retention period expires, object is deleted whenhold is released.

If hold is released during a retention period it will be deleted after retention periodexpires

hold status is not preserved with export

Deletion Hold/Release


20/50

20






15/08/2008

Deletion Hold and Release API Function

Day 0

Data storedin TSM

RETMIN

API Eventissued

AutomatedExpiration

Data deleted in

TSM

RETVER

X

Day 0

Data storedin TSM

AutomatedExpiration

Data deleted inTSM

RETVER

X

API Holdissued

API Releaseissued

API Releaseissued

API Holdissued

Event-Based Retention Policy

Chronological Retention Policy

New!Archive APIfunction


21/50

21






15/08/2008

Expiration of Objects for SSAM

Automated Expiration:

Inventory expiration removes the SSAM database entries for objects marked forexpiration

Server does not immediately overwrite the data in storage pools Immediate overwrite performed by Data Shredding for Random Access

Storage Pools only

Storage space reallocation or SSAM reuse of media:

Random access storage pool space reused Occurs when storage is reallocated for new data

Sequential access storage pool overwrite

Occurs when entire VOLUME is reclaimed and recycled for new data

Sequential WORM space (tape) not reclaimed


22/50

22






15/08/2008

Overwrite of Deleted Data Data Shredding

Allows disk storage pools to be designated as shreddable

Random-access (not sequential access disk (devtype=file), not tape or other media types)

Primary pools

When a data object is moved or deleted from a shreddable pool, TSM serveroverwrites the object (number of overwrites can be set up to 10)

Sensitive data objects are destroyed when deleted/moved, preventing undesirabledata discovery

a b

ab

a c

b

Shreddable Storage Pool

b b

a c

Database

Database referencesto object a

a b

ab

a c

b

Shreddable Storage Pool

b b

a c

Database

Database referencesdeleted and object

a overwritten

Object a

deleted/moved


23/50

23






15/08/2008

TSM Transparent encryption

TSM provides simple key management

Available for TSM API applications; e.g. Content Manager, CommonStore, FileNet

TSM API Client generates a random encryption key password

Per data object selected by include. encrypt and exclude. encryptcriteria

TSM API Client passes encrypted data object encryption key password to TSM Serveracross the wire

TSM Client-Server Session Key used to encrypt/decrypt

AES128 encryption used if BOTH Client and Server support it

HW Tape Drive Encryption ( IBM TS1120, TS1130 + LTO4 )Device Class specification

One encryption key for each tape volume/cartridge

TSM Server generates, encrypts, and stores the encryption key in the TSM databasealong with other tape volume metadata

TSM supports HW provided encryption modes, Application, System and Library

TSM manages key in Application mode, other modes transparent to TSM

Backup-Archive client simple key management support (TSM 5.5)


24/50

24






15/08/2008

DR550 DR2

Ranges from 8 to 244 TB of raw disk storagecapacity

Single or dual server options Choice of Ethernet connections

Synchronous or asynchronous replication

options

Tape ready

Optional TS1120, LTO-3 or LTO-4 WORMIBM DR550 DR2

IBM DR550 DR1

DR550 DR1

Ranges from .88 to 48.88 TB raw disk storagecapacity

Choice of Ethernet connections

Optional 25U rack with lockable doors

Tape ready

Optional TS1120, LTO-3 or LTO-4 WORM

IBM System Storage DR550 Family


25/50


26/50

26






15/08/2008

File level Archive Support in Standard TSM

Supported with TSM Basic Edition and TSM Enterprise Edition

TSM B/A client and Api client

Event and Chronological based retention policies

Deletion Hold

TSM support for WORM media

Network Appliance SnapLock EMC Centera

3592 WORM tape

LTO WORM tape

No Data Retention Protection Data can be deleted by user or TSM Administrator

Policies can be modified and deleted


27/50

27






15/08/2008

Archiving Can Increase Efficiency

Inactive or infrequently accessed data can be migrated to different types/

tiers of storage in order to increase efficiencies

Source: SNIA/Source ConsultingSavings can result from less costly infrastructure and improved efficiency

According to ForresterResearch 85% of productiondata is inactive

68% not accessed inlast 90 days

According to IDC ActiveArchiving Survey, 40% of

fixed content is active orfrequently accessed

Inactivedata

Activedata

Data Statistics

Source: Forrester Research, IDC

ArchiveDisk OnlineTape

InactiveData

High Duty Low DutyCycle Cycle

$/G

B

ProductionDisk

Active Data

5Years

2

0Years

3Years

1

Year

Retention Systems


28/50

28






15/08/2008

Definition of Hierarchical Storage Management (HSM)

A policy based management system for automaticallymoving inactive files to less expensive storage,resulting in reduced backup windows and reducedstorage costs

A policy based management system for automaticallymoving inactive files to less expensive storage,resulting in reduced backup windows and reducedstorage costs

The Space Management Face of File System Archiving


29/50

29






15/08/2008

250TB of storage, 25% growth over 10 years

Tape

$946,405

$7

$3.5

$0

Millions

SATA Disk

$6,365,950

Blended Diskand Tape

$2,255,346

Hardware

Prod + DR Carts

Maintenance

Power & Cooling

Floor space

10 year TCOBlended Tape and Disk Best Practices

Source: IBM TCO study for System Storage DR550, 2007

IBM Blended Disk and Tape Solutions Lower TCO

Consider the long-term costs of

ownership

High capacity SATA disk canoffer lower cost access toonline data

Tape is less expensive and

consumes less energy, but isnot often the best choice foronline access to data

A blended solution provides

online access to the mostrecent data, with longer term

storage on energy-efficient

tape


30/50

30






15/08/2008

TSM Space Management

Centralized point of control forHierarchical Storage Management

(HSM) File system selection

Effectiveness reporting

Automated management

Policy-based file selection

On-demand scheduling

Transparent recall

Platforms Supported

AIX, HP-UX, Linux-x86, Solaris

Windows 2003 NTFS v5


31/50

31






15/08/2008

TSM HSM the Basics

File Systems on Disk

TSM ServerTSM HSM Client

(File-/Appl.-Server)

Local or SAN-attached

Local or SAN-attached

LANor LAN-free

migrate

recall

Near line Storage

send with or w/ocompression


32/50

32






15/08/2008

TSM Storage Pool Hierarchy

Migration

Migration

Reclamation

Asynchronous

Copy

Storage Pool Hierarchy

Copy Pool

Synchronous (simultaneous)

Copy

Store

Client

TSM supports hundreds of storagedevices (disk, tape, optical, DVD)

Server storage is organized into storagepools, each consisting of like devices

Hierarchical organization of storage poolsexploits attributes of each device type(cost, performance, ability to removemedia)

Storage pool hierarchy supports automatic,policy-based management of stored objects

Initial storage of objects in specified pool Automatic migration to lower pools in hierarchy Automatic space reclamation Copying of data for disaster protection

(synchronously with initial store orasynchronously)

Migration of data to new storage technologies

DB

TSM database tracks location

of files as they are copied or

moved in the hierarchy

N E i F I f i I f IBM S S d S N ki T h i l S i


33/50

33






15/08/2008

SAN

SunSolaris

HP/UX Linux(Intel)

IBM AIX

LAN

FCP

TCP/IP Others

Any disk

including low cost S-ATA storage

Any tape

Windows 2003

IBM TSM HSM for Windows IBM TSM for Space Management

TSM: HSM - Managed Environments

New Enterprise Forum: Information Infrastructure IBM System Storage and Storage Networking Technical Symposium


34/50

34






15/08/2008

Overview of Client HSM

The Tivoli Storage Manager product family provides two HSM products

IBM Tivoli Storage Manager for Space Management

AIX (JFS2, GPFS), Linux (GPFS), Solaris (VxFS), HP-UX (VxFS)

IBM Tivoli Storage Manager HSM for Windows

Windows 2003 Server, Windows 2003 Enterprise Server (32 Bit), NTFS V5

HSM client automatically migrates files from on-line to near-line storage

(typically from disk to tape), based on policy settings

Small stub filesare retained on disk, appearing as the original files, thusensuring transparency of HSM for user applications

Stub files contain information pointing to corresponding entry in TSM

server DB

Migrated files are recalled automatically back to disk as required

2008 IBM C ti 2008 IBM C tiNew Enterprise Forum: Information Infrastructure IBM System Storage and Storage Networking Technical Symposium


35/50

35






15/08/2008

Migration policy on UNIX

Primary selection criteria

Age and/or size

Weight factors allow for setting priorities

Exclude small files from becoming eligible at all (minmigfilesize)

Include/exclude statements allow directory and file selection

High Threshold defines when to start automigration The maximum allocated file system capacity

Low Threshold defines when to stop automigration

Pre-migration percentage (optional)

Additional data migrated after Low Threshold

No stub created

Frees up online storage space faster during automigration

2008 IBM Corporation 2008 IBM CorporationNew Enterprise Forum: Information Infrastructure - IBM System Storage and Storage Networking Technical Symposium


36/50

36






15/08/2008

UNIX



37/50

37






15/08/2008

Migration policy on Windows

An ILM type approach based on value of file

Does not avoid out of space conditions

Configure one or more migration jobsA migration job consists of a Path or Directory name

Include/exclude filters can be added for

Known file groups or specify file name extensions

File size and age

Jobs can be executed periodically by starting them from the scheduler

Supports Recall Quota

Controls amount of recalls per user/group in a specific time period

All configured through the GUI



38/50

38





p y g g g y p

15/08/2008

Windows



39/50

39

p


p

2008 IBM Corporation15/08/2008

Windows Migration Changing of File Properties

Resident File

Offline Bit

Size on disk is one Cluster

Time stamp unchanged

Overlay Icon

Migrated File



40/50

40 Archive with TSM for Compliance and Noncompliance 2008 IBM Corporation15/08/2008

Interfaces to the TSM server

TSM

Server

File System

File U

TSM HSM

Client

UNIX File Server

File System

File W

TSM HSM

Client

Windows File Server

TSM API

library

HSM

Verbs

TSM Server

Storage Pool(SPACEMGMT)

Storage Pool

(Archive)

Unique Object ID

Stub U

File U

different

interfacesused internally

Pathname Qualifier(FS/HL/LL)

Stub W

File W

Recommended toset archive retention

forever to keepdata integrity!



41/50


Backup integration: UNIX and Windows

UNIX File Server Restore of stubs and complete files

through the same interface

Number of retained versions

controlled by VEREXISTS option

Inline copy can avoid network traffic

Windows file Server Restore of stubs and complete files

through the same interface

Number of retained versions controlled

by VEREXISTS option

Both ways of backup integration allow restoring stubs for faster restore of a file

system

Older versions of file contents are kept on the TSM server

Best practice is to require file be backed up prior to migration

New inV5.5



42/50


Backup integration (backup)

TSM

Server

File SystemTSM HSM

Client

UNIX File Server

File System

Windows File Server


Storage Pool

(Archive)

TSM Backup

Client*

TSM HSM

Client

TSM BackupClient

Storage Pool

(Backup)

Stub U

File U

File U

File W

File W

File W

Inline copyknow the link!

(no network traffic)

TSM Server

Stub W

Stub W



43/50


Backup integration (restore)

TSM

Server

File SystemTSM HSM

Client

UNIX File Server

File System

Windows File Server


Storage Pool

(Archive)

TSM Backup

Client*

TSM HSM

Client

TSM BackupClient

Storage Pool

(Backup)File W

TSM Server

restore

File U

File U

Stub U restore(migstate=yes)

Recreating uniqueobject ID

Stub W

Stub W File W

File W



44/50


Backup integration (versions)

TSM

Server

File SystemTSM HSM

Client

UNIX File Server

File System

Windows File Server


Storage Pool

(Archive)

TSM Backup

Client*

TSM HSM

Client

TSM BackupClient

Storage Pool

(Backup)File W

Stub U

File U

File U

File W

File W

TSM Server

Stub U*

File U*

File U*

copygroup(verexists=2)

U*/W*

file U/W contains

now modified

contents and

gets migrated

again

Stub W*

Stub W*

copygroup(verexists=2)



45/50


Value of IBM TSM HSM Archive Solutions

IBM offers complete end-to-end HSM solutions for Windows and Unix file

systems

Policy based automation reduces manual effort

Identify and move low-activity or inactive files to a hierarchy of lower-cost storage

Eliminate manual file system clean-up activities

Speed file-server recovery times by restoring files to migrated state, deferring data

restore until neededSupport for transparent media migration for longevity and cost-effectiveness

Tiered information infrastructure reduces cost

Blend disk and tape solutions for optimum efficiency and performance

Ensure information integrity with flexible and scalable non-erasable, non-rewriteablestorage

Secure and protect information with encryption and automated retention policies



46/50


For More Information

Tivoli User CommunityAn active and lively community for Clients, Business Partners, and IT professionals.Free membership provides you with valuable resources, tools and networkingcapability. Log on to www.tivoli-ug.org or visit the ped in the IBM Pulse Expo

Tivoli TrainingIBM offers technical training and education services to help you acquire,maintain and optimize your IT skills. For a complete Tivoli Course Catalog andCertification Exams visit www.ibm.com/software/tivoli/education

Tivoli Services

With IBM Software Services for Tivoli, you get the most knowledgeable expertson Tivoli technology to accelerate your implementation. For a complete list ofServices Offerings visit www.ibm.com/software/tivoli/services

Tivoli SupportIBM Software Premium Support provides an extra layer of proactive support,

skills sharing and problem management, personalized to your environment.Visit www.ibm.com/software/support/premium/ps_enterprise.html



47/50




48/50


References Tivoli Storage Manager for Space Managment

TSM for Space Management on the Web (also Demo available): http://www-306.ibm.com/software/tivoli/products/storage-mgr-space/

http://www-306.ibm.com/software/tivoli/library/demos/dem-stg-mgmt-hsm.html?S_CMP=rnav

TSM for Space Management Client Manual (Order Number: SC32-0148-01): http://www.elink.ibmlink.ibm.com/publications/servlet/pbi.wss?SSN=08DBJ0038813141909&FNC=

PBL&PBL=SC32-0148-01PBCEEBO200040677&TRL=TXTSRH

TSM for Space Management platform support information:

http://www-306.ibm.com/software/tivoli/products/storage-mgr/platforms.html

TSM for Space Management IBM Tivoli Enterprise Space Management Console: http://www-306.ibm.com/software/tivoli/resource-center/storage/code-ent-con-stor-mgr-space.jsp

TSM for Space Management white paper (will be updated 08/08): http://www-1.ibm.com/support/docview.wss?uid=swg27002498

RSCT download http://ftp.software.ibm.com/storage/tivoli-storage-

management/maintenance/client/v5r5/Linux/LinuxX86/HSM_RSCT/ http://www14.software.ibm.com/webapp/set2/sas/f/csm/download/home.html


49/50


50/50

Documents

SAC01 Archive with TSM for Compliance.pdf