FDR/UPSTREAM the ONLY File Level z/OS Data Protection ...– Improved internal communication • Faster restores by improved changes to sorting prior to restore larger record sizes

www.share.org/sanantonio-eval

http://creativecommons.org/licenses/by-nc-nd/3.0/

FDR/UPSTREAM the ONLY File Level z/OS Data Protection Solution for Linux on z Systems. Patrick Fitzsimmons [email protected] INNOVATION Data Processing August 2, 2016

© Copyright 2016 INNOVATION Data Processing. All rights reserved.




•  Patrick Fitzsimmons Product Manager FDR/UPSTREAM for 25 years

Who I am and this session

2




Examples of Big Data…Enterprise Backup Pain Points Petabytes and ZettaBytes here we are A zettabyte is equal to 1 billion terabytes •  RFID tags – 30 billion in use •  Airline – flight from France to JFK generates 640TB of data

•  My attempt to get to Atlanta •  Data Mining – Customer Loyalty Card •  Smart Phones •  GPS •  ObamaCare •  Social Media Outposts

–  Facebook 500 TB data every day –  Twitter 12 TB per day

•  Data Warehouse –  Larger volumes are necessary to handle data growth

3



© Copyright 2016 INNOVATION Data Processing. All rights reserved. 4

Utility companies are now offering smart meters that are doing meter readings once every hour or less compared to once a month. Data is collected, then sent to the data warehouse and then mined. How many TB? What is the retention period of the collections?

Examples of Big Data… What does yours look like?




What type of data protection do you want? •  Your choices are:

–  Physical Image –  File Level –  Database –  Disaster Recovery

5




FDR

FDRSOS

FDR/UPSTREAM

UPSTREAM/SOS Or SAN EXPRESS

FICON CKD

FICON FCP

FICON CKD or FCP NAS DAS

SAN

FICON CKD or FCP NAS DAS

SAN

Image

Image

File Level & Database

File Level & Database

FICON

SAN

OSA & HiperSockets

SAN

z/OS

z/OS

z/OS or Reservoir

z/OS or Reservoir

User Data Storage Technology

Backup Type

Data Transport

Storage Server

INNOVATION’s Solution Selection Matrix for Linux on z Systems

6




What’s NEW for FDR/UPSTREAM •  Up to 25% Faster performance over 10GB & HiperSockets

–  Improved internal communication •  Faster restores by improved changes to sorting prior to restore

larger record sizes •  Synthetic Differential Merge •  UPSTREAM WEB PORTAL •  I/O Optimizer •  zEDC support

–  Known support for USS and works well in compressing data. Linux soon? Run Time & CPU time? Carefully consider.

•  VMware support

7




•  The Web Portal, using an application server and external database, provides a single web hosted environment that has the ability to monitor, control, and perform UPSTREAM actions against multiple storage servers.

•  No Need to know Profile Names. •  Can easily switch between systems. •  For end users, no need to be aware of the behind the scene backup setup. •  Besides providing Web Access, the Web Portal also has a set of data

gathering tasks running that contact backup clients and also the storage servers to obtain statistics, scan backup versions, collect data set information and history, and so on.

•  Uses an Apache Tomcat application server, or WebSphere, with a Derby database that becomes the foundation for the new Web Services provided.

•  Next version will support MySQL, SQL Server, Oracle.

8

What’s NEW for FDR/UPSTREAM UPSTREAM Web Portal




What’s NEW for FDR/UPSTREAM UPSTREAM Web Portal •  Automated client upgrades •  Edit Client Configurations •  Data Protected Storage Reporting

–  TB protected, # of Files, # of Tapes •  Administrative Roles Assigned •  Restores for Non Admin personnel •  Simple and Improved Restores •  ORACLE RMAN configuration

9




What’s NEW for FDR/UPSTREAM •  The UPSTREAM WEB Portal collects detailed information about the installed

inventory of UPSTREAM clients including version, Operating systems platform, storage server it connects to, DNS or IP address, members of any groups, etc. This compilation of information provides a consistent dashboard view to the storage administrator for any of the supported UPSTREAM clients.

10

•  Backup Host Repository(s) / Security Maintain your list of host repository(s). Setup Storage Resource(s) for the host(s) •  Backup Storage Usage View how much backup storage is in use by the entire system, each host or each client.

View/delete the backup version(s) of each client •  Client List Manage your list of all clients, add update additional information about each client. Build, view and

manage your Client Groups and Client Filters. Update a Client’s configuration, restart a Client. Initiate a Client backup and view its backup history

•  Client Upgrade Use this panel to upgrade your UPSTREAM Client(s), show Client upgrade history •  Users Create, update and delete Web Portal users. Assign to Admin Users and Restore Users their

Client Group/Filters to give them access to those Clients •  Restore Perform restore operations to a Client •  Restore Status / History Displays the status of a Restore operation, show/manage the history of Client restore(s) •  Storage Policies Setup the backup Storage policies •  System Displays various system information, shutdown Web Portal




UPSTREAM Web Portal Inventory of clients being protected

11




What’s NEW for FDR/UPSTREAM Centralized Controlled of Client upgrades

12



© Copyright 2016 INNOVATION Data Processing. All rights reserved. 13

What’s NEW for FDR/UPSTREAM Centralized Controlled of Client upgrades




Helpful Tools •  Display of unused profiles •  Web Portal provides an

address book for system administrators contacts, info

14




•  Administrators Main administrators that by definition have access to all backup clients.

•  Sub-Administrators Scoped administrators that have been granted access to administrate and delegate access to a specific set of backup clients. This is done with Groups and Filters.

•  Restore Users End Users are only granted access and do not delegate access to clients.

15




Helpful Tools •  Create groups and filters •  Groups are static and Filters dynamic

16




Web Portal Users

17




Reporting of Storage Servers Usage

18




What’s NEW for FDR/UPSTREAM •  Synthetic Differential Full Merge •  The Deferred Merge, USTMERGE processing time and

I/O are dramatically reduced using a new merge feature, the Synthetic Differential Full Merge.

•  By copying and consolidating only the most recent changed files that have been backed up since the most recent full backup (full merge or first time full), typically this is a small number of files.

•  This can greatly reduce CPU and wall clock time. •  It reduces copying and storing data that does not change very

often when creating the synthetic full backup. •  Fewer tape mounts are required, reducing I/O and elapsed and

drive time. 19




Full Merge Synthetic Technology Minimizes Data Transmission

20




8

Files

New Merged Full Backup

7

Files

Full Merge (Incremental & Directory List)

6

Files

Mon-Fri Server Incremental Backups

“B, G, L, N, S” are changed files.

B G

L N

S

5

Files

New Merged Full Backup

4

Files

Full Merge (Incremental & Directory List)

3

Files


“A, F, K, P, R” are changed files.

A F

K P

R

2

Files A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Full Merge

z/OS First Time Full or Full Backup

1

21

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z



A B C D E F G

H

I J K L M N O P Q R S T U V W X Y Z




8

Files

USTMerge

7

Files

Deferred Full Merge

6

Files


“B, G, L, N, S” are changed files.

B G

L N

S

5

Files

USTMerge

4

Files

Deferred Full Merge

3

Files


“A, F, K, P, R” are changed files.

A F

K P

R

2


Deferred Full Merge

z/OS First Time Full or Full Backup

1


A F

K P

R


B G

L N

S

Co-Located Files Co-Located Files

22




Deferred Full Merge…Process   Start with any full backup, it contains every file that currently exists on the server.   An incremental backup identifies the changed files and copies them into the incremental backup dataset. In the example above files “A, F, K, P, and R” have been changed since the full backup.   The Deferred Full Merge is specified by setting the Backup Profile MERGE option to “DEFER”. The process is as follows:

a)  Server Files changed since the last incremental backup are copied into the Deferred Full backup dataset. b)  Unchanged server files have pointers created to prior backups, they are reflected in the diagram with grayed out file names. c)  This backup can now be used to restore any and all files.

  The USTMERGE command is executed which resolves the pointers and co-locates all the prior backup on a single tape set. The final result is a complete FULL backup with a copy of EVERY file on the server at the time the deferred full was run.

WEEK TWO   The incremental cycle continues with files “B, G, L, N, and S” being changed and subsequently backed up.   Another deferred Full Merge is run

a)  Server Files changed since the last incremental backup are copied into the Deferred Full backup dataset. b)  Unchanged server files have pointers created to prior backups, they are reflected in the diagram with grayed out file names. c)  This backup can now be used to restore any and all files.

  The USTMERGE command is executed which resolves the pointers and co-locates all the prior backup data, on a single tape set. The final result is a complete FULL backup with a copy of EVERY file on the server at the time the deferred full was run.

23

4

3

2

1

7

6

5




Points Back to Unchanged Files on Master Backup


Files

USTMerge to Master

7

A B F G J L

W R

Files

Todays INCRs Deferred Full Merge

6

Files

Mon-Fri Incremental Backups

“B, G, L, R” are changed files.

“K” is a deleted file.

B G

L

5

Files

USTMerge to Master

4

Synthetic Differential Full Merge

Master Backup

A F J K

W

Files


3

J

W

Files


“A, F, K, J, W” are changed files.

A F

K

2


1

R

24




Synthetic Differential Full Merge…Process   Start with any full backup to begin the Synthetic Differential Full Merge process. This will become known as your “MASTER” backup. Its retention MUST be long enough to keep it from expiring while the later backups that are dependent upon it are in use.   An incremental backup identifies the changed files and copies them into the incremental backup dataset. In the example above, files “A, F, and K,” have been changed since the full backup.   The Deferred Full Merge is specified by setting the Backup Profile MERGE option to “DEFER”. The process is as follows:

a)  Server Files changed since the last incremental backup are copied into the Deferred Full backup dataset, as example “J and W” illustrate.

b)  Unchanged server files have pointers created to prior backups, they are reflected in the diagram with grayed out file names. c)  This backup can now be used to restore any and all files.

  The USTMERGE command is executed with the MASTER option. Only the most recent version of individual changed incremental files will be co-located onto the full. Files from the prior MASTER FULL will NOT be co-located and the pointers created in the Deferred Full Merge are copied. The USTMERGE process will end quickly. The final result is the completed Full Merge now contains the most recent copy of every file on the server that has CHANGED since the time the MASTER full was run. In this example, only “A, F, J, K, and W” are on the backup dataset. This is where the “differential” name comes from.

25

4

3

2

1




Synthetic Differential Full Merge…Process WEEK TWO

  Week 2 begins and the incremental cycle continues with files “B, G and L” being changed and subsequently backed up. File “K” has been deleted, so this is reflected.   Another Deferred Full Merge is run and any files that have changed since the last backup are copied, as example “R” illustrates above. Then pointers are created to all the other UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.   The USTMERGE command is executed with the MASTER option. Only the most recent version of individual changed incremental files will be co-located onto the full. Files from the prior MASTER FULL will NOT be co-located and the pointers created in the Deferred Full Merge are copied. The USTMERGE process will end quickly. The final result is the completed Full Merge now contains the most recent copy of every file on the server that has CHANGED since the time the MASTER full was run. In this example, only “A, F, J, W, B, G, L, and R” are on the backup dataset. Another differential has been created!

26

7

6

5




Files

USTMerge New Master Backup

A B C D E F G H I J L M N O P Q R S U V W X

Z 13

Files


12

Files


“C, D, Q” are changed files. “T” is a deleted file.

11


Files

USTMerge to Master

10

Q

Files


9

X

Synthetic Differential Full Merge continued

Files


“N, S, X” are changed files. “Y” is a deleted file.

N S

8

1

1

1

A BB F G J L N

S W X

R

C D Files

USTMerge to Master

7

A B F G J L

W R

27




Synthetic Differential Full Merge…Process continued   Week 3 begins and the Incremental cycle continues with files “N, and S,” being changed and subsequently backup and “Y” is deleted.   Another Deferred Full Merge is run and any files that have been changed since the last backup are copied, as example “N and S” illustrate. Then pointers are created to all the UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.   The USTMERGE command is executed with the MASTER option. ONLY the most recently changed version of individual changed Incremental files will be co-located on to the full. Files from the prior MASTER FULL will NOT be co-located and the pointers created in the Deferred Full Merge are copied. The USTMERGE process will end quickly. The final result is the completed Full Merge now contains the most recent copy of every file on the server that has CHANGED since the time the MASTER full was run. In this example, only “A, F, J, W, B, G, L, R, N, S, and X” are on the backup dataset. A 2nd differential that now contains the 3rd weeks changes has been created!

11  Week 4 begins and the incremental cycle continues with files “C, D” being changed and subsequently backed up. File “T” has been deleted, so this is reflected.

12  Another Deferred Full Merge is run and any files that have changed since the last backup are copied, as example “Q” illustrates above. Then pointers are created to all the other UNCHANGED files currently on the server but that are on the MASTER full. The pointers are reflected in the diagram with grayed out file names.

28

11

10

9

8

12




Synthetic Differential Full Merge…Process continued 13. Since a sufficient amount of incremental changes have occurred to warrant creating another MASTER:

a)  The USTMERGE command is run WITHOUT the MASTER option. b)  The most recently changed Incremental files will be co-located on to the full. c)  All unchanged files from week 1, the original MASTER full, will be co-located to create this new MASTER FULL.

29

13

The final result is that the completed Full Merge now has the most recent copy of every file on the server. In this example, the most recent version of all changed files from this weeks incrementals and prior deferred to master backups, are co-located onto the new MASTER Full. Deleted files are reflected.




How to Identify Candidates for Synthetic Differential

•  INNOVATION utility that examines the existing UPSTREAM backup history to select the best candidates

•  The best candidates are servers that have a overall low change rate as determined by the utility –  Based on 2 criteria –  Size of all files changed – i.e. 10-15% per week –  # of duplicate files changed every week

30




UPSTREAM Report

31




How to Identify Candidates for Synthetic Differential •  At some point the benefits of Merge to Master diminish •  Servers that have very large incrementals that are larger

than previous full •  At that point run another First Time Full or Full Merge

(up to you) •  Does this feature benefit you and your server change rate?

32




0

500

1000

1500

2000

2500

3000

Elapsed Minutes CPU Seconds Bytes GB

Types of Full Backup Comparison by Resource

First Full Full Merge Deferred Full Synthetic Differential (week 6)

Impressive Results

33




670.9

336.6

121.4

46.8

2751.742

873.601

549.833

135.635

457

458

448

88

0 500 1000 1500 2000 2500 3000

First Full

Full Merge

Deferred Full

Synthetic Differential (week 6)

Types of Full Backup Comparison by Type

Bytes GB CPU Seconds Elapsed Minutes

Impressive Results

34




FDR/UPSTREAM I/O Optimizer •  The I/O Optimizer is a new function that improves the performance of

UPSTREAM backups where large numbers of files are on a server. •  The I/O Optimizer starts several threads of execution on the client

system and pre-opens and caches the file data in parallel for transmission to the storage server.

•  There is still only one communications channel to the storage server, just many processes for pre-reading the files. Currently supported x86 Linux or Linux on z Systems under z/VM.

•  It uses quite a bit of memory (over 1GB) but has the advantage of some performance improvements, in some environments.

35




FDR/UPSTREAM I/O Optimizer •  Option to Skip

backup scan •  Default 6 threads

–  5 reading files –  1 sending data

36




•  In-house testing shown 20-35% improvement in run time on a mixture of large and small files (networking bandwidth and memory provided).

•  The results are different and heavily depend on: –  network bandwidth and storage servers ability

to store backup meta data and data; –  disk characteristics (real disk, array, etc); –  amount of memory available for ability to tune

number of read threads, cache buffer sizes, query length.

•  Your mileage will vary!

FDR/UPSTREAM I/O Optimizer

37




What’s coming?

•  Simplified ISPF panels •  Improved logging •  Enhanced Web Portal

–  No batch jobs –  No external scheduler –  Internally managing versions

•  File Level VMware •  Cloud support?

38




Centralized Enterprise Data Protection Using Mainframe Infrastructure Strengths •  Enterprise Data Protection

Solution Benefits –  z/OS controls backup data from

OS server to z/OS tape –  Virtualization helps solve backup

window problems –  Efficient tape drive and tape media utilization –  Centralizes scheduling, tracking and auditing –  Meet Disaster Recovery restore time objectives

39




OK…then the rest of the products FDRSOS for FCP Devices

40

z/OSz/VM

Linux Virtual Machines

FDRSOS

Linux Dataon FCP Devices

EMC or IBM Array

FICO

N

FICON

FCP




FDRSOS for FCP Devices •  Simple to use DR data protection •  Image level data protection/restore of array resident data

(i.e. DS8700/8800, DMX, VMAX etc.) •  No file level support •  To get the best backup you should dismount the volume first •  Works with SOSINSTANT, IBM FlashCopy and TimeFinder Clone •  Great for bare metal recovery of virtual machines

41




FDRSOS for FCP Devices •  Requires either IBM zDDB and EMC z/SOS no cost microcode features •  Agent-less solution

–  High speed image backup across FICON channel to z/OS –  No network traffic and no TCP/IP usage –  Consolidate mainframe and distributed data protection –  Extend the value of z/OS tape management & security systems –  Improve distributed Data Disaster Recovery with reduced backup and

recovery times •  SOSINSTANT

–  Advanced distributed data disaster recovery backup with IBM FlashCopy and EMC TimeFinder

42




UPSTREAM/SOS to z/OS

43




UPSTREAM/SOS to z/OS •  File level and database support to z/OS Storage Server •  Backup data travels across FICON & Metadata on TCP/IP •  Reduces z/OS and client CPU usage by TCP/IP •  Makes use of small communications disk in array (LBV) •  User data and LBV can be in different arrays •  Centralized z/OS Tape Management, Security and Scheduling •  Hot Database Support for DB2/UDB, ORACLE,

IBM Notes, SQL Server, MS Exchange, etc. •  User interfaces – Web Portal, Director & EUR GUI’s, ISPF •  Provides Data De-duplication Methods, Synthetic Merge Backup,

Block Level Segment Support

44




FDR/UPSTREAM File Level Support to z/OS

45




FDR/UPSTREAM z/OS •  File level and database support to z/OS Storage Server •  Backup & Metadata travel on the TCP/IP LAN or WAN

Communications Network •  Centralized z/OS Tape Management, Security and Scheduling •  Hot Database Support for DB2/UDB, ORACLE,

IBM Notes, SQL Server, MS Exchange, etc. •  User interfaces – Web Portal, Director & EUR GUI’s, ISPF •  Provides Data De-duplication Methods, Synthetic Merge Backup,

Block Level Segment Support

46




z/OS Data Protection for Open Systems Benefits Summary •  Extends investment in existing mainframe resources;

CPU, ATL tape silos, VTL, tape drives and staff. •  Protects the customers software investment by utilizing

existing tape library management software. •  Extends mainframe security to non-disruptive

Open Systems data protection. •  Extends mainframe automated operations to

non-disruptive Open Systems data protection. •  Backup and recovery for very large amounts of data without

a negative performance impact on corporate networks. •  Hot backups for online Open Systems databases. 47




Stay tuned for our WEBINAR… •  Learn how FDR/UPSTREAM can help you maintain massive

amounts of data in this era of BIG DATA. Wednesday, November 16, 2016 10:00 PT / 12:00 CT / 1:00 ET –  Hear how UPSTREAM provides file level support while also providing

hot database support for Oracle, DB2, IBM Notes in addition to Linux on z Systems recovery ability.

–  Come see a demo of the UPSTREAM User Interfaces to see how administrative tasks can be easily accomplished.

–  Discover how AIG implemented UPSTREAM to protect their Linux on z Systems and USS Systems to take advantage of the z/OS infrastructure.

48


http://creativecommons.org/licenses/by-nc-nd/3.0/ 49

Thank You for Attending!

FDR/UPSTREAM…the ONLY File Level z/OS Data Protection Solution for Linux on z Systems.

Session 19153




Trademarks and statements

FDR, FDRSOS, SOSINSTANT, FDR/UPSTREAM and UPSTREAM/SOS are service marks, trademarks or registered trademarks of Innovation Data Processing Corporation. EMC, DLm, SYMMETRIX, VMAX, DMX and TimeFinder are trademarks or registered trademarks of the EMC Corporation. IBM, z/OS, ProtecTIER, zDDB, FlashCopy, z Systems and FICON are trademarks or registered trademarks of International Business Machines Corporation. All other service marks, trademarks or registered trademarks are the property of their respective owners.

16056 - 16071

Documents

FDR/UPSTREAM the ONLY File Level z/OS Data Protection ...– Improved internal communication • Faster restores by improved changes to sorting prior to restore larger record sizes