ZDLRA in Action

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

ZDLRA – in Action

Daniele Massimi – Principal Consultant

BE-IMS

Oracle Engineered Systems - Family

ZDLRA - in Action2 21.09.2016

Oracle Engineered Systems - Family


Who am I


Daniele Massimi, Trivadis AG (CH, Bern)Principal Consultant [email protected]

Seit 2012 bei Trivadis IMS (Infrastructure Service Management)

Oracle DBA seit >16 Jahre (Exadata ~5 Jahre)

Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration, High Availability, Backup & Recovery, Enterprise Manager, Engineered Systems (Exadata, ODA)

Oracle CertificationsOracle Certified Professional Oracle Certified RAC ExpertOracle Exadata Implementations Specialist

mailto:[email protected]

Agenda


1. ZDLRA Functionality

2. Installation and Configuration from bottom up

3. Have a look inside

4. From protected Databases to backups

5. Recovery Capabilities

6. Recovery Appliance Views and Utilities

7. Conclusion


ZDLRA Functionality

Common problems with Oracle backups


RPO depends on archive backup frequency (many minutes or hours)

Backup window and nightly batches compete for resources

High network bandwidth usage for full backups

Low-speed backups and restores

Backup and restore success ratio penalized by shared infrastructure

Critical databases require Data Guard for transaction safety

– Additional space, storage, license and computation required

Expiration of backup set on ML: need to crosscheck and delete expired

Lack of backup validation

…

Recovery Appliance Architecture


Eliminate data loss – Real Time redo transport


Real Time redo transport

– Data Guard like but asynchronous

– Changes out of Logbuffer transfered

– Validate and writes to a stage area

Redolog switch on database

– RA converts the staged redo data to a

compressed archived redolog backup

– Writes this archlog backup to catalog

– Ready for future restores

Explicit Archivelog backup no more necessary

Reduces RPO to (near) zero

Minimal Backup Overhead

10 21.09.2016

Traditional Backup

– Read/Write for Disk/Tape Backup

– Deduplication

– Compression

– Validation

– Deletion

– Merging

All on database host

Degrade performance

Recovery Appliance

– Offloading operations

– Delta Push

– Delta Store

Delta Push


Delta Push is a highly optimized form of source-

side deduplication

– Through RMAN block change tracking

– No reading of unchanged data

– No commited undo

Two operations on the protected Database

– Incremental "forever" backup

– Real time redo transport

One-time full backup as prerequirement

Afterwards Incremental forever backup

Validate incoming Backups against corrupted

physical blocks

Compress the Backups using special block-level

Algorithm

Backuped Database

Changed Data

Less data, less I/O, less network traffic

Delta Store – the “brains” of the Recovery Appliance


Backup

validates the incoming blocks

compresses, indexes and stores

Creates a Virtual Full Database

Backups

10 times less storage usage

High number of virtual full backups

Higher recover window

Delta Store

Day NIncr

Day 1 Virtual Full

Day N Virtual Full

Day 1Incr

Day 0Full

Delta Store – fast restore/recover


Restore

Recreates a physical full backup out

of a virtual one

No need of transfer and apply

incremental backups

Roll forward with restored redologs

Uses Exadata performance and

scalability

Delta Store

Day 0Full

Day 1Incr

Day NIncr

Day ‘N’ Full Backup

Delta Pools


Chunks inside the Delta Store

Backups will be indexed and stored inside the Delta Pools

Set of datafiles blocks from which RA constucts Virtual Full Backups

Each backuped datafile have his own delta Pool

Automated delta pool space management

– Protection Policy will be inherited to database retention policy

– Delete automatically expired or obseleted Backups

– Periodically reorganising for performance improving

– Delta pool optimization when old blocks are deleted and new incremental Backup

arriving

Massive Scale out – Oracle Database Backup Service


10/100/1000 DBs across datacenters

RMAN-integrated subscription-based

backup storage in the public cloud

RMAN compression and encryption

quickly and easily setup with a

secure, low-cost backup

Flexibility in backup/archive concepts

The Recovery Appliance flexibly

supports both tape and cloud archival

option


Installation and Configuration

from bottom up

Installation Steps


Create the ZDLRA configuration using

the latest OEDA

Start the Re-Imaging and Setup from the first

Computing Node (like an Exadata Setup)

[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16

Installation Steps


Recovery Appliance specific Installation Steps

[root@zdldbadm01 install]# ./ra_install.pl --help

ra_install.pl - Recovery Appliance Installer

You must be logged in as root to run this command.

All steps should be run successfully for install.

Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--

stdout]

Options:

--help: displays this help message

--step=number: Which step to run

--stdout: Print all messages to stdout rather then the log file

--install: Installs the step [Default]

--uninstall: Uninstalls the step

Step Numbers:

Step 1 - Validation and Configuration Prep

Step 2 - OS Setup

Step 3 - Oracle User Setup

Step 4 - DBFS Setup

Step 5 - Tape Backup configuration

Step 6 - ZDLRA DB Backup Setup

Step 7 - Enable ZDLRA Services

Configuration Steps


Deployment of the Agents on the compute Nodes

Recovery Appliance Discovery

Installation of the Backup Module on any Clients

[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass

manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir

/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib

Recovery Appliance Install Tool, build 2015-11-03

Recovery Appliance credentials are valid.

Recovery Appliance wallet created in directory

/u01/app/oracle/admin/orcl02/xdb_wallet.

Recovery Appliance initialization file

/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.

Downloading Recovery Appliance Software Library from file ra_linux64.zip.

Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.

Download complete.

[oracle@odax30 zdlra]$


Have a look inside

(just particularities)

On the computing Nodes


Metadata Database

– RAC Database (also used for Recovery Catalog (VPC))

– Non Container Database

– DBFS Mount-Points for Backup and Replication purpose

– DBFS added as Cluster Resources

– HTTP Service inside the database will be started for communication between the

clients and the ZDLRA (dbms_ra.startup_recovery_appliance)

oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs

[email protected]:/

957M 120K 956M 1% /dbfs_repdbfs

[email protected]:/ 300G 23G 278G 8% /dbfs_obdbfs

SQL> exec dbms_ra.startup_recovery_appliance;

PL/SQL procedure successfully completed.



ASM Configuration

– Two Diskgroups DELTA (for Backups), CATALOG (Metadatas)

– New special sub-Directory CONTAINER for storing Container Files which contains

the Delta Pools and Delta Stores

– Each Container File as a size of 2 TB ASMCMD [+delta/zdlra] > ls -l

Type Redund Striped Time Sys Name

Y ARCHIVELOG/

Y CONTAINER/

Y DATAFILE/

Y TEMPFILE/

ASMCMD [+delta/zdlra/CONTAINER] > ls -s

Block_Size Blocks Bytes Space Name

512 4294959104 2199019061248 4398059094016 con.365.920823681

512 4294959104 2199019061248 4398059094016 con.366.920823687

512 4294959104 2199019061248 4398059094016 con.367.920823699

...



Backup of Metadata Database

– Done using Export of RASYS Schema every 2 Hours

– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export

– Scheduling via "anacron"[root@zdldbadm01 ~]# cat /etc/anacrontab

SHELL=/bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

# the maximal random delay added to the base delay of the jobs

#RANDOM_DELAY=45

# the jobs will be started during the following hours only

START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command

1 65 cron.daily nice run-parts /etc/cron.daily

7 70 cron.weekly nice run-parts /etc/cron.weekly

30 75 cron.monthly nice run-parts /etc/cron.monthly

[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/

total 4

-rwx------ 1 root root 409 Nov 10 2015 0anacron

lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl

On the cell Nodes


Fortunatelly nothing special

Same appearance as in Exadata Storage Cell

During the Installation the "Write Back" Option on the Flashdisks will be enabled

Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks

instead of 12 for the CATALOG ASM Disk Group

User Model ZDLRA


Component Account Type User Name Description

EM Cloud Control Cloud Control Super User SYSMAN EM Admin User

EM Cloud Control Cloud Control Admin User specified User to manage RA and protected databases

Recovery Appliance RA Metadata Super User SYS DB Admin User, needed for create RA User

Recovery Appliance RA Admin RASYS RA Schema Owner

Recovery Appliance RA User Account User specifiedUser on RA for managing Metadata on VPC

of protected databases

Protected DatabaseProtected Database

Backup Admin

User Account with

SYSBACKUP privileges

User for doing backup, restores and recovery

of protected databases

User Accounts:


From protected Databases

to backups

Protected Databases


Once the Environment meets all the requirements we can add the databases to the

ZDLRA

The requirement are:

• Network connectivity

• Backup Module is installed for every Oracle Home

• Definition of Protection Policies are created (Retention Time)

Block Change Tracking not mandatory, but recommended !!!

Two Methods are possible:

• Enterprise Manager

• Manually DBMS_RA PL/SQL Package

Prerequisite for Adding Protected Database

(EM and manually)


Create an VPD User on the Metadata Database

Grant the Create Session privilege

SQL> create user ra_orcl03 identified by <pwd>;

SQL> grant create session to ra_orcl03;

Adding Protected Database with EM


Add Protected Database on Recovery Appliance Page

– Choose the Protection Policy

Adding Protected Database with EM


Configuring Backup Settings– Register database

– Add RMAN configuration (configure...)

– Enabling Redo Transport

(add parameter to spfile and restart DB)

Scheduling Protected Database with EM


Protected Database is ready for backing up

– We need a Schedule



Configure the repeating interval for

the Incremental-Forever Backups

Configure the Archive Log Backups and

deletion rule



Also useful as Script Generator here the Script is not modifiable !!!



Output of an Backup Schedule on EM13c

Adding Protected Database manually


Add Database for the protected database

Grant access to Virtual Private Catalog

Configure the RMAN Channel for ZDLRA

begin

dbms_ra.add_db (

db_unique_name => 'orcl03',

protection_policy_name => 'bronze',

reserved_space => '250G');

end;

begin

dbms_ra.grant_db_access (

db_unique_name => 'orcl03',

username => 'ra_orcl03');

end;

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS

"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,

ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet

credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Adding Protected Database manually


Enable Real-Time Redo (Delta Push)

Restart the Database

Configure the RMAN Channel for ZDLRA

ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;

[oracle@odax30 ~]$ srvctl stop database -db orcl03

[oracle@odax30 ~]$ srvctl start database -db orcl03

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS

"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,

ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet

credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Execute Backup manually


Backups are like in traditional way

Only "forever" incremental level 1 backups should be schedule

Level 0 Backup are of course always possible

It works also with old fashion syntax soft link from libra.so to libobk.so are

needed !!!

Scheduling is also free selectable, EM is recommended !

[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra

...

connected to target database: ORCL02 (DBID=3592015831)

connected to recovery catalog database

RMAN> backup incremental level 1 database;

run {

allocate channel c1 type sbt_tape;

backup incremental level 1 database;

}

...

ORA-27211: Failed to load Media Management Library


Recovery Capabilities

General statement about Recoveries of protected

Databases


Recoveries of protected databases can be done with EM or manually

With EM several Recovery-Scenarios are possible

All Recovery scenarios with EM are full automated

(e.g. Creation of auxiliary Databases, etc.)

EM generates RMAN Scripts for each Scenario

with the possibility of adaption

Possibility of selection between full and point-in-time

recoveries

Duplicate Database (for standby) need some preparation

(e.g. FS-Structure (admin tree), Password-File, etc.)

Protected Database Recovery with EM


From database Page under Availability Backup & Recovery Perform Recovery



Choose the needed Recovery Scenario

... You can choose an other restore

location if needed...



A schedule will be created

... And you can adapt the script if needed and submit the job...



The Job can be monitored by clicking the "View Job" button...

Protected Database Recovery manually


Restore can be done also manually

– of course db must be mounted before...

– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)

Or a point-in-time Recovery

run {

restore database;

recover database;

alter database open;

}

run {

set until time '08.09.2016 08:45:00';

restore database;

recover database;

alter database open resetlogs;

}


Recovery Appliance Views and

Utilities

Some RA Views


It is also possible to read the Recovery Appliance Information over Views from Recover

Appliace Metadata Database

View Name Description

RA_DATABASE Information about Protected Databases

RA_DB_ACCESS Describes User Accounts that have access to which protected database

RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance

RA_DATABASE_STORAGE_USAGE Storage usage for each protected database

RA_RESTORE_RANGE Describe Restore Range for each protected database

RA_INCIDENT_LOG Describe Recovery Appliance Incidents

RA_REPLICATION_SERVER Replication Server Information (if used)

rastat.pl Utility


rastat Utility help to evaluate the performance of Recovery Appliance

– NETBACKUP/NETRESTORE

• Measure the network performance

– ASMREAD/ASMWRITE

• Measure the ASM Disk I/O Performance

– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC

• Measure the Container Disk I/O Performance

oracle@zdldbadm01:/opt/oracle.RecoveryAppliance/client/ [zdlra1] perl

/opt/oracle.RecoveryAppliance/client/rastat.pl --test=ASMREAD --filesize=2048M --

rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=+CATALOG

RECOVERY APPLIANCE READ IO TEST FROM DISK

Disk Group: +CATALOG

2048 MB, 1.90s read IO time, .10s CPU time, 1076.40 MB/s, 5.26% CPU usage

PL/SQL procedure successfully completed.

dbms_ra Package


dbms_ra provides administration function from inside the RA Database

Ca. 50 Procedure for different types of administration

– Start/Stop of Recovery Appliance

– Add/Delete Protected Databases

– Grants Access to specific User to enable to backup and restore

– Add/Update Protection Policies

– Manage Tape Configuration (if available)

– Add Repliaction Server (if available)

But, the utilization of Enterprise Manager is recommended !!!


DEMO

Demo


Recovery Appliance Page

Configure Protected Database

Run a Backup

Real Time Redo Transport

Restore a Protected Database


Conclusion

Conclusion


ZDLRA worked in our Tests as

expected

Very good implementation of well

known Backup and Recovery

Technology on Engineered System

RPO is (near) zero with Real Time

Redo Transport

Even RTO ist much better then

existing systems

Very simple Management with EM

Risk reduced, cost reduced,

qualitiy improved

ZDLRA will find it’s

customer !

Questions and Answers...

Daniele Massimi – Principal Consultant

Tel. +41 58 459 50 92

[email protected]

21.09.2016 ZDLRA - in Action53