Oracle Backups Benchmarked and Best Practicescoug.ab.ca/wp-content/uploads/2017/09/2017_CoP_Perf_Tune...Oracle Backups Benchmarked and Best Practices Ryan Behiel Solution Architect

Oracle BackupsBenchmarked and Best Practices

Ryan Behiel

Solution Architect

Agenda

• An Oracle Backup Performance Retrospective

• Key Oracle Backup Concepts – a Few Simple Things you Should know

• NetBackup Copilot for Oracle - Overview

• Testing Performance Limits (AKA Don’t Shoot the Messenger!)

• Oracle Database Performance Enhancing Configuration Suggestions

• Copilot For Oracle Performance Numbers

• Q&A

Copyright © 2017 Veritas Technologies LLC2

An Oracle Performance Backup Retrospective


April 15, 2002

“eWeek Labs and engineers from Veritas Software Corp.'s labs, in Mountain View, Calif., recently completed a high-end Oraclebackup performance test…”

“eWeek Labs worked directly with Veritas engineering on thisbenchmark.”


Veritas Oracle Backup Benchmark Hardware Configuration


1

3

2

1

3

1

2

4

STK L700 Fibre

Fibre

Channel

Zoned

STK L700 Fibre

Brocade 3800/3200

2 Gb Switches

L700 Control

Path

1

1

VERITAS

NetBackup™

4.5

Sun Fire

6800

STK

9176(x2)

2.88TB

Oracle9i

STK

9176(x2)

2.88TB

Oracle9i

Benchmark Hardware List

Backup Server

Sun Micro 6800

24 750MHz CPUs

25GB of RAM

Disk StorageSTK 9176B (x2)

2.88 TB (each)

Tape Drives STK 9840B (x24)

Tape Libraries STK L700 (x2)

2 Gb Fibre Switch Brocade (x6)

2 Gb Fibre HBA Emulex (x31)

Est. HW Cost $1.9MM (2002 $’s)

A total of 31 Emulex 2 Gb Fibre cards were required to connect all this hardware together.

In 2002, 2Gb Fibre was relatively new.

The first batch of 2 Gb cards were from JNI. We found a hardware defect that only occurred when restoring data. All 31 cards had to be replaced and the zoning completely reconfigured due to new WWN’s.

This defect was very costly to JNI.


6 Brocade 2 Gb Fibre switches wereneeded to connect the massive amountof equipment together.


StorageTek (now part of Oracle) loaned us 24 STK 9176B tape drives for this benchmark.

This was the only tape drive that offered a combination of two critical performance criteria:

– 5 second tape load speed

– 20 MB/sec backup performance (native)

There were faster tape drives but most drives of the day took over a minute to load. Achieving a 2 TB benchmark number would have been impossible if it took 24 minutes to simply load all of the tape drives.


Benchmark Results

Drum roll, please…

“It took us 1 hour to back up 2 terabytes of data.”

To put this in perspective, a few months before this benchmark, an Oracle Corporation benchmark was only able to achieve 1TB/hour.

Let’s use the lessons learned from this type of benchmark to help us create the fastest and most efficient Oracle backup environment.


The NetBackup Appliance – Performance Powerhouse


Oracle 2TB/hr Benchmark (2002) NetBackup 5330 (2017)

315U (7.5 Racks) Rack Units 14U

2TB Backup Storage 687TB

2TB/hr Backup Performance 7TB/hr

$1.9MM Price Priceless

Key Oracle Backup Concepts

A Few Simple Things You Should Know


Key Oracle Concepts

RMAN: Recovery Manager. Oracle created backup API. Introduced in Oracle 8

Online redo logs: Files that store changes to database as they occur.

Archived redo logs: Online redo logs that are transitioned to an offline state

Datafile: The majority of Oracle data is stored inside of datafiles

Oracle Restore: Physical recovery (e.g. datafile) of database

Oracle Recovery: Logical recovery of database to point in time using redo logs

BCT: Block Change Tracking. When enabled, makes Oracle incremental backups faster


NetBackup Copilot for Oracle

Quick Overview


Backup Processing Overview – Copilot for Oracle


NetBackup Deduplication Appliance

OracleServer

Oracle RMAN script is automatically generated and

database backup is scheduled and run by

NetBackup

1

NetBackup automatically sweeps the Oracle backup data into NetBackup share

2

NetBackup knows exactly which DB the backup images belongs to, how long to retain

images, and automatically manages the database disk

backup location (MSDP, Tape, etc.)

3

Testing Environmental Performance Limits

Don’t Shoot the Messenger


Baseline Performance Testing

“The lesson here: Analyze the entire data path when designing backup systems. You'll be wasting money on high-end tape and disk hardware if the systems and network between them are slow.”

• Baseline performance testing is arguably the most important step in the process of optimizing the backup performance of your environment

• Baseline testing determines performance characteristics of hardware external to the NetBackup Appliance

• Focus on the network, SAN and Oracle environment

– The performance of the NetBackup Appliance is thoroughly tested and documented


Baseline Performance Testing – Oracle Database Server

• Basic performance metrics

– 10 GbE = 600 MB/sec (nominal)

– 8 Gb Fibre = 600 MB/sec (nominal)

– Bonding increases throughput but not 100% linear

• I/O Capacity

– So many variables associated with I/O

– Only true way of determining I/O capabilities is through testing of every storage component



OracleServer


Use the NetBackup agent for basic perf testing

• Backup simple (non-Oracle) data from all storage connected to the Oracle server

– This is a safe, non-destructive test

– Configure a non-Oracle policy for single and multiple stream backups

– Document backup performance achieved as streams are added

– Likely to saturate performance with 2 or 3 streams

– This test will give you an indication of achievable Oracle backup performance



OracleServer

1

For standard Oracle client and Copilot for Oracle

backups, the NetBackup Oracle agent will be installed

agent

1

1

1


Additional baseline performance testing utilities:

RMAN BACKUP VALIDATE command– Run this from the DB server

– Non-destructive

– Designed to validate datafile blocks

– Also provides a good indicator of I/O capacityassociated with datafiles

– Have DBA run this if backup performance is in question

IOMETER– Open source I/O testing tool

– Non-destructive

– Provides detailed analysis of all I/O patterns


OracleServer

Oracle Database Performance Enhancing Configuration Suggestions


“An effective backup strategy must be tailored to the type of data being backed up.”

Oracle Database Performance Enhancements

• Enable Block Change Tracking (BCT)

– Significantly reduces I/O impact of incremental backups

– Without BCT, 100% of every datafile is scanned during every backup

– Added impact on Oracle database is negligible

• Limit number of full backups

– Use differential incremental backups

– Oracle’s Incremental Merge can help (more on this later)

• More datafiles enables better backup parallelism

– Typical to have 1 backup stream per datafile

– This can impede backup performance if number of datafiles is small

– You have little impact on this setting but you can discuss with DBA


OracleServer

Copilot for Oracle Performance Enhancers

• Optimized Share Reserve provides optimal performance

– I/O is load balanced across entire shelf (6 LUNS)

– Minimize I/O contention by scheduling the “sweep” portion of the dump and sweep activity after all backups have completed.

• NetBackup buffers

– Default setting is generally good

– For some environments, adjustments may be necessary

– Might be worth looking at if performance is subpar

• Is the NetBackup Appliance oversubscribed?

– The Appliance is extremely scalable

– Any hardware has limits



Copilot For Oracle Performance Numbers


“Our primary concern for this

test was performance…”

NetBackup Oracle Agent vs. NetBackup Copilot for Oracle

This test is a comparison of two backup methods

– Standard Oracle agent

– Copilot for Oracle

Database size = 15TB


Backup Method (Full) Throughput CPU Load - DB Server

NetBackup Oracle Agent 3.04 TB/hr 8.01%

NetBackup Copilot – Std Share 2.84 TB/hr 5.74%

NetBackup Copilot - OSR 4.12 TB/hr 8.8%

With Copilot, DBAs are empowered to perform ad-hoc backups, process all restore requests and are provided comprehensive

reporting through the RMAN catalog

Recommendation: OSR offers optimal performance while offloading

complicated RMAN tasks to Oracle DBA

NetBackup Copilot for Oracle Concurrent Throughput

This test measures concurrent throughput of multiple DB backupsto an Optimized Share Reserve

Multiple Oracle databases configured on single server

Recommendation: Two simultaneous backups from single DB server delivers optimal mix of aggregate performance and load on DB server


Backup Method (Full) Aggregate Throughput CPU Load - DB Server

Single DB backup 4.12 TB/hr 8.8%

2 Simultaneous DB Backups 7.0 TB/hr 8.06%

3 Simultaneous DB Backups 7.2 TB/hr 9.12%

Copilot for Oracle Incremental Merge

• Incremental Merge technology was developed by Oracle

– A.K.A. Incremental Update

• A logical full backup without the impact of a full backup

– Significantly reduces backup impact on the Oracle server

– Reduces network overhead & provides significant storage savings

• NetBackup engineers developed a method of combining Oracle’s Incremental Merge technology with the Copilot for Oracle feature

• The result is a unique “incremental forever” Oracle backup integrated with the Copilot for Oracle “backup to disk” method

• Implementation is simple

– Entire process is handled by NetBackup Copilot for Oracle and completely automatic


NetBackup Copilot for Oracle Incremental Merge

Measure the performance advantages that incremental merge provide

Database size – 10 TB

Recommendation: Effectiveness of Incremental Merge highly dependent on change rate in Oracle DB. Oracle has documented this.


Backup Method Backup Duration CPU Load - DB Server

Full 3:02 (hh:mm) 8.83%

Incremental Merge – OLAP (0%) 4:00 (mm:ss) ---

Incremental Merge – OLTP (3.2%) 7:01 (hh:mm) 1.52%

Copilot for Oracle Instant Recovery

“No matter how fast, how big or how much is spent on a backup implementation, none of it matters if you can't restore the data accurately and quickly.”


Copilot for Oracle Instant Recovery Process (NBA 3.1)

1. Oracle “backup to disk” image is cloned

– No changes to original backup image ever occur

– Any RMAN full/incr backup images located on backup disk can be used

2. Cloned image is exported to Oracle server over NFS

– Oracle redo log group is cleared

3. Oracle database is mounted, open and fully accessible



1

2

3

1

2

3

Copilot for Oracle Instant Recovery - Backup To Disk and Sweep

• Instantly recover Oracle database– Full backup

– Incremental backup

– Incremental merge backup

Recommendation: Understand the impact that the Oracle reset log process can have on overall instant recovery performance


Oracle Redo Log Cleared?Database provisioned,

mounted and open

No 31min 37sec

Yes 4min 46 sec

Thank you!

Copyright © 2017 Veritas Technologies LLC. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of VeritasTechnologies LLC or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

Ryan Behiel

[email protected]

mailto:[email protected]

Documents

Oracle Backups Benchmarked and Best Practicescoug.ab.ca/wp-content/uploads/2017/09/2017_CoP_Perf_Tune...Oracle Backups Benchmarked and Best Practices Ryan Behiel Solution Architect