Database Virtualization and

Instant CloningKyle Hailey

http://dboptimizer.com

Database Cloning Challenge

Business want data now.

Business don’t understand DBAs.

Databases getting bigger & harder to copy.

Developers want more copies.

Reporting wants more copies.

Everyone has storage constraints.

If you can’t satisfy the business demands your process is broken.

Two Parts

I. Cloning TechnologyII. Accelerate your business

Part I : Cloning Technology

3. Virtual2. Thin Provision1. Physical

=

database

1. Physical Cloning

Problem

Developers

QA and UAT

Reports

First copy

Production

• CERN -  European Organization for Nuclear Research

• 145 TB database• 75 TB growth each year• Dozens of developers want copies.

workaroundsDevelopers

QA and UAT

ReportsShared

Sub set copy

Production

Many copies

Physical Clones

Database SubsetsShared Databases

Subsets

ProductionThe Production ‘Wall’

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

Shared Full

Shared access = Poor Productivity

Developers and tester get frustrated

Databases become old and unrepresentative of production.

Requires complex scheduling and management

Never enough environments

Average customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle

Physical Copies – Time & Space consuming

Time consumingTime to make copies, days to weeks

RMAN backup, archive logs, copy data over, recover

Meetings , days to weeksSystem, Storage ,Database ,Network Admins , manager coordination

Space consuming 12 devs x 10TB production = 120TB20 report DBs x 40 TB = 800TB

Slow Clones mean bottlenecks

Setup Develop

Setup

Setup Develop

QA

$40M

$75M

$850M

$27,000M

Storage

IT

Develop

Business

ERP Project Failures 2011

• NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millions

Standish : IT Project Failure Rate

1994 1996 1998 2000 2002 2004 2009

31% 40% 28% 23% 15% 18% 24%

★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html

Clone 1 Clone 3

99% of blocks are Identical

Clone 2

2. Thin Provision Cloning

Clone 1 Clone 2 Clone 3

Thin Provision

2. Thin Provision Cloning

Requires Cornerstone Technology

I. clonedbII. Copy on Write Snapshots

a) EMC BCV b) EMC SRDF or Recover Pointc) Vmware

III. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS

2. Thin Provision Cloning

RMAN backup

dNFSsparse file

I. clonedb

RMAN backup

dNFSsparse file

I. clonedb

I. CloneDB• dNFS 11.2.0.2+

– cd $ORACLE_HOME/rdbms/lib– make -f ins_rdbms.mk dnfs_on

• Clonedb.pl initSOURCE.ora output.sql– MASTER_COPY_DIR="/rman_backup”– CLONE_FILE_CREATE_DEST="/nfs_mount”– CLONEDB_NAME="clone"

• sqlplus / as sysdba @output.sql– startup nomount PFILE=initclone.ora – Create control file backup location– dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf');– alter database open resetlogs;

Tim Hallwww.oracle-base.com/articles/11g/clonedb-11gr2.php

I. CloneDB

RMAN

physical

Target A NFS Server

Clone 1

Clone 2

Read only Clone 3

Target B

Clone 1

Clone 2

Clone 3

I. clonedbII. Copy on Write + snapshots

a) EMC BCVb) EMC SRDF or Recover Pointc) Vmware

III. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS

2. Thin Provision Cloning

D

ActiveFile

SystemSnapshot

CBA

File System constant - EMC

II. Copy on Write a) EMC snapshots

D

ActiveFile

SystemSnapshot

DCBA

Changes written in place - EMCChanges to new area - VMware

II. Copy on Write a) EMC snapshots

• Goal: backup– Create BCV , take snaps of BCV– Zone and mask LUN to target host– Full copy of disk, snapshot, recover

• Snapshots– 16 snapshots– Zero or one snapshots of snapshots

• “Golden Copy”– EMC uses a save area, area for changes to the snapshot– Initial snapshot has to stay

II. Copy on Write a) EMC BCV

Non-Prod FilerEMC Filer

ProductionDatabase

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

SnapshotsMax 16

a) EMC BCVIII. Allocate on Write

BCV Day 1

snapshot

Snapshotor break

Day 2

Timefinder needed for snapshots on across multiple LUNs

• SRDF – Symetrix Remote Data Facility– Stream Changes to a remote machine

• Recover Point– Capture changes on wire– Split changes send to remote site

II. Copy on Write b) EMC SRDF

Non-Prod FilerEMC Filer

ProductionDatabase

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

File system level

SRDF orRecover Point

Snapshot day 1

Snapshot day 1

Snapshot day 2

Snapshot day 2

II. Copy on Write b) EMC SRDF

D

ActiveFile

SystemSnapshot

DCBA

Changes written to a new locationChanges to new area - VMware

II. Copy on Write, c) VMware

II Copy on Write c) VMwareData Director : Linked Clones

• Not support for Oracle databases• Golden Copy : rebuild after 32 snapshots• x86 host databases only• Performance issues

– “Having several linked clones can affect the performance of the source database and the performance of the linked clones.”http://bit.ly/QOXbyE (on http://pubs.vmware.com )

– “If you are focused on performance, you should prefer a full clone over a linked clone.”http://www.vmware.com/support/ws5/doc/ws_clone_typeofclone.html

– Performance worse with more snapshots– Performance worse with more concurrent users

I. clonedbII. Copy on Write

a) EMC BCVb) EMC SRDF or Recover Pointc) Vmware

III. Allocate on Writea) ZFSb) Netapp (EMC VNX)c) DxFS

2. Thin Provision Cloning

Data Blocks

root

a. ZFSb.Netapp (EMC VNX)c. Dxfs

III. Allocate on Write

a) ZFS

• 1 disk = 1 filesystem• ~1990: N disks = 1 FS• 2001: ZFS starts• 2005: ZFS ships• 2008: ZFS storage appliance ships• 2010: Delphix heads ZFS open source

III. Allocate on Write

FS vs. ZFS• FS per Volume

• FS limited bandwidth

• Storage stranded

• Many FS in a pool

• Grow automatically

• All bandwidth

Storage PoolVolume

FS

Volume

FS

Volume

FS ZFS ZFS ZFS

3a) ZFSIII. Allocate on Write

a) ZFS

Snapshot rootLive root

Delphix Proprietary and Confidential

ZilIntent Log

III. Allocate on Write

ZFS Appliance + RMAN

1. ZFS Appliance– Project db_master– Project db_clone – 4 file systems: datafile, redo, archive, alerts

2. Source Database– NFS Mount ZFS Appliance – Backup with RMAN

3. ZFS Appliance• Select db_master• Snapshots• Then each filesystem on db_master clone it onto db_clone

4. Target Host– NFS Mount db_clone– recover clone

cloning-solution-353626.pdf

a) ZFS III. Allocate on Write

Oracle ZFS Appliance + RMAN

1. physical

ZFS Storage Appliance

RMANcopy

RMAN Copyto NFS mount

Target A

Clone 1

Clone 1

Snapshot

NFS

a) ZFS III. Allocate on Write

cloning-solution-353626.pdf

Oracle ZFS Appliance + Dataguard

Dataguard

ZFS Storage Appliance

datafiles

NFS

Target A

Clone 1

Clone 1

Snapshot

NFS

a) ZFS III. Allocate on Write

Datafiles via NFS

maa-db-clone-szfssa-172997.pdf

NetApp FilerNetApp Filer

ProductionDatabase

Database Luns

snapshot

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

File system level

snapshot

clonesSnap mirror

Snapshot Manager for Oracle

Flexclone

b) NetappIII. Allocate on Write

Requires a database

NetApp FilerNetApp Filer

ProductionDatabase

Database Luns

snapshot

Target A

Target B

Clone 1

Clone 2

snapshot

snapshot

snapshot

b) NetappIII. Allocate on Write

NetApp FilerNetApp Filer

Physical Database

Database Luns

snapshot

Target A

Target B

Clone 1

Clone 2

snapshot

snapshot

snapshot

b) NetappIII. Allocate on Write

• Compression • typically 3x

• Block sharing• DxFS optimized for databases

• Write optimizations• Space allocation and destroy• Shared blocks in memory

c) DxFSIII. Allocate on Write

I. clonedbII. Copy on Write Snapshots

a) EMC BCV b) EMC SRDF or Recover Pointc) Vmware

III. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS

2. Thin Provision Cloning

• Clonedb • EMC BCV

• Netapp• EMC SRDF• EMC VNX

2. Review: Thin Provision Cloning

2. Thin Provision Cloning

2. Thin Provision Cloning

3. Database Virtualization

• Oracle SMU• Delphix

Virtualization Layer

Virtualization

SMU

a) Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance

• Requires ZFS Appliance• Supports Linux , Solaris 10+, Windows 2008+• GUI

– snapshot source databases – provision virtual databases

3. Database Virtualization

Virtualization Layer

x86 hardware

Allocate StorageAny type

SMU

Choose your virtualization Layer:• Delphix and Oracle SMU automated out of box• Netapp and EMC SRDF/Recover Point require massive scripting• Delphix and possibly SMU share blocks in memory

ZFS Storage Appliance

One time backup of source database

Database

Production

Instance

File system

RMAN APIs

Delphix Compress Data

Database

Production

Instance

File system

Data is compressed typically 1/3 size

Incremental forever change collection

Database

Production

Instance

File system

Changes are collected automatically foreverData older than retention widow freed

Typical Architecture

Database

File system

Production

Instance

Database

File system

Development

Instance

Database

File system

QA

Instance

Database

UAT

Instance

File system

Clones share duplicate blocks

Development

Instance

Database

Production

Instance

File system

vDatabase

QA

Instance

UAT

Instance

vDatabase vDatabaseNFS

Source Database Clone Copies of Source Database

Fiber Channel

Part II: Virtualization Case Studies

1. Development2. QA3. Recovery

1. Development Acceleration

Database Virtualizationlayer

3 clones of same source database 3 virtual clones of same source database

Virtualization layer orchestrates I/O access between database and storage

Source Timeflow

Source Timeflow

Source Database

Target HostVirtual

Database

NFS

Fiber

Fiber

RMAN over TCP

Source

Fresh

VDB

Source

VDB

VDB

VDB

Frequent

Fresh data

Full clonesFrequent (many)

fast

Self Service

Devv2.6 v2.6v2.6

QA UAT

v2.6Production

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

Source Control for the database data

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

DevProd

2.6

Dev

QA

Prod

2.6

Dev

QA

UAT

Prod

2.6

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Data Control = Source Control for the Database

Collaborative Provisioning

VDB2

Source2

Source3

Source1

VDB3

VDB1

Collaborative sources

“I looked like a hero”Tony Young, CIO Informatica

1: Review of Development

• Provision , Self Service– Fast– Full– Fresh– Frequent (Many)

• Source control for the database• Federated cloning

2. Rapid QA

dSource

Pass to QA

Developer VDB

QA VDB

Source

Punt to Dev

Developer VDB

QA VDB

Developer VDB

dSource

Forensic Analysis

VDB

dSource

Upgrades, Patches, RAT

VDB

A/B Testing

• Production vs Virtual– invisible index on Prod– Creating index on virtual

• Flashback vs Virtual• Confidence testing

– change reporting SQL– compare new code over last 30 days

2: review QA

• Fork copies off of Dev to QA• QA fork copies back to Dev• Instant replay

– performance A/B– Upgrade, patching– Forensic analysis of Prod

3. Backup and Recovery

MONSUN WEDTUE THU SATFRI MONSUN TUE

MONSUN WEDTUE THU SATFRI MON

SUN TUE

Dat

a Pr

oces

sing

Re

quire

d

Current Volume of Work

Source

Rollback / Time machine

VDB

Source

Rollback / Time machine

VDB

VDB Rolled back

Source

Recovery

VDB

V2P

3: review Backup

• Backup 50 days in size of 1 copy• logical recovery on prod, dev• Restore virtual to production

Review Part II: Data Virtualization1. Development: Full, Fresh, Fast, Data SCCS = SDCS2. QA : forking, replay, forensics3. Recovery: 50:1, time machine, V2P

Review Part I: Technology1. Full Clones2. Thin Provision : dbclone, EMC, Netapp3. Database Virtualization : SMU, Delphix

over 10 times

"perhaps the single largest storage consolidation opportunity history“

Oracle 12c

80MB buffer cache ?

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

200GBCache

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

200GBCache

8000

Tnxs

/ m

inLa

tenc

y

600 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

Database Virtualization

Matrix of featuresCloneDB ZFS

ApplianceDelphix Data

DirectorNetApp EMC

Time Flow No Yes Yes No Yes No

HardwareAgnostic

Yes No Yes Yes No No

Snapshots No Unlimited Unlimited 31 255 16 (96 read only)

Snapshots of snapshots

No Unlimited Unlimited 30 255 1

Automated Snapshots

No No Yes No Yes No

Automated Provisioning

No No Yes No No No

Any O/S Yes Yes Yes Nox86 only

Yes Yes

Max size None None None ? 16-100TB

?

Conclusion : Enterprise Solutions• VMware Data Director • EMC Timefinder

– offer limited ability to benefit from cloning• Clonedb ***

– fast easy way to create many clones of the same copy– limited to 11.2.0.2 and systems with sparse file system capability– suffers the golden image problem and performance

• NetApp Flexclone, Snap Manager for Oracle– offers a rolling solution – limited database awareness– file system clones– limited snapshots– Vendor lock-in

• Oracle ZFS Appliance– Vendor Lock-in

• Delphix– Agility : Automation, unlimited snapshots, clones, multi-database

Appendix• CloneDB

– http://www.oracle-base.com/articles/11g/clonedb-11gr2.php• ZFS

– http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/zfslast.pdf• ZFS Appliance

– ZFS setup for Oracle http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf– using RMAN http://www.oracle.com/technetwork/articles/systems-hardware-architecture/cloning-solution-353626.pdf– using dataguard http://www.oracle.com/technetwork/database/features/availability/maa-db-clone-szfssa-172997.pdf

• Data Director– http://pubs.vmware.com/datadirector-25/topic/com.vmware.ICbase/PDF/vfabric-data-director-25-administration-guide.pdf

p107 linked clone not support for Oracle• EMC

– https://community.emc.com/servlet/JiveServlet/previewBody/11789-102-1-45992/h8728-snapsure-oracle-dnfs-wp.pdf– http://www.centroid.com/knowledgebase/blog/cloning-oracle-databases-with-emc-snapview-and-recoverpoint– http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:39960485246162 BCV and SRDF

• NetApp– http://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdf– OEM for Netapp http://www.youtube.com/watch?v=J7fnfLS5Dxg&feature=youtu.be

• Oracle SMU– http://www.oracle.com/us/products/servers-storage/storage/nas/smu-bus-wp-final-1857065.pdf– http://www.oracle.com/us/products/servers-storage/storage/nas/snap/smu-datasheet-1903178.pdf– http://www.oracle.com/us/products/servers-storage/storage/nas/snap/overview/index.html

• Oracle OEM Snapshot Manager for Netapp– http://www.youtube.com/watch?v=J7fnfLS5Dxg&feature=youtu.be– Demo starts at 3:10 goes to 19:43

Other

• Can’t run multiple clones on same host with ASM https://forums.oracle.com/forums/thread.jspa?messageID=1810251

• EMC VNX http://virtualgeek.typepad.com/virtual_geek/2012/05/vnx-inyo-is-going-to-blow-some-minds.html

NetApp Limits

Controller Size Limit

32 bit controllers 16TB

FAS3140/FAS3040/FAS3050 40TB

FAS3160/FAS3070 50TB

FAS6040/FAS3170 70TB

FAS6080 100TB

All sources have to be in the same aggregate to be snapshot together.

Limit of 255 snapshotssnaps are limited to the same aggregate (storage pool)Aggregates have size limits depending on controller

b) NetappIII. Allocate on Write

b) NetappIII. Allocate on Write

b) NetappIII. Allocate on Write

Netapp vs

EMC

Production

Initial Incremental Incremental

Redo

Clones

Redo

a) ZFS III. Allocate on Write

Goal - How Can we do it?

Data Director

CloneDB NetApp

EMC

Delphix

SMU

ZFS

Flawed

Limited UseManual

low touch: virtualization

Automated (easy)

Manual (Hard)

Full

Func

tiona

lity

Doe

sn’t

wor

k

Application Life Cycle Management

Number of required clones change during Project and Extended Project Life CycleProjects with phased Go-Lives can have Implementation, Handover ‘Cut-Over’ and Operations ‘RUN’ all running concurrently.

Development & maintenance is hard

UserAcceptance

TestTestDevelopment

Production

Training & Education

ProductionSupport

New release / upgrades lifecycle…Bespoke, enhancements, configuration changes etc…

Production Support lifecycle…Internal bug fixes, 3rd Party Patches, Data fixes etc..

UserAcceptance

TestTestDevelopment

Training & Education

ProductionSupport

Multiple Clones used by Production Support Analysts, DBA team and for internal training

DBA/

Note: All environments periodically refreshed from production

Database’s not just used by developers

Data Migration TeamData Mapping & CleansingDevelopment of ETL (Extraction, Transformation & Load) Routines…

UserAcceptance

TestUser

AcceptanceTestSystem

ValidationTesting

Data MigrationData

MigrationData Migration

Technical TeamsSystem IntegrationEnterprise Scheduling Automated TestingPerformance Testing…

Application Functional Specialists

Development of System Set-upBusiness Process ConfigurationSandbox…

Business Process OwnersApplication FunctionalProcess validationCRP (Conference Room Pilot) …

UserAcceptance

TestUser

AcceptanceTest

UserAcceptance

Test

AutomatedTestingAutomated

TestingAutomatedTesting

Source Timeflow

Source Timeflow

Source Database

Target HostVirtual

Database

NFS

Fiber

Fiber

RMAN over TCP