Data Replication With Oracle GoldenGate Looking Behind The ... München... · 7 8 9 n . Supplemental Logging: Source Activate minimal database level supplemental logging on the source

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

– Looking Behind The Scenes –

Robert Bialek Principal Consultant Partner

Data Replication With Oracle GoldenGate

Who Am I

Principal Consultant, Partner and Trainer at Trivadis GmbH in Munich

– [email protected]

Focus:

– Oracle Database High Availability

– Database Architecture/Internals

– Backup/Recovery

– Troubleshooting/Performance Tuning

– Linux Administration

Trainer for the following Trivadis courses

– Oracle Grid Infrastructure (O-GRINF)

– Oracle Real Application Cluster (O-RAC)

– Oracle Data Guard (O-DG)

2 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes

Our company.

Trivadis is a market leader in IT consulting, system integration, solution engineering

and the provision of IT services focusing on and technologies

in Switzerland, Germany, Austria and Denmark. We offer our services in the following

strategic business fields:

Trivadis Services takes over the interactive operation of your IT systems.

O P E R A T I O N

16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes 3

COPENHAGEN

MUNICH

LAUSANNE

BERN

ZURICH

BRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

Data Replication With Oracle GoldenGate - Looking Behind The Scenes 4 16.07.2015

14 Trivadis branches and more than

600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:

CHF 5.0 / EUR 4 million

Financially self-supporting and

sustainably profitable

Experience from more than 1,900

projects per year at over 800 customers

16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes 5

Technology on its own won't help you. You need to know how to use it properly.

Agenda

6

1. Introduction

2. Oracle GoldenGate Architecture

3. Oracle GoldenGate Setup

4. Core Messages

16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes

Introduction


What is Oracle GoldenGate?

Oracle strategic Middleware software product used for data extraction and replication

– Use-cases: data/service high-availabillity, “zero-downtime” upgrades/migrations,

easy fallback (minimize risk)

Supports

– asynchronous data transfer (DML, DDL)

– heterogeneous sources and targets (databases, operating systems)

– different extraction, propagation and apply methods

– mappings, transformations, filtering, error handling, …

Topologies

– Active/passive, active/standby, active/active

– Broadcasting, cascading, consolidation

Source as well as the target environment requires a license

– Includes (EE): Oracle Active Data Guard and Oracle Global Data Services

(12.1.0.1+)


Why Oracle GoldenGate and not Streams?

Oracle Streams is deprecated in the Oracle 12c version (but still supported!)

– https://docs.oracle.com/database/121/UPGRD/deprecated.htm

According to the Oracle GoldenGate Statement of Direction

– http://www.oracle.com/technetwork/middleware/goldengate/overview/statement-of-

direction-gg-132120.pdf

Beginning with the version 12.1.0.2, the non-CDB architecture is also deprecated

– As of now, Streams is not supported with the new CDB architecture

Oracle Streams continues to be supported but will not be enhanced to

support any new database features in future releases. This means that

Streams will continue to support exactly what it supports today (11.2)

in future releases. It also means that any new database features, data

types, or capabilities will not be supported by Streams.


Oracle GoldenGate Architecture


Oracle GoldenGate Architecture: Big Picture


Source DB Target DB

Primary

(Regular)

Extract

Primary

(Passive)

Extract

Secondary

Extract

Data Pump

Collector

Replicat

Local Trail Remote Trail

Primary

(Alias)

Extract

Op

tio

nal

Oracle GoldenGate Instance Oracle GoldenGate Instance

Static

Configuration

Runtime

Configuration

Runtime

Configuration

Static

Configuration

Optional Optional

Manager

(VIP) Manager

(VIP)

MONITOR MONITOR

Primary Extract Group: Capture

Primary Extract can run in either

– Classic Capture mode (all database versions)

– Integrated Capture mode (full support: 11.2.0.3+)

Extracts and optionally filters/transforms the transactions from the source database

– Only for configured objects

Many Extracts per source database can operate at the same time

Only committed transactions (CSN) will be propagated to the target system (in

contrast to Oracle Streams…)

– Optionally written to a local trail (recommended)

– Without a local trail, the primary Extract send the data to the target system

– Not committed transactions will be discarded from memory or dirtmp directory

12

Primary

(Regular)

Extract

Primary

(Passive)

Extract

16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes

Source DB

Primary Extract Group: Classic Capture

Classic Capture mode

– Traditional mode, reads directly Oracle database redo logs

– Able to mine redo logs in ASM (two methods)

– All database versions supported

With some restrictions. For example no support for:

– BASIC, OLTP, Exadata Hybrid Columnar

compression

– PDML on RAC database

– CDB architecture!

DDL, TDE support – no out-of-the box support

Not transparent to RAC configuration changes (#threads)


Source DB

Primary

Extract

Secondary

Extract

Data Pump Local Trail

Redo Logs

OS

System Calls

Primary Extract Group: Integrated Capture

Integrated Capture mode

– Interacts with the database logmining server

– Required to capture data from a CDB architecture

– Fully transparent with RAC, TDE, ASM

– Full support for databases 11.2.0.3+

– Source database 11.2.0.4+ does not require

supplementary objects for DDL support

– Supports more complex data types than classic

capture

– Integrated capture deployments

• Local or a downstream deployment

The best choice for most deployments


Source DB

Primary

Extract

Secondary

Extract

Data Pump Local Trail

Redo Logs Logmaining

Server

LCR LCR

Replicat Process: Apply

The Replicat process can run in either

– Classic Replicat mode (nonintegrated) - all database versions

– Coordinated Replicat mode (nonintegrated) - all database versions, GoldenGate

12c required

– Integrated Replicat mode – database versions 11.2.0.4+

The Replicat process reads the remote trail, reconstructs

the SQL and applies them to the target database

– Optionally filters/transforms the data

The application of the data can be optionally delayed (DEFERAPPLYINTERVAL

parameter)


Target DB

Replicat

Remote Trail

Replicat Group: Classic/Coordinated Mode

Classic Replicat

– Constructs SQL statements from the trails

and applies the SQL through OCI

– Supported for all database versions

– Transactions are applied sequentially

(single-threaded)

Coordinated Replicat

– Transactions are applied in parallel (multi-threaded)

– There is 1 coordinator plus

many apply threads

– Supported for all database

versions


OCI

SQL

Target DB

Replicat

Remote Trail

OCI

SQL

Target DB

Remote Trail

Coordinator

Thread 1

Thread 2

Thread n

Replicat Group: Integrated Mode

Integrated Replicat

– Supported for target database

versions 11.2.0.4+

– Constructs LCRs and transmits

DMLs to the database inbound

server

Some changes (e.g. DDL) are not supported directly by the inbound server

– In those cases the Replicat process disconnects from the inbound server and

executes the transaction in direct apply mode through OCI

With many transactions applied in direct apply mode it might be necessary to switch

to nonintegrated Replicat


LCR

Target DB

Replicat

Remote Trail

Database

Inbound

Server

SQL

OCI

Capture/Apply: CDB & Pluggable Databases

Extract must operate in integrated mode

– Extract must connect to the CDB$ROOT container as a common user (c##)

– One Extract can capture from one or many PDBs to a single trail

Replicat can connect and apply only to one PDB (one Apply/PDB, connect using

service)

Source DB – CDB

CDB$ROOT PDB1 PDB2

Target DB – CDB

CDB$ROOT PDB1 PDB2

Primary Extracts

INTEGRATED Data Pumps Collectors

Replicats


common_user_prefix

(12.1.0.2)

Oracle GoldenGate Setup


Oracle GoldenGate: Setup Flow


Architecture

Design

Software Installation/

Patches Database Preparation

Manager Configuration

Primary Extract, Data

Pump, Replicat

Configuration Files

Target Database

Instantiation

Register/Start Primary

Extract, Data Pump

Register/Start Replicat Cluster Integration /

INIT Scripts

Tuning/Customizing/

Maintenance/…

1 2 3

6 5 4

7 8 9

n

Supplemental Logging: Source

Activate minimal database level supplemental logging on the source database

– To add row chaining information to the redo stream

Add schema or table level supplemental logging on the source database

– Schema level logging is required for DDL support

For all schema tables

– Activates unconditional supplemental logging for PK and conditional for unique/foreign

keys (scheduling columns) – all existing an future tables

For a table

– Depending on the use-case, different parameters are necessary

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

Database Preparation

3


ADD SCHEMATRANDATA <schema> [ALLCOLS]

ADD TRANDATA <schema>.<table> [COLS (…) | ALLCOLS]

Manager Configuration: Target/Source

Manager configuration (tune the values according to your needs)

– <GG_HOME>/dirprm/mgr.prm

Manager Configuration

4

PORT 7809

DYNAMICPORTLIST 7810-7910

AUTOSTART ER *

AUTORESTART ER *, RETRIES 4, WAITMINUTES 4, RESETMINUTES 60

PURGEOLDEXTRACTS <...>/dirdat/*ext*/*t*, USECHECKPOINTS, &

MINKEEPHOURS 4

LAGREPORTMINUTES 30

LAGINFOMINUTES 5

LAGCRITICALMINUTES 10

unreserved and unrestricted


Primary Extract Configuration

Create the configuration file for the primary Extract (source)


Pump, Replicat

Configuration Files

5


EXTRACT app1ext1

SETENV (ORACLE_HOME = "/u00/app/oracle/product/11.2.0.4")

USERIDALIAS ggadm_app1_11204

LOGALLSUPCOLS

UPDATERECORDFORMAT COMPACT

DDL INCLUDE MAPPED EXCLUDE OBJNAME APP1.T4 &

EXCLUDE OBJNAME APP1.T5

ENCRYPTTRAIL AES256

EXTTRAIL <...>/dirdat/app1ext1/lt

SEQUENCE APP1.*;

TABLE APP1.*;

A Master-Key Wallet is

necessary

You need GoldeGate

Credential Store

Secondary Extract/Replicat Configuration Files

Create the configuration file for the Data Pump Extract (source)

Create the configuration file for the Replicat group (target)


EXTRACT app1pmp1


RMTHOST ggmgr-cl121.trivadis.com, MGRPORT 7809,ENCRYPT AES256,&

COMPRESS, COMPRESSTHRESHOLD 1200

RMTTRAIL <...>/dirdat/app1rep1/rt

SEQUENCE APP1.*;

TABLE APP1.*;

Not necessary if the

trail is already

encrypted

REPLICAT app1rep1

SETENV (ORACLE_HOME = "/u00/app/oracle/product/12.1.0.2")

DBOPTIONS INTEGRATEDPARAMS(parallelism 6)


DDLOPTIONS REPORT

ASSUMETARGETDEFS

MAP APP1.*, TARGET APP1.APP1.*;

Mapping: non-CDB CDB


Pump, Replicat

Configuration Files

5

PDB

Primary/Secondary Extract Group

Register the primary Extract with the database (integrated)

Create the primary Extract group (integrated)

Add a local trail an associate it with the primary Extract

Add the Data Pump Extract


DBLOGIN USERIDALIAS gg_app1_11204

REGISTER EXTRACT app1ext1 DATABASE [CONTAINER (APP1)]

Extract APP1EXT1 successfully registered with database at SCN 3630440.

GoldeGate Credential

Store

ADD EXTRACT app1ext1, INTEGRATED TRANLOG, BEGIN NOW

Register/Start Primary

Extract, Data Pump

6

ADD EXTTRAIL <...>/dirdat/app1ext1/lt, EXTRACT app1ext1

ADD EXTRACT app1pmp1, EXTTRAILSOURCE <...>/dirdat/app1ext1/lt

Secondary Extract/Replicat Group

Add a remote trail and associate it with the secondary Extract

Add the Replicat group (integrated)

Start the Replicat on the target with the correct CSN


ADD RMTTRAIL <...>/dirdat/app1rep1/rt, EXTRACT app1pmp1

ADD REPLICAT app1rep1, INTEGRATED,

EXTTRAIL <...>/dirdat/app1rep1/rt

GGSCI> START REPLICAT app1rep1, AFTERCSN 3693221

Register/Start Replicat

8

Oracle GoldenGate: Demo Environment


Source DB: 2-nodes RAC 11.2.0.4

Schema: APP1

GoldenGate version: 12.1.2.1.2

Active/Passive data replication

CDB$ROOT APP1 APP2

Primary Extract

INTEGRATED

Target DB: CDB 2-nodes RAC 12.1.0.2

PDB: APP1, Schema: APP1

GoldenGate version: 12.1.2.1.2

Active/Passive data replication

APP3

Schema: APP1

Local Trail

Secondary Extract

Data Pump

RemoteTrail

Collector

Replicat

INTEGRATED

Core Messages


Core Messages

GoldenGate is a very powerful replication software framework

For most deployments the integrated Extract/Replicat is the best choice

Supplemental logging generates more undo and redo

– In some cases you need even to supplementally log all columns!

Script all the configuration steps

– Very useful if you need to re-setup the whole environment

You don’t need to replace your current Streams installation ASAP by Oracle

GoldenGate , but

– Streams is deprecated in the version 12c and will not be enhanced in future

releases

– There is a framework to ease the migration from Streams to GoldenGate

(streams2ogg)

– As of now Streams is not supported with CDB architecture

– For new projects it might be a better choice (licensing)!


Questions and answers…

Robert Bialek

Principal Consultant

Partner

Tel. +49 89 99275930

[email protected]


Documents

Data Replication With Oracle GoldenGate Looking Behind The ... München... · 7 8 9 n . Supplemental Logging: Source Activate minimal database level supplemental logging on the source