Golden Gate an Overview

8/4/2019 Golden Gate an Overview

1/39

GoldenGate An Overview


2/39

Agenda

3

4 Conclusion

1 GoldenGate - Introduction

2 Architecture

Implementation


3/39

Introduction


4/39

GoldenGate Software Inc. is a leading provider of high availability and real-time dataintegrationsolutions. GoldenGate helps mission-critical systems maintain 24/7 operations to meet

end-userdemand for continuous access. GoldenGate also helps organizations to immediately

distribute

critical data across the enterprise, to enable timely and accurate decision making. Inshort,GoldenGate solutions enable real-time access to real-time information.

GoldenGate solutions are powered by GoldenGates Transactional Data Management(TDM)

platform, which provides continuous capture, routing, transformation, and delivery oftransactional data between heterogeneous systems in real time and with minimal

overhead.

GoldenGate - Introduction


5/39

SOLUTIONS FOR HIGH AVAILABILITY AND DISASTER TOLERANCE

GoldenGate High Availability and Disaster Tolerance solutions eliminate downtimecaused by

unplanned and planned outages and boost the performance and scalability of systemsburdened

by the tremendous growth of data or usage demands.

GoldenGate Live Standby significantly improves recovery time for business-criticalsystems.GoldenGate Zero-Downtime Operations enables uninterrupted business operationsduringnecessary system upgrade, migration, and maintenance activities.GoldenGate Active-Active improves performance and reliability for two or moredatabases by enabling workload balancing.Database Tiering allows companies to ensure the highest performance of theirproductionsystems while still easily supporting necessary read-only activities.

GoldenGate Introduction (Contd)


6/39

SOLUTIONS FOR REAL-TIME DATA INTEGRATION

GoldenGate Real-Time Data Integration solutions provide real-time change data to datawarehouses, operational data stores, reporting systems, or other OLTP-supporting

databases without batch windows and with minimal performance impact.

GoldenGate Real-Time Data Warehousing enables continuous, real-time capture,transformation, and delivery of the most recent change data between OLTP systemsand thedata warehouse. It can also integrate with existing ETL solutions.GoldenGate Live Reporting off-loads reporting activity from the production database

to alower-cost secondary system, where the most current data is always available tosupportreal-time reporting needs.GoldenGate Transactional Data Integration offers point-to-point data integrationbetween

OLTP systems in real time and with minimal impact.



7/39

Goldengate TRANSACTIONAL DATA MANAGEMENT (TDM)

GoldenGate TDM software moves transactional data between heterogeneous systems inreal

time, with transactional integrity.

Key Features:Real-Time Data As new transactions come in, the data is immediately captured,

transformed(as needed), and delivered to other systems with only sub-second latency.Continuous Availability GoldenGate works without requiring system interruption or

outage windows.Heterogeneous Support The underlying databases and platforms for criticalapplications can

be heterogeneous, allowing significant flexibility for the IT department.High Performance with Low Impact GoldenGate can support the movement ofthousands of transactions per second yet imposes negligible impact on the source and

targetsystems.



8/39

Architecture


9/39

Sources Supported

Databases: Platforms:Oracle HP NonStopDB2 HP-UXMicrosoft Sql Server HP TRU64Sybase ASE HP OpenVMSIngres Windows 2000, 2003, XPTeradata LinuxEnscribe Sun SolarisSQL/MP IBM AIX

SQL/MX IBM z/OS

Architecture


10/39

Targets Supported

Databases: Platforms:Oracle HP NonStopDB2 HP-UXMicrosoft Sql Server HP TRU64Sybase ASE HP OpenVMSIngres Windows 2000, 2003, XPTeradata LinuxEnscribe Sun SolarisSQL/MP IBM AIXSQL/MX IBM z/OS

MySQLHP NeoviewNetezzaGreenplumJMS Message BusETL ServersAny ODBC compatible database

Architecture (Contd)


11/39

The following are the components & processes involved in the GoldenGate

Capture (Extract)Capture module reads the transaction logs (redo or archive logs) of the desired sourcedatabase.Captures committed transactions - insert, update & delete operations - and maintainstransaction integrity.Immediately writes the transactions to GoldenGate external trail file for distribution totargetsProcess efficiently and with low impact - does not create additional work in the databaseor create triggersOffers selective capture and delivery capability: Advanced filtering at thedatabase/schema, table, row or column level or based on operation type, matchedpatterns and value thresholds

Source/Target Trail FilesTrail Files contain the changed data operations in a transportable, platform-independent, universal data format.Exist outside of the databases to minimize overhead and enable improved reliability andheterogeneity

Help to checkpoint data, to ensure data accuracy and completeness between sourceand target systems



12/39

Data Pump (Capture Once, Apply Many)

The GoldenGate Pump reads the trail file and distributes the data to the designatedtarget databaseMultiple pumps may be configured to send the data to multiple targets, even differentdatabases (ie. capture from Oracle, apply to Teradata & Sql Server)

Routing

GoldenGate send data over TCP/IP - this means no distance constraints betweensource and target systems!Encryption may be appliedAdditional compression may be applied as desiredCan send thousands of transaction operations per second (TPS)Extremely efficient, low-bandwidth implementation

Delivery (Replicat)GoldenGate Delivery reads changed data transactions from the Trail FileContinuously applies changed data to the target database using the native Sql for thatRDBMSEnsures transaction integrity by applying transactions in the same order that they werecommitted on the source system



13/39

Transaction commit boundaries are preserved throughout each step of the datamovementDelivery also supported to non-RDBMS targets via flat file integrationSupports high-volume environments while keeping source and target "in sync"All of this happens with only sub-second latency!!

ManagerManager is GoldenGates control process. Manager must be running on each systeminvolved in GoldenGate synchronization.

Manager performs the following functions:

Monitor and restart GoldenGate processes.

Issue threshold reports, for example when Extract processing slows down or whensynchronization latency increases.Maintain trail files and logs.Allocate data storage space.Report errors and events.Receive and route user requests from the user interface.



14/39

Collector

Collector is a process that runs in the background on the target system to receive datasentacross the TCP/IP network and write it to the trail or extract file. Typically, Manager startsCollector automatically



15/39

Implementation


16/39

GoldenGate can be installed and configured quickly and easily in justfour easySteps:

1. Prepare the environment

2. Configure Change Capture

3. Initial Data Load

4. Configure Change Delivery

Implementation


17/39

Step 1: Set up each system

Install GoldenGate

Start GoldenGate Manager

Generate source definitions for heterogeneous configurations

Add supplemental log data from Oracle or MS SQL log-basedsolution

Add loop-detection for bi-directional configuration

Implementation - Prepare the environment


18/39

Step 2: Capture changes to the source

Add Extract checkpoint into the transaction log

Configure Extract parameter file

Define GoldenGate trails

Start Extract

Implementation Configure ChangeCapture


19/39

Step 3: Load in the starting data

Configure Extract Parameter File

Configure Replicat Parameter File

Start Extract

Start Replicat

Implementation Initial Data Load

l


20/39

Step 4: Apply changes to the target database

Add Replicat checkpoint

Configure Replicat parameter file

Start Replicat

Implementation Configure ChangeDelivery


21/39

Unix/Linux1. FTP the file to the UNIX system in binary mode, and place it in the directory where

you want GoldenGate to be installed.2. Extract the tar file (use the gzip or tar options appropriate for your system).

Syntax: /ggs> gzip -dc filename.tar.gz | tar -xvf All GoldenGate files are placed in the current directory.

3. Download the GoldenGate password file in ASCII mode and save it with a file nameof pw(lowercase, no file extension) in the same directory where you installed theGoldenGate software.

4. From the GoldenGate directory, run the GGSCI program.GGSCI

5. In GGSCI, issue the following command.CREATE SUBDIRS 6. Issue the following command to exit GGSCI.

EXIT 7. Configure the Manager process

Installation Easy Steps


22/39

Windows

1. Unzip the downloaded file(s) using PKUNZIP or WinZip2. Move the files in binary mode to a folder (for eg: C:\GGS) on the drive where you

want to install GoldenGate.Do not install GoldenGate into a folder that contains spaces in its name, for example GoldenGate Software. The application references path names, and the operating system does not support path names that contain spaces, whether or not they are within quotes.

3. Download the GoldenGate password file in ASCII mode and save it with a file nameof pw (lowercase, no file extension) in the GoldenGate folder (for eg: C:\GGS).

4. From the GoldenGate folder, run the GGSCI programGGSCI

5. In GGSCI, issue the following command.CREATE SUBDIRS

6. Issue the following command to add service & event messages to the system registryC:\GGS> INSTALL ADDSERVICE ADDEVENTS USER PASSWORD

7. Issue the following command to exit GGSCIEXIT

By default, the errors logged are generic. To enable the display of more specific error content, copy thecategory.dll &

ggsmsg.dll files from the GoldenGate installation directory to the SYSTEM32 directory. The enhanced output

Installation Easy Steps (Cond)


23/39

Configuring Manager

To configure Manager, create a parameter file.From the GoldenGate directory, run the ggsci program. This is the GoldenGateSoftwareCommand Interface, commonly known as GGSCI.In GGSCI, issue the following command.EDIT PARAMS MGR

The only required parameter for Manager is PORT, which defines the port number onwhich Manager runs on the local system. The default port is 7809. You must specifyeither the default port or another port. This must bean unreserved, unrestricted port.GGSCI uses the port to request Manager to start processes. The Extract processuses the port to request Manager to start a remote Collector process or an initial-loadReplicat process. PORT

Note: The port number also must be specified with the MGRPORT argument of the Extract parameter RMTHOST.

To run Manager from GGSCIFrom the GoldenGate directory, run GGSCI.In GGSCI, issue the following command.

START MANAGER

Installation Easy Steps (Cond)


24/39

Using GoldenGate parameter filesMost GoldenGate functionality is controlled by means of parameters specified in parameter files.A parameter file is an ASCII file that is read by an associated process. GoldenGate uses twotypes of parameter files: a GLOBALS file and runtime parameter files.

Overview of the GLOBALS fileThe GLOBALS file stores parameters that relate to the GoldenGate instance as a whole. This is in

contrast to

runtime parameters, which are coupled with a specific process such as Extract. Parameters in theGLOBALSfile apply to all processes in the GoldenGate instance, but can be overridden by specific process

parameters.Once set, GLOBALS parameters are rarely changed, and there are far fewer of them than runtime

parameters.

A GLOBALS parameter file is required only in certain circumstances and, when used, must be createdfromthe command shell before starting any GoldenGate processes, including GGSCI. The GGSCI program

readsthe GLOBALS file and passes the parameters to processes that need them.

To create a GLOBALS file

1. From the GoldenGate home location, run GGSCI and enter the following command (or you cansimply open a file in an ASCII text editor).

GoldenGate Parameter Files


25/39

3. Save the file. If you used a text editor, save the file with the name GLOBALS (all capital letters onUNIX-based systems) without a file extension in the root GoldenGate directory (where the

binaries are stored). If you created the file in GGSCI, the file is saved that way automatically. Donot move this file.Exit GGSCI and start a new session before issuing commands or starting processes thatreference theGLOBALS file.

Overview of runtime parameters

Runtime parameters give you control over the various aspects of GoldenGate synchronization, suchas:Data selection, mapping, transformation, and replicationDDL selection, mapping, and replication (where supported)Error resolutionLoggingStatus and error reportingSystem resource usageStartup and runtime behavior

Note: There can be only one active parameter file for the Manager process or an Extract or Replicat group.

There are two types of parameters: global (not to be confused with GLOBALS parameters) and table-specific.



26/39

MAP parameter statements that specify the database objects whose data or DDL is to beprocessed, and it should be listed only once in the file. When listed more than once, only the last

instance of the parameter is active. All other instances are ignored.Table-specific parameters enable you to create different processing rules for different sets oftables that are specified with TABLE or MAP parameter statements. GETINSERTS andIGNOREINSERTS in Figure 5 are examples of table-specific parameters. Table-specificparameters take effect in the order that each parameter is listed in the file.

The following are examples of basic parameter files for Extract and Replicat.

Sample Extract parameter fileEXTRACT captUSERID etluser, PASSWORD *********DISCARDFILE /ggs/capt.dsc, PURGERMTHOST sysb, MGRPORT 7809RMTTRAIL /ggs/dirdat/aaTABLE fin.*;TABLE sales.*;



27/39

Sample Replicat parameter fileREPLICAT delivUSERID etluser, PASSWORD ****SOURCEDEFS /ggs/dirdef/defsDISCARDFILE /ggs/deliv.dsc, PURGE

GETINSERTSMAP fin.account, TARGET fin.acctab,

COLMAP (account = acct,balance = bal,branch = branch);

MAP fin.teller, TARGET fin.telltab,WHERE (branch = NY );

IGNOREINSERTSMAP fin.teller, TARGET fin.telltab,WHERE (branch = LA );

Changing a parameter fileYou can change parameter settings after processing has started. You can change any Manager

parameter

except the port number while the Manager process is running; to change the port number, you muststop



28/39

running, changes to parameter settings can have unpredictable and adverse consequences,especially if you are

adding tables or changing mapping or filtering rules.

To change Manager parameters1. If changing the port number, stop the Manager process by using the STOP MANAGER command

in GGSCI. If you are not changing the port, do not stop Manager.STOP MANAGER

2. Open the parameter file by using a text editor or the EDIT PARAMS command in GGSCI.EDIT PARAMS mgr

3. Make the edits, and then save the file.4. Do one of the following:

Refresh the Manager process if it is still running,REFRESH MANAGER

Start Manager if it is not running.

START MANAGER To change Extract or Replicat parameters1. Stop the process.

STOP {EXTRACT | REPLICAT} 2. Open the parameter file by using a text editor or the EDIT PARAMS commanding GGSCI.

EDIT PARAMS mgr 3. Make the edits, and then save the file.4. Start the process.

START {EXTRACT | REPLICAT}



29/39


30/39

Define GoldenGate queues2. Start Extract and verify its running

Step 2: Perform initial data load using any of the 3 methods discussed above and follow the belowsteps

1. Configure Extract to read directly from the source database via the SOURCEISTABLE parameter2. Configure Replicat with the parameters SPECIALRUN3. Start the initial data Extract

Step 3: Start the delivery of changesWhen the initial data load completes1. Configure Replicat on the target system with the HANDLECOLLISIONS and END parameters2. The END parameter is the time recorded for the completion of the source file Extract process3. Start the Replicat delivery process4. When the Replicat process stops, remove the END and HANDLECOLLISIONS parameters

5. Start the Replicat process for incremental data synchronizationQueue Data Example parameter fileExample Extract parameter file

SOURCEISTABLE USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809

RMTFILE /ggs/account.dat, purge TABLE SALES.ACCOUNT;

Initial Data Load


31/39

Example Replicat parameter fileSPECIALRUN END RUNTIME USERID tgtuser, PASSWORD ********* ASSUMETARGETDEFS EXTFILE /ggs/account.dat MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT;

Queue Data ExecutionTo execute the Extract and Replicat process, execute the EXTRACT and REPLICAT programs from

thecommand shell.

WindowsC:\GGS> extract paramfile dirprm\initext.prm

C:\GGS> replicat paramfile dirprm\initrep.prm Unix$ /ggs> extract paramfile dirprm/initext.prm $ /ggs> replicat paramfile dirprm/initrep.prm

Initial Data Load


32/39

Direct Load Example parameter fileExample Extract parameter file

EXTRACT INITEXT USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTTASK REPLICAT, GROUP INITREP TABLE SALES.ACCOUNT;

Example Replicat parameter fileREPLICAT INITREP USERID tgtuser, PASSWORD ********* ASSUMETARGETDEFS MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT;

Direct Load ExecutionTo execute the Extract and Replicat process, first define the groups within GGSCI using the

SOURCEISTABLEor SPECIALRUN arguments

GGSCI> ADD EXTRACT INITEXT, SOURCEISTABLE GGSCI> ADD REPLICAT INITREP, SPECIALRUN GGSCI> START EXTRACT INITEXT

Initial Data Load


33/39

Direct Bulk Load Example parameter fileExample Extract parameter file

EXTRACT INITEXT USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTTASK REPLICAT, GROUP INITREP TABLE SALES.ACCOUNT;

Example Replicat parameter fileREPLICAT INITREP USERID tgtuser, PASSWORD ********* BULKLOAD ASSUMETARGETDEFS MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT;

Direct Bulk Load ExecutionTo execute the Extract and Replicat process, first define the groups within GGSCI using the

SOURCEISTABLEor SPECIALRUN arguments

GGSCI> ADD EXTRACT INITEXT, SOURCEISTABLE GGSCI> ADD REPLICAT INITREP, SPECIALRUN

GGSCI> START EXTRACT INITEXT

Initial Data Load


34/39

ExtractExtract can be configured to capture changed data from Oracle online redo and archive logs DB2 logs MS SQL logs GoldenGate log table produced by triggers

Extract can distribute data to other remote systems Directly from GoldenGate trails

Extract can be configured for initial data load Directly from source tables

Extract ConfigurationTo configure Extract to capture changes from transaction logs Add initial Extract checkpoint into the transaction log (use GGSCI ADD EXTRACT command) Configure Extract parameter file (use GGSCI EDIT PARAMS command) Define GoldenGate trails (use GGSCI ADD RMTTRAIL or ADD EXTTRAIL commands) Start Extract (use GGSCI START EXTRACT command)

Example Oracle:GGSCI> EDIT PARAMS ODS

EXTRACT ODS

USERID etluser, PASSWORD *********

Change Capture and Delivery


35/39

RMTHOST targethost, MGRPORT 7809 RMTTRAIL /ggs/diirdat/rt TABLE SALES.ORDERS; TABLE SALES.INVENTORY;

GGSCI> ADD EXTRACT ODS, TRANLOG, BEGIN NOW GGSCI> ADD RMTTRAIL /ggs/diirdat/rt, EXTRACT ODS GGSCI> START EXTRACT ODS

Example DB2 and SQL SERVER:GGSCI> EDIT PARAMS ODS

EXTRACT ODS SOURCEDB dsn, USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809

RMTTRAIL /ggs/diirdat/rt TABLE SALES.ORDERS; TABLE SALES.INVENTORY;

GGSCI> ADD EXTRACT ODS, TRANLOG, BEGIN NOW GGSCI> ADD RMTTRAIL /ggs/diirdat/rt, EXTRACT ODS GGSCI> START EXTRACT ODS



36/39

GoldenGate Trails :As Extract captures data, it stores the captured data and a description of that data, into files known asGoldenGate trails. Trail files are created as needed and are aged automatically to allow processing to

continuewithout interruption for file maintenance. A trail can exist on the source or target system, depending on

howyou configure GoldenGate. On the local system it is known as an extract trail (or local trail). On a

remotesystem it is known as a remote trail. Only one Extract process can write to a trail. GoldenGate writes

data tothe trail in universal data format, a proprietary format which allows it to be exchanged rapidly and

accuratelyamong heterogeneous databases. The records in GoldenGate trails contain header and data areas.

The headercontains information about the transaction environment, and the data area contains the actual data

values thatwere extracted. All file names in a trail begin with the same two characters, which you assign when

you createthe trail. As files are created, each name is appended with a unique six-digit serial (sequence) number

from000000 through 999999, for example \ggs\dirdat\tr000001. By default, trails are stored in the dirdatsub-directory of the GoldenGate directory

Replicat



37/39

Replicat - ConfigurationTo configure Replicat to apply changes from GoldenGate trails to the target database: Create checkpoint table (use GGSCIT ADD CEHECKPOINTABLE command) Setup parameter file for Replicat (use GGSCI EDIT PARAMS command) Add initial Replicat checkpoint into GoldenGate trails (use GGSCI ADD REPLICAT command) Start the Replicat process (use GGSCI START REPLICAT command)

Example:

GGSCI> DBLOGIN USERID etluser PASSWORD pw GGSCI> ADD CHECKPOINTTABLE ggs.checkpt GGSCI> EDIT PARAMS REPORD

REPLICAT REPORD TARGETDB dsn USERID etluser, PASSWORD ********* (For Oracle)USERID etluser, PASSWORD ********* (For DB2 or SQL SERVER)

ASSUMETAREGTDFS DISCARDFILE /ggs/dirrpt/REPORT.dsc, APPEND MAP SALES.ORDERS, TARGET SALES.ORDERS; MAP SALES.INVENTORY, TARGET SALES.INVENTORY;

GGSCI> ADD REPLICAT REPORD, EXTRAIL /ggs/dirdat/rt GGSCI> START REPLICAT REPORD



38/39

Conclusion


39/39

Other Information:There are several other commands, parameters, functions, etc used during the Manager, Extract,Replicat

Processes which we are not going in detail here. Pls refer the GGS_WinUnix_Reference_9.5.pdf &GGS_WinUnix_QuickRef_9.5.pdf documents comes with the GoldenGate software.

References:GoldenGate websiteGoldenGate Fundamentals Student Guide study materialGoldenGate reference manuals

Conclusion

Documents

Golden Gate an Overview