Integrating EnterpriseDB in your Oracle Environment

Insert focus area name here

1 Paper 112

IINNTTEEGGRRAATTIINNGG EENNTTEERRPPRRIISSEEDDBB IINN YYOOUURR OORRAACCLLEE EENNVVIIRROONNMMEENNTT

Lewis R Cunningham, PricewaterhouseCoopers

INTRODUCTION TO ENTERPRISEDB

The bulk of this paper, and the associated presentation, is based on writings I have written for my blog, An Expert's Guide to Oracle, http://blogs.ittoolbox.com/oracle/guide and from my forthcoming book, EnterpriseDB: The Definitive Reference (to be published in June 2007), http://www.rampant-books.com/book_2007_1_enterprisedb.htm

The majority of this paper is reproduced from Chapter 7, which, in the book contains additional text, examples and screen shots. The book also contains plenty of source code files that you are free to use and can modify as you see fit.

WHAT IS ENTERPRISEDB?

EnterpriseDB is an enterprise class database built on PostgreSQL. EnterpriseDB, the company, is adding enterprise class tools like a database management server, replication server, development tools, etc under a single umbrella for pricing and support. You also get Oracle SQL and PL/SQL compatibility.

What that boils down to is that EnterpriseDB, the company, has brought together several open source tools (database, replication, IDE) and added software they have developed (and are developing) like the management server, replication server and Oracle compatibility.

The current version of EnterpriseDB is based on PostgreSQL 8.1.5. PostgreSQL 8.2 has been released and an EnterpriseDB version using that version of PostgreSQL is being developed. EnterpriseDB will be upgraded at some point in the future to keep current with current versions of PostgreSQL.

Oracle, as we all know, comes with PL/SQL. In my (totally unbiased) opinion, PL/SQL is the best language ever built. It blows Java away in terms of write once, run anywhere.

PostgreSQL ships with several languages but the one I am most familiar with is PL/pgSQL. This is a very PL/SQLish language and many applications have been successfully written in PL/pgSQL.

EnterpriseDB, however, ships with EDB-SPL, the Superset Procedural Language. This language is syntactically compatible with PL/SQL, including package support. Basically, EDB-SPL is PL/SQL for PostgreSQL. How cool is that?!?!?!

To make things just a little nicer, EnterpriseDB also provides an Oracle compatible data dictionary (all_tables, all_views, etc). It's not complete and you don't get the V$ tables but it helps a lot when you're trying to get up to speed. Having a compatible data dictionary can dramatically reduce the learning curve.

WHERE DO I GET ENTERPRISEDB?

As this is not a marketing paper, I will not get into licensing or costs. You can get that information from EnterpriseDB, or get an overview when you buy my book.

Like Oracle Express, EnterpriseDB offers a free copy for light production use. Light production is defined as 1 CPU, 1 GB of Ram and less than 6 GB of disk. This gives you the opportunity to download and build some real applications before deciding if it's right for you.

You can get the software from http://www.enterprisedb.com/

You'll need to register with the site but when you register you get access to some online forums also.

IS ENTERPRISEDB THE ORACLE KILLER?

The short answer to this is NO! If DB2 and SQL Server combined can't do it, EnterpriseDB surely won't. Oracle used to be the database company, in the present they are a much larger entity: apps, java, tools, n-tier components, fusion, etc. EnterpriseDB simply provides an Oracle compatible database alternative that can be a much cheaper solution for certain uses.


2 Paper 112

Are you a RAC user? EnterpriseDB is probably not for you. Massive data warehouses? Stick with Oracle. Work group data marts or reporting databases? EnterpriseDB may be a cheaper solution for you.

The reason I personally like EnterpriseDB (besides cost) is that it is easy to maintain and has SQL and PL/SQL compatibility with Oracle. You could save some money by moving your reporting databases to MySQL or pure PostgreSQL. However, how many of your developers know the dialects of those databases? What kind of enterprise class tools do they provide? How many vendors will you need to contract with to get a decent level of support on all of the non-integrated tools you'll use with them?

EnterpriseDB provides the syntax compatibility and provides all of the tools you'll need in a single package.

Ok, that's the end of WHY. Now we get to the fun stuff of HOW.

INTEGRATING ENTERPRISEDB

There are many points where EnterpriseDB can be integrated into an environment but this paper will stick to the EnterpriseDB replication server. EnterpriseDB provides one-off replications via the EnterpriseDB SQL Studio IDE but that is more useful in a development or testing environment. The replication I will cover today is the EnterpriseDB Replication Server, which allows you to schedule replication.


3 Paper 112

ENTERPRISEDB REPLICATION SERVER ARCHITECTURE

ENTERPRISEDB REPLICATION SERVER

EnterpriseDB provides the answer to these needs with the EnterpriseDB Replication Server. The EnterpriseDB Replication Server provides a replication console to configure and monitor replication from Oracle to EnterpriseDB.

The EnterpriseDB Replication Server provides a repository-based solution that allows you to maintain the configuration of all of your replication jobs in a single location. You can configure multiple destinations for a single source and you can create multiple replication jobs from a single database.

Replication via the EnterpriseDB Replication Server is accomplished in a publish/subscribe model. The EnterpriseDB Replication Server provides an additional level of abstraction by inserting a replication repository between the sources and destinations.

For example, you can configure the replication server on a host different from both the source and target databases. The replication server can run on one host, the source on another and the target on yet another. The replication server is a one-stop replication hub. In a production environment, you will likely want to run the DBA Management Server for all of your instances and the replication server on an instance together, separate from your production instances.

NOTE: Most activity in the Replication Console is performed via a right-click, smart menu.


4 Paper 112

REPLICATION CONSOLE

The replication console is a Java application that will connect to your databases via JDBC. You will be connecting to three databases. Database 1 is the repository database (usually the local database), database 2 is the publisher (in this case Oracle) and database 3 is the target database (possibly the same as database 1 but not recommended in a production environment).

For the examples in this paper (in the book really), I will be using a local version of EnterpriseDB as both database 1 and database 3. I will be connecting to an Oracle 9iR2 database as database 2. I didn't choose 9iR2 for any particular reason. The example below would work just as well on a 10g instance and, while I didn't personally test it, probably on an 8i instance also.

I will be using the HR sample schema that comes with Oracle. I will replicate some of the tables in the HR schema but not all.

To start this working example, I will first verify my schemas. I have no HR schema in my EDB instance and I do have an HR instance in my Ora92 instance. If you do have an HR schema in your EDB instance already, right click on the schema and choose Drop Cascade to remove it.


5 Paper 112

CONFIGURE ORACLE ADMINISTRATOR

Now that I have verified that my databases are ready, I must perform some administrative set up in the Oracle database. EnterpriseDB replication is accomplished by creating data structures and triggers in the source database (which I will cover in detail below).

To allow this, I must configure a replication administrator account in the Oracle database. This administrative user requires specific grants to successfully create the tables and triggers that it needs to function.

You can run the commands below to create the replication administrator user. You can call the user anything that meets your naming standards. I chose to use rrepadmin (I got that from the EnterpriseDB example).

While not required, you may want to assign a default tablespace to the replication administrator account so that any created objects will be in that location.

The steps in creating the user are:

Create the user

Grant Create Any Trigger

Grant Select Access on Replicated Tables

CONFIGURE REPLICATION MANAGEMENT SERVER

The first step within the console it to configure the management server. This defines the database that will act as the repository for all of you replication jobs. In the Replication Console, right click on the Replication Servers node and choose Register Management Server.


6 Paper 112

Enter the information and press the Register button. After registering the replication management server, you should have an additional node in the navigator, DBA Management Server.

Registering the Replication Management Server creates three schemas in the identified database. The schemas are:

_edb_replicator_pub – Publisher information such as database, publisher/subscriber links and log records

_edb_replicator_sub - Subscription information

_edb_scheduler – Replication schedule information

STARTING THE SERVICES

Click on the plus (+) next to the DBA Management Server node and you will see the Publication Server and the Subscription Service in the Navigator. Right-click on both of those and choose Start Service.

CONFIGURE THE PUBLISHER (ORACLE)

Once you have the publication and subscription services running, right click on the Publication Service and choose Add Database. This will launch the Publication Service – Add Database dialog.


7 Paper 112

The publication service is the replication server's connection to the Oracle database. The user field is the schema that will contain the tables and triggers required for replication (the Oracle replication administration user).

In addition to configuring the connection, you may also configure a cleanup job. The cleanup job will remove successfully replicated data on a scheduled basis. The frequency of the cleanup job will be dependant on the load on your Oracle database and the volume of transactions being replicated.

CONFIGURE THE SUBSCRIPTION

The Subscription is the target database. This database may or may not be the same database as the DBA Management Server. In this example, it is the same but I would recommend a stand-alone database in a production environment.


8 Paper 112

To get to the Subscription Service – Add Database dialog, right click on the running Subscription Service node in the navigator and choose Add Database.

Enter the required information and choose save. You can choose test to verify that your connection information is correct.

CREATE PUBLICATION

A publication is a replication job. The replication job may be one or more tables. This step creates a series of tables, triggers and a package when successfully executed.

The tables created include definitions about which tables are to be replicated and what replication sets exist. The tables structures and table names may change from release to release so I will not detail them here. I don't think the specific tables are as important to understand as is understanding how the replication itself works.

For each table replicated, an identical table, with replication columns added, is created. This is referred to the as a shadow table. The cleanup job needs to be scheduled to run and cleanup these shadow tables periodically.

Each table to be replicated also has a set of triggers created for it. The replication server creates an Insert, Update and Delete trigger for each table. These triggers are responsible for inserting a record in the appropriate shadow table.

Obviously, this code is subject to change. DO NOT make changes to this code or you run the risk of breaking replication and/or not allowing yourself to upgrade.

The package that is created by the replication server contains procedures and functions that the replication server calls to replicate data and cleanup the tables.

There are performance considerations to think about before implementing any kind of replication. Trigger based replication has a particularly high performance impact for two reasons:

SQL to PL/SQL context switch – Anytime pure SQL has to switch to procedural code and any time procedural code switched to pure SQL, there is what is referred to as a context switch. This is a measurable delay in processing. It is minimal, but it is measurable.

Double IO – By taking each DML statement and inserting the data into another table, the trigger is doubling the apparent IO. One insert is equal to two inserts, one update is equal to one update and one insert, and one delete is equal to one delete and one insert.

On large data loads, the additional overhead may be unacceptable. Alternatively, you may choose to unschedule replication during the load and then manually resynchronize afterwards.


9 Paper 112

This is not a problem specific to the EnterpriseDB replication solution. This is a general problem that must be addressed by any trigger-based replication solution.

The EnterpriseDB Replication Console has addressed this issue by enabling and disabling the triggers intelligently. The triggers are turned on and off as needed to perform the replication. Even though this is better than just leaving the triggers enabled all the time, there is still additional overhead with this type of replication method.

If the triggers are too much of a burden, you may want to use Oracle Streams to capture changes, propagate them to a less critical instance, and then use EnterpriseDB replication to read form that instance. This is just one possible solution. The downside is that it would add additional Oracle license costs to create the additional instance if using Enterprise or Standard Editions. Depending on size, the free Oracle 10g XE database may be a good candidate for this use.

In this example, I will be choosing to replicate all of the HR tables, except the history tables. I could set up multiple publications to handle the replication but I am instead grouping all of these objects together into a single publication. This makes sense as some of the tables have foreign key relationships to other tables.

To get to the Create Publication dialog, right click on the Publication Database that was created above and choose Create Publication.

Enter a name for the publication. This name can be anything that makes sense to you and helps you remember the objects being replicated.

The Snapshot-only replication checkbox is the equivalent of Snapshot replication (which is discussed in my book in the section on Migrating from Oracle).

Check the boxes next to the tables to be replicated. If there were a need to limit the number of rows being replicated, say only employees in the Accounting department were to replicated to the Accounting workgroup server, the Filter Clause could be set appropriately to do so.


10 Paper 112

CREATE SUBSCRIPTION

A Subscription defines the target and links the target database to the published source. Enter the subscription name and connection information. If the Publication Name drop down is empty, press the Load button to refresh the list.

Press Create to create the subscription.

SNAPSHOT

Before turning on synchronization, a snapshot must be taken to create the replicated objects in the EnterpriseDB instance. The snapshot copies all of the replicated structures, including indexes and other supporting object.


11 Paper 112

For reporting, you may have different index requirements than you would in an Oracle OLTP or OLAP database. After running the snapshot, you may drop and/or create any indexes that you feel may help performance. Do not alter the structure of any replicated tables or you might break replication.

Examine any triggers that are copied and decide if they are needed in the replicated instance. In most situations, they are not required.

You can get to the Taking Snapshot dialog by right-clicking on the subscription identified above and choosing Take Snapshot.

Press the Take Snapshot button and examine the output window for any errors before proceeding. You will only need to take the snapshot one time.

SYNCHRONIZE

Synchronization is the process of catching the target up with the source database. You can get to the Synchronize dialog by right-clicking on the subscription and selecting Synchronize.


12 Paper 112

The synchronization process may run for several minutes depending on how much data is being synchronized. Wait for a successful completion before moving on to the next step.

THE SCHEDULE

While it is possible to manually synchronize your databases, doing so kind of defeats the purpose of having an enterprise class replication tool.

You will probably want to schedule automatic refreshes. Depending on your uses and needs, you may want a near, real-time refresh or a more relaxed refresh such as daily, weekly, or monthly. You may also schedule refreshes via crontab.

A point to make is that replication is never truly real-time. At best, transactions will experience some percentage of time delay. It takes time to enqueue captured transactions at the source, propagate to the target and then apply them at the target.

The Schedule Task Wizard is the tool within the Replication Console that will allow you to schedule your replication jobs.

You may choose a complete refresh on each run (Snapshot) or you may choose to synchronize. For this example, I will choose Synchronize.

As mentioned above, you may choose to schedule the replication continuously (near real-time), daily, weekly, monthly or via crontab.


13 Paper 112

In Continuous replication, you configure a refresh rate to gather and apply changes. Remember that the more often you refresh, the heavier the load on both the source and the target databases.

For a Daily replication task, you define the hour and minute that you want to begin the synchronization.

For a Weekly replication task, you define the hour and minute, as well as the days of the week that you want to begin the synchronization.

For a Monthly replication task, you define the hour and minute, the day of the month and which months that you want to begin the synchronization.

For crontab scheduling, you must define a valid crontab expression. You must also have cron running to schedule jobs via cron.

After successfully defining the scheduling method you require, press the Next button to arrive at the Schedule Summary dialog. Verify that the summary accurately reflects your choices and press the Finish button.

Once the schedule has been defined, the replication server will ensure that it executes the data movement as scheduled. You can view the replication log and real-time monitor by clicking to the lowest node beneath the subscription service.

VERIFY EDB SCHEMA

The last few steps in this example are to demonstrate how to verify a replication set up. The first step in verification is to ensure that the new HR schema exists in EnterpriseDB (our snapshot should have created that).

We can check this by opening Developer Studio and check that a new HR schema exists.

I can see in the figure above that the tables I selected for replication: countries, departments, employees, jobs, locations and regions, do exist in the edb database.

VERIFY ORACLE CONFIGURATION

The next step is to verify that the expected triggers were created for the HR objects and to verify that the shadow tables were created in the rrepadmin schema in Oracle.

INSERT SOURCE DATA


14 Paper 112

Once we are sure that all of the expected objects exist in both Oracle and EnterpriseDB, I will insert a record in Oracle and then monitor EnterpriseDB to see it arrive.

The screen shot below shows an Oracle SQL*Plus session connecting to the HR account and inserting a record into the HR.locations table. Before inserting the row, I did a describe to see which columns were not null (so I could successfully insert a row) and then selected any rows with a location ID of 9999 to make sure I was inserting a unique row.

SELECT TARGET DATA

After successfully inserting a row in Oracle, I check the EnterpriseDB instance to see if the new row arrives. The schedule was configured as continuous (execute every 10 seconds).

It will take at least 10 seconds before I see the transactions. I run the command several times before the transaction appears.

The error in the above screenshot was a typo on my part. Not an error in replication.


15 Paper 112

CONCLUSION

The EnterpriseDB Replication Server is integrated with the DBA management Server instance and is available for a fee with a premium subscription to the EnterpriseDB Network. The replication console is the GUI configuration and monitoring tool.

EnterpriseDB replication works via a repository based, publish/subscribe model. Publications and subscriptions are maintained in a central repository and the replication server maintains the appropriate connections and refresh schedules.

IMPLEMENTATION CHECKLIST

Implementation of Oracle to EnterpriseDB replication via the replication console can be reduced to a checklist:

Configure Oracle Replication Administrator

Grant create any trigger

Grant select on replicated tables

Configure Replication Management Server

Start Replication Services (Publish and Subscribe)

Configure the Publisher (Oracle)

Configure the Subscriber (EnterpriseDB)

Create the Publication (List of replicated objects)

Create the Subscriptions (Who gets what)

Take a snapshot (Create replicated objects in target)

Synchronize (Play catch-up)

Schedule

Verify

Test

If you follow the checklist in order, any errors will prevent you from continuing through the process until they are fixed. That will save you from having to redo steps later.

One thing to watch out for is performance impacts due to the triggers being added to the source database. Testing replication does not just mean testing data propagation; you must test all of the pieces to ensure a quality implementation.

The bulk of this paper, and the associated presentation, is based on writings I have written for my blog, An Expert's Guide to Oracle, http://blogs.ittoolbox.com/oracle/guide and from my forthcoming book, EnterpriseDB: The Definitive Reference (to be published in June 2007), http://www.rampant-books.com/book_2007_1_enterprisedb.htm

The majority of this paper is reproduced from Chapter 7, which, in the book contains additional text, examples and screen shots. The book also contains plenty of source code files that you are free to use and can modify as you see fit. If you find this paper helpful, please purchase the book.

Documents

Integrating EnterpriseDB in your Oracle Environment