ODI Advanced

Advanced Topics in ODI-EE: Hands on Lab

Advanced Topics in ODI-EE: Hands on Lab Page 1 of 26 7/9/2010 6:08:00 PM (version 1.0) Oracle, All rights reserved

Advanced Topics in Oracle Data Integrator: Hands-on Lab



Supported Version: Oracle Data Integrator 11.1.1.3

Objectives: This lab will highlight to Developers, DBAs and Architects some of the best practices for implementing techniques to improve performance for real-time data warehousing using Oracle Data Integrator Enterprise Edition (ODI-EE). In this lab participants will walk through the steps to enable change data capture on a source relational table, then deploy and manage these implementations to load and transform records to a relational target.

Prerequisite: Basic knowledge of Oracle Data Integrator, Installed Oracle Data Integrator software and Oracle Database instance.

S308461 - Using ODI-EE and Oracle BI for Real-Time Insight: Hands on Lab



Table of Contents Part 1: Configuring a Model for CDC ........................................................................... 4 Part 2: Creating a Package to Set Up CDC ................................................................. 8 Part 3: Creating the Integration Flows to Consume the Changes ............................... 13 Part 4: Run the CDC Set Up and Consumption Processes in Production ................... 22 Summary .................................................................................................................. 26



Part 1: Configuring a Model for CDC In this exercise you will configure an existing source Oracle data model to support ODI CDC feature. The CDC mode used will be Consistent-Set CDC. Oracle Log Miner will be used to track the changes.

1. Double-click the icon Reset_ODI.bat from the desktop. This will reset the files for all Hands-on Labs to the initial state. Warning: If you have custom ODI projects in the repository on this machine, they will be deleted.

2. Open the Oracle Data Integrator Studio using the Start Menu > Oracle > Oracle Data Integrator > ODI

Studio menu.

3. Go to the Designer Navigator and press Connect to Repository

4. Select HOL Advanced from the drop-down list of logins and press OK.

5. Open the Designer navigator menu and make sure that the Default Execution Context is set to the

Development context (not Production).



6. Open the Models accordion in the Designer navigator by clicking on the label.

7. Right-click on the Oracle CDC Source model and select Changed Data Capture > Add to CDC. Click Yes to add all the tables to the CDC.

8. Double-click on the Oracle CDC Source model. The editor for this model opens.



9. Go to the Journalizing tab.

10. Select the Consistent Set journalizing mode, click OK to close the popup window then the JKM Oracle 10g Consistent (Log Miner) knowledge module. Set the AUTO_CONFIGURATION KM option to Yes.

11. Go to the Journalized Tables tab, select the SRC_REGION table and click on the Remove

from CDC button.

12. Click on the Reorganize button to order the tables according to their foreign-keys. Press yes on all confirmation dialogs. The datastores are automatically organized as shown below.



Note: Consistent Set Journalizing tracks changes to a group of the model's datastores, taking into account the referential integrity between these datastores. The group of datastores journalized in this mode is called a Consistent Set.

13. Click Save( ) on the toolbar to save the Model.



Part 2: Creating a Package to Set Up CDC In this exercise you will create a package that sets up the CDC infrastructure in a given context.

1. In the Designer navigator, Click on the Projects accordion.

2. Expand the Advanced Topics in ODI-EE project and then expand the First Folder folder.

3. Select the Packages node, right-click and select New Package. A package editor opens.

4. Enter the following package name: Tutorial Part 2 - CDC Setup

5. Go to the Diagram tab.

6. Drag and drop into the diagram the Oracle CDC Source data model from the Designers Models tree view. A new step appears in the diagram, named after your data model.



7. Click on this step. In the Properties panel: Select Journalizing Model in the Type drop-down list

Click the Start and Add Subscribers checkboxes. Enter CONSUMER_1 in the Subscribers field, and then click Add. Enter CONSUMER_2 in the Subscribers field, and then click Add.

8. Click Save( ) on the toolbar to save the package.

9. Click on the Execute( ) button on the toolbar.

10. In the Execution window, select Development for the context and then press OK.



11. Press OK when the Session Started window appears.

12. Open the Operator navigator.

13. In the Operator, select the Session List tab, and expand the All Executions node.



14. Check that the last session ran correctly. You can review the steps and tasks that have activated the CDC process.



Part 3: Creating the Integration Flows to Consume the Changes In this exercise you will create the integration flows that will consume the changes captured using CDC.

1. In the Designer navigator, open the Projects accordion.


3. Expand the Procedures node.

4. Right-click on the CDC Changes procedure and select Execute.



7. The procedure did the following: 1 insert in SRC_CITY. 1 insert in SRC_CUSTOMER. 1 update in SRC_CITY. 2 updates in SRC_CUSTOMER.



8. Expand the Interfaces node and double-click on Load SRC_CITY CDC.

9. Go to the Mapping tab.

10. Click on the source datastore SRC_CITY and check the Journalized Data only checkbox. This ensures that ODI will load only the changed data (latest inserts, updates or deletes) from the source instead of the entire source dataset.

11. Click on the filter and modify the filter expression from JRN_SUBSCRIBER = 'SUNOPSIS' to JRN_SUBSCRIBER = 'CONSUMER_1'



12. Right-click on the filter and select Data to review the changed data.

13. Save and close the interface.

14. Double-click on Load SRC_CUSTOMER CDC in the Interfaces node.

15. Go to the Mapping tab.

16. Click on the source datastore SRC_CUSTOMER and check the Journalized Data only checkbox.



17. Click on the filter and modify the filter expression from JRN_SUBSCRIBER = 'SUNOPSIS' to JRN_SUBSCRIBER = 'CONSUMER_1'.

18. Right-click on the filter and select Data to review the changed data.

14. Click on the Models tab in the tree view.



19. Expand the Flat Files Sources model.

20. Expand the Flat Files Sources model and drag and drop the SRC_SALES_PERSON.txt datastore from the Designers Models tree view into the source panel of the interface.

21. Create a join between the SRC_CUSTOMER table and the SRC_SALES_PERSON.txt file. To create it, drag and drop the SALES_PERS_ID column from the SRC_CUSTOMER table into the SALES_ID column from the SRC_SALES_PERSON.txt file. Note: The join expression is: SRC_CUSTOMER.SALES_PERS_ID = SRC.SALES_ID



22. We now need to map the remaining target column: - SALES_PERS: map FIRST_NAME and LAST_NAME from the SRC_SALES_PERS.txt

file into it. The two source columns should be concatenated with an extra space between them then we use the LTRM and RTRIM to remove any whitespaces. Note: The mapping expression is LTRIM(RTRIM(SRC.FIRST_NAME)) || ' ' || LTRIM(RTRIM(SRC.LAST_NAME))

23. Save and close the interface.


25. Select the Packages node, right-click and select New Package.

A package editor opens.

26. Enter the following package name: Tutorial Part 3 - Load CDC Data

27. Go to the Diagram tab.


29. Click this step. In the Properties panel: Select Journalizing Model in the Type drop-down list.

Click the Extend Window and Lock Subscribers checkboxes.



Enter CONSUMER_1 in the Subscribers field, and then click Add.

30. Drag and drop the Load SRC_CITY CDC and Load SRC_CUSTOMER CDC interfaces from the Designers Projects tree view.


32. Click this step. In the Properties panel: Select Journalizing Model in the Type drop-down list. Click the Purge Journal and Unlock Subscribers checkboxes.

Enter CONSUMER_1 in the Subscribers field, and then click Add.

33. Link the package steps together using green arrows.

34. Click Save( ) on the toolbar to save the package.



35. Click on the Execute( ) button on the toolbar.



38. Open the Operator navigator.


40. Check that the last session ran correctly. There should be 2 inserts and 3 updates.

We can see that the changed data for CONSUMER_1 has been propagated correctly.

41. Open the Models navigator and expand the Oracle CDC Source data model.

42. Right-click on SRC_CITY, select Changed Data Capture > Journal Data



Since we used CONSUMER_1 in our package, the changes for CONSUMER_2 have not been propagated. Having multiple subscribers enables us to execute multiple processes loading the same changed data without any interference.



Part 4: Run the CDC Set Up and Consumption Processes in Production In this exercise you will run in a production environment the two packages you have created to set up CDC, and then consume the changes captured by the CDC process. ODI is already configured to connect to the production environment. A context called Production has been created and it uses our Oracle production schema (CUST_DW_PROD) instead of the development schema (CUST_DW_DEV).

1. In the Designer navigator, open the Projects accordion.


3. Expand the Packages node.

4. Right-click on Tutorial Part 2 - CDC Setup and select Generate Scenario. Press OK in the New Scenario dialog. Note: A scenario is a frozen version of an ODI component, it is used to put that component into production. In this lab we have simplified the deployment process and we can simply run our scenarios in production using the Production context. Typically we would have exported a scenario from the development repository and then imported it into the production repository.

5. Expand the package and the Scenarios node then right-click on TUTORIAL_PART_2__CDC_SETUP and select Execute.



6. In the Execution window, set the Context to Production and press OK.


8. Go to Operator and make sure the session ran successfully.

9. In Designer, go to the Projects tree view and expand the Procedures node in the Advanced Topics in ODI / First Folder folder.

10. Right-click on the CDC Changes procedure and select Execute.





13. Click on the Operator button in the toolbar.


15. Validate that the procedure ran correctly, 2 inserts and 3 updates should have been performed.

16. Under Packages, right-click on Tutorial Part 3 Load CDC Data and select Generate Scenario then click OK.

17. Expand the package and the Scenarios node then right-click on TUTORIAL_PART_3__LOAD_CDC_DATA and select Execute.





20. Open the Operator navigator and select the Session List tab, and expand the All Executions node.

21. Validate that the last session ran correctly.



Summary At the end of these exercises the participants understand how to:

- Enable Change Data Capture. - Design Integration Flows to consume the changes captured by the

CDC process. - Run their processes in a Development environment as well as in a

Production environment.

Questions & Clarifications: If you have any comments or need additional information, please communicate through the Oracle Data Integrator forum at: http://forums.oracle.com/forums/forum.jspa?forumID=374

Documents

ODI Advanced