Oracle Data IntegratorAuthor-Priyanka Payal Mehta

*

Document ControlChange Record

Reviewers

DateAuthorVersionChange Reference20-Oct-08Priyanka0.1Initial Draft

NamePositionAnuj Mohan SherryProject Manager

*

General InformationThis document contains the course notes for ODI training.Scope- The scope of the training will introduce to ODI & how to use this tool for Data Integration applications.Training Prerequisites- Knowledge of SQL.

ConventionsODI Oracle Data Integrator

Notes - Please refer to the notes attached to the slides for detailed explanation and for Navigations

*

Course AgendaODI Architecture & ComponentsCreating Master & Work RepositoryODI Logical & Physical SchemaCreating ODI objectsModelProjectInterfacesProceduresPackagesCreating Scenario ,Monitoring & Scheduling ScenarioDifferent Features of ODI

ODI Architecture & ComponentsIntroduction

*

Oracle Data Integrator- An IntroductionOracle data integrator is a member of the Oracle fusion middleware family of products. It mainly addresses data integration needs in heterogeneous environments.

It is a comprehensive data integration platform that covers all data integration requirements from high-volume, high-performance batches, to event-driven, trickle-feed integration processes, to SOA-enabled data services

This software product is based on a unique E-LT architecture. Its main features are its graphical modules and software agents that helps in checking data consistency and data flow.It enhances user productivity modularized design approach

*

Technical capabilities

It deals with various technologies ( Hypersonic SQL, XML, Oracle) etc.

It reduces the documentation and programming phases by more than 90%.

Erroneous data can be corrected either on the source or on the target wherever it is located in the information system.

It manages the isolation and recycling of erroneous data.

It stores the business rules independent of underlying technology. This helps in modifying the interface without changing the transformation rules (mapping, calculations, filters, aggregation, etc).

Rule maintenance is an isolated operation.

*

Architecture Overview

Oracle Data Integrator is built on several components all working together arounda centralized metadata repository.

Below are the four graphical modules that are used to create and manage ODI objects:

Designer is used to define data stores (tables, files, Web services, and so on), interfaces (data mappings), and packages (sets of integration steps, including interfaces). Topology Manager is used to create and manage connections to data sources and agents and is usually restricted so that only administrators have access. Operator is used to view and manage production integration jobs. Security Manager manages users and their repository privileges.

Being Java-based, these applications can run in any Java environment, including Microsoft Windows, Macintosh OS X, and Linux.

*

Architecture Overview (contd.)

Graphical modules connect the repository

Creation of Master and Work Repository

*

Creating Master Repository To create a Master RepositoryClick on Start Menu Programs Oracle Oracle Data Integrator Repository Management Master Repository Creation.

*

*

Master Repository Creation (Contd.)

*

To create Login for Master RepositoryClick on Start Menu Programs Oracle Oracle Data Integrator Topology Manager Login Creation

*

Login Creation(Contd.)

*

Login to Topology Manager

*

Topology Manager

*

Creating Work Repository

*

Work Repository Creation (Contd.)

*

Work Repository Creation (Contd.)

ODI Logical & Physical Schema

*

Creating Context

*

Context Creation (Contd.)

*

Creating Data Server for File

*

Creating Data Server for File

*

Creating Data Server for File

*

Creating Data Server for File(Contd.)

*

*

Creating Physical Schema for File(Contd.)

*

Physical and Logical Architecture for File

*

Creating Data Server for Oracle

*

Creating Physical Schema for Oracle

*

Physical and Logical Architecture for Oracle

ODI Objects - Model

*

Oracle Data Integrator- Objects Model A model is a set of data stores corresponding to data structures contained in a physical schema. Model is organized into a set of model folders basically to categorize each of the technologies being used . Datastore Datastore are part of model which defines data structure. Each model should contain at least one Datastore. We have different type of Datastore depending upon source e.g. Flat File Datastore ,Oracle Table Datastore

*

Logical Structure (Models Tab)

*

Creating a Model Folder in Models tab

*

Creating a File Model

*

Creating a File Model (contd.)

*

Creating a File Model (contd.)

*

Creating a File Datastore

*

Creating a File Datastore (contd.)

*

Creating a File Datastore (contd.)

*

Creating a File Datastore (contd.)

*

Viewing Source File Data

*

Viewing Source File Data (contd.)

*

Creating a Oracle Model

*

Creating an Oracle Model

*

Creating a Oracle Model (contd.)

*

Creating Oracle Datastore

*

Creating Oracle Datastore (Contd.)

*

Creating Oracle Datastore (Contd.)

*

Creating Oracle Datastore (Contd.)

ODI Objects - Project

*

Oracle Data Integrator- Objects Project PackageThe package is the largest unit of execution in Oracle Data Integrator. A package is made up of a sequence of steps organized into an execution diagram.InterfaceAn interface consists of a set of rules that define the loading of a Datastore or a temporary target structure from one or more source Data stores.ProcedureA Specific Procedure is a reusable component that groups operations that do not fit in the Interface framework - loading a target Datastore from one or more sources

*

*

Creating a Project Folder

*

Creating a Project Folder (contd.)

*

Creating a Project

*

Adding Knowledge Modules

*

Knowledge Modules (contd.)

*

Knowledge Modules (contd.)

*

Knowledge Modules (contd.)

ODI Objects - Interfaces

*

Creating an Interface

*

Creating an Interface (contd.)

*

Creating an Interface (contd.)

*

Creating an Interface (contd.)

*

Creating an Interface (contd.)

*

Creating an Interface (contd.)

*

Executing the Interface

*

Executing the Interface (contd.)

*

Viewing Result

*

Viewing Result (contd.)

*

Viewing Data

*

Viewing Data (contd.)

ODI Objects Procedure & Packages

*

Creating Procedure

*

Creating Procedure

*

Creating Procedure(Contd.)

*

Creating Procedure(Contd.)

*

Procedure: A reusable component that groups operations

*

Creating Package

*

Package: Largest unit of execution in ODI. Made up of a sequence of steps

Creating Scenario ,Monitoring & Scheduling Scenario

*

Creating Scenario

*

Scheduling Scenario

*

Scheduling Scenario

Different Functionalities available with ODI

*

Column Mapping: Using a source or any calculation rule on source information

*

Mapping Rules: Using database functions or default values

*

Variable: A value stored in Oracle Data Integrator

*

Multi-table Joins(Equi/Cross/Outer)

*

Filters: Return functionally useful records. Tracking of error at any level is easy.

*

Objects can be exported then re-imported in one or more different contexts.

*

A new version can be generated to identify subsequent modifications

*

Thank You

When we Login into Oracle Data Integrator ,We actually Login into a Work Repository.Work Repository is the Place where all Extraction, Loading and Transformation processes takes place.Work Repository is inside a Master Repository.So we need to create a Master Repository first.

Oracle Data Integrator comes with an Oracle DI Local Repository. So it is not compulsory to create a master repository and then the work repository. However, it is a good practice to create a Master Repository for your Project and then subsequently create work repository for different modules of the project.

To create a Master RepositoryClick on Start Menu All program Oracle Oracle Data Integrator Repository ManagementMaster Repository Creation.

*After Clicking the Master Repository Creation, we get the Master Repository Creation Wizard.The Wizard asks for the following details. 1.Driver : Drivers are used to connect to databases. Oracle Data Integrator provides 12 drivers to connect to the database. 2.URL : URL is the path where database is located. This option helps us to connect to database irrespective of its location.(The database can be local or in some other Machine) 3.User :The user id provided in User textbox is used to connect to the database specified in the URL. 4.Password :Pass is the password textbox .It is used to connect to the database. 5.Test Connection :Through this button, we can test our connection. If all the details provided are correct, we will get a message Successful Connection. 6.Id : ID is used to give an identification number to the master repository we create. 7.Technology : Technology textbox specifies the technology of the database which we select. In our Demo we are using oracle 10g so we choose oracle as the Technology, you can choose technology based on your requirement.

8. After filling all the details we click on OK

*1.After clicking on OK a sequence is fired as shown in the above screenshot. The Master Repository imports all the required components. 2.After successfully loading all the objects we get a pop-up message Repository creation Successful. Do you want to quit the Oracle Data Integrator Wizard. This message indicates that the repository has been created.

*After Creating the Master Repository ,we need to create a Login to access the Master Repository .For creating the login we use topology manager.

*Click on New to create Repository Connection Enter the Connection Details in Repository Creation windowThe Repository connection screen is divided into two frames Oracle data Integrator Connection and Database Connection.The Details we provide in Oracle Data Integrator Connection frame will be used to connect the ODI components (Designer ,Operator, Topology or Security ) once the login is createdThe detail we provide in Database connection frame is used to connect to the repository.

In Oracle Data Integrator Connection 1.Login name: Is the name we use for login into ODI. (Conversion is the login name) 2.User :User is the id or the name through which we login into oracle data integrator the user name has to be SUPERVISOR, it is so because when we create a new master repository it imports the default security settings. So once we successfully create our login we can change it. 3.Password :it is the password for the user name we provide in the User textbox, the password has to be SUNOPSIS, as it is imported default.

In Database Connection 4.User :is the user name through which we connect to the database. 5.Password :it is the password for the user which is specified in the user textbox. 6.Driver List :from the Driver list we select the driver which is used to connect to the repository. 7.Driver Name :It is the name of the driver we select. 8.URL :url is the path where the database is located. all the details provided in the Database Connection frame should be same as the details provided while creating the master repository.

We can test the connection using the Test Button. This helps us to validate the detail which we fill during login creation.

If we check the Default Connection checkbox then the login we create will act as a default connection so the next time we login, we will login to this repository .

9. After entering all the details click on OK button*Topology Manager defines the physical and logical architecture of the infrastructure. Servers, schemas, and agents are registered in the master repository through this module, generally by the administrators of the infrastructure or project. *Main Components of Topology Manager :1.Physical Architecture2.Context3.Logical Architecture

Physical Architecture:-The physical architecture defines the different elements of the information system, as well as thecharacteristics taken into account by Oracle Data Integrator.

Context:- Context allows to link the physical architecture with logical architecture

Logical Architecture:-The logical architecture allows a user to group physical schemas containing data stores that arestructurally identical but located in separate places, into Logical Schemas, structured pertechnology. Apart From this: Language:- which describes the different type of languages available Repository :- It contains the master repository, the work repository and information related to them.*All the extraction, loading ,transformation and integration work happens in ODI with the help of work repository.So first we need to create work repository.Work repository is a metadata repository which contains all the project objects.Work repository is accessed by designer, operator and agents during runtime.

1.Right Click on Work Repository2.Click on Insert Work Repository*On clicking Insert work Repository , Data Server connection window will open .Under Definition Tab1.Enter Data Server Name2.Enter User name3.Enter Password

Under JDBC Driver Tab :Select JDBC DriverEnter JDBC URLClick on Apply*On clicking Apply on Data Server Window ,Work Repository Creation window open1.Enter Name of work repository .(All other details are populated by default)2.Click on Apply and then Click OK*Context allows to link the physical architecture with logical architecture1.Click on Insert Context Icon*1.Give the name of Context under Definition Tab2.Click on APPLY and Then Click on OK*In ODI each database is represented as a technologyInside technology we can create data server which contains physical schema/Logical Schema.

*1.Right Click On File Technology2.Click on Insert Data Server*1.Enter name for Data Server*1.Give Name for DataServer under Definition Tab Under JDBC tabEnter the Name of DriverEnter the URL of DatabaseClick on Test to test the server connection*Under Data Server Creation Screen1. On Clicking Apply Button in Data Server Window, Physical schema creation Screen will Open upIn Physical Schema creation screen, Under Definition Tab :1.Enter name for physical schema2.Enter the Directory (Schema). Directory(Schema) is the place where the source data files are stored and it should be in our local machine. So we manually enter the path where the files are stored in the textbox3.Enter the Directory (Schema). Directory(Work Schema) is the place where ODI will create its temporary objects.So we manually enter the path where we want the temporary objects to be stored*Under Context Tab1.Click on the new context button.2. Select the context from the drop down box.3.Enter the logical schema name and then click on OK .This will automatically create logical schema which can be checked under the file technology, in logical architecture

*So ,here we can see Under Logical Architecture ,XXCNV_LOGICAL_SCHEMA_FILE got created*1.Right Click on Oracle Technology2.Click on Insert data Server. A new Screen will appear for Data Server Creation In the Data Server Creation Window, Under Definition tab:1.Enter the name for Data Server2.Enter The name for Instance3.Enter the database username4.Enter the database password*Under JDBC Tab of Data Server1.Enter the JDBC Driver2.Enter the JDBC URL3.Click on Apply, Physical Schema creation screen will appear.Under definition tab of Physical schema creation screen1.Enter the name for physical schema2.Enter the name of schema Under Context Tab of Physical schema creation screen1.Click on Insert Context icon2.Select Context from the dropdown3.Enter the name of Logical schema and click on Apply. This will create logical schema automatically

*Under Logical architecture, in Oracle technology we can see the newly created Logical Schema*Navigate to Model section from the menu in the ODI.1.Click on Insert Model Folder Icon2.Under Definition tab give name and description for new model folder3.Click on apply and then ok

*1.Right Click on Model Folder2.Click on Insert Model

*1.Enter the name of model2.Select Technology as File for Data file Model3.Select the respective logical Schema

*Under Reverse tab of Model Window1.Select the Context2.Click on apply and then ok

*1.Right Click on Model 2.Click on Insert Datastore

*1.Enter name of Datastore 2.Select Datastore type as Table3.Browse to source file under Resource name to create file Datastore.*1.Select File format (Note: The source flat file is comma (,) delimited file)2.Select field separator in case of Delimited file (Note: The heading column signifies the row number after which your data exists)3. Click on Apply

*1.Click on Reverse tab Note: This will populate columns details, in case 1st row of file contains the column details2.Click on Apply and then OkNote: After clicking on Reverse tab in the name filed all the columns name given in file should come. If it is not coming then you have to give all the columns name manually.3.Similarly create other file Datastore

*1.Right click on File Datastore2.Click on View data

*1.Right Click on Model Folder2.Click on Insert Model

*Create Model For Oracle Table1.Select Technology as oracle for oracle table model2.Create all required oracle Datastore for given scenario

*Under Reverse tab of Model Window1.Select the Context2.Click on apply and then ok

*1.Right Click on Oracle Data model2.Click on Insert Datastore

*1.Enter Name of Datastore2.Select Datastore type as Table3.Enter Table name as Resource name

*1.Click on Reverse Under Columns Tab

*After Clicking on Reverse Tab all columns of table will be shown as above

*1.Click on Insert Project Icon

*1.Enter Name under Definition tab

*Right Click on Project Folder and then Click on Insert1.Give name of Project Under Definition tab

*Knowledge Modules are predefined procedures that once imported are fired at backend. These Procedures perform some pre-specified sets of instructions and thus reduce the time required to perform those steps manually.1.Right click on Loading (LKM)2.Click on Import Knowledge Modules to import oracle provided module*Select the path that contains all the models present

*Select the file that needs to be added*Select different knowledge modules required to run the interface.We choose CKM Oracle, IKM Oracle Incremental Update(PL SQL) & LKM File to SQL from the list of KMs available*1.Right Click on Interface2.Click on Insert Interface

*1.Enter Name of Interface Under definition tab*1.Drag and drop the Datastore of source under diagram tab of interface(Note: Its good practice to drag the target datastore and then source datastore

*1.Drag and drop the target Datastore under diagram window of interfaceNote: One Note asking for auto mapping will pop up. Click on Yes for auto mapping

*1.Make one column as Key(Note: This step will uniquely identify the records while loading the target table)*1.Select LKM as LKM to SQL under Flow tab window of interfaceNote: This LKM is protocol defined in ODI for loading data in different technology*1.Click on execute2.Click on Ok*1.Click on Ok*1.Click on Operator Icon to view the results*1.Click on Sessions Menu2. Indicates interface executed successfully3. Indicates interface resulted in error.Note: for details execution process, double click on the step and check the description to give the detail.*1.Right click on Target Datastore2.Click on data*1. Right Click on Procedures2. Click on Insert Procedure*1.Enter the Name of Procedure2. Select Source and Target Technology*1.Click on Details Tab2.Click on ADD Button *1.Enter name of command2. Select Execution Context3.Select the Schema4.Select Commit5.Right Command under command panel and Click on apply

**1.Right Click On Package2.Click on Insert Package*1.Right Click on Package2.Click on Generate ScenarioA new window will appear 1.Click on OK to create new Scenario*The Scenario can be scheduled for a pre defined time of the day.For scheduling, go back to the Designer and Right Click SchedulingSelect Insert Scheduling, to create a schedule.

*In the window that appears.Select Context Name from the available Contexts from the Drop Down.Select the Log Level as desired and Agent Name .We can make the changes in the time and date from this window and also the frequency of this scenario can be selected as On Startup, Daily, Single, Hourly, Weekly etc.On clicking OK the Scenario will run at the scheduled time.

*