Partner Webcast – Oracle Data Integration for Big Data

Stay Connected

BLOGS.ORACLE.COM/IMC

TWITTER.COM/ORACLEIMC

YOUTUBE.COM/ORACLEIMCTEAM

FACEBOOK.COM/ORACLEIMC

http://blogs.oracle.com/imc

http://twitter.com/oracleimc

https://www.youtube.com/OracleIMCTeam

https://www.facebook.com/OPN.PartnerHub.Migration.Center

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |

Oracle Data Integration For Big Data

Milomir Vojvodic Director Business Development Oracle Europe, Middle East and Africa central Data Integration team October 15th 2015.

Presented by


Oracle Big Data Appliance

Oracle Exadata

Acquire Organize Analyze & Visualize Stream

Oracle Exalytics

Load from big data processing into your data warehouse for further analysis Access your customer information while you process through your big data in order to look for patterns

Big Data Value Is From Correlation



Staging

6

Sqoop

HDFS

Hive

Flume

Cap

ture

Trai

l

Ro

ute

De

live

r

Pu

mp

Transformation

Data Streaming Kafka (MPP Pub/Sub)

Storm and Trident

Spark Streaming

HBase

Discovery Sandbox/s

R Oracle GoldenGate

Oracle Data Integrator

Oracle Data Governance

Oracle Data Preparation

Model First Analytics

• Reporting-oriented • Often enterprise wide

in scope, cross LoB • “you know the

questions to ask”

Reports & Dashboards

Data First Analytics

• Data Exploration • Highly visual and/or

interactive • “you don’t know the

questions to ask”

Discovery

• Telematics • Industry Services • Internet of Things • Sentiment

Data Services

Oracle Big Data Integration and Governance Ecosystem

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

BIG DATA MOVEMENT &

TRANSFORMATION

BIG DATA GOVERNANCE

BIG DATA PREPARATION

REAL-TIME BIG DATA

ORACLE BIG DATA

INTEGRATION

Load Data into Hadoop in Real-Time

Move and Transform Data in Bulk

Manage & Monitor Metadata and Data Quality

Reduce Time Spent on Data Preparation

Oracle Big Data Integration


Target DB

OGG

Source DB

First OGG Differentiator Accessing directly transaction

logs

Second OGG Differentiator Moving only committed transactions

Why Is OGG Different?


TIME REQUIRED FOR THE END OF DAY

PROCEDURE

Hours

NO OF CPUs REQUIRED FOR SAME

PERFORMANCE*

No Of Required CPUs

ESTIMATED COSTS FOR SERVER AND

LICENSE**

Estimated Cost of Purchase in USD

0

20

40

60

80

100

120

140

Year1 Year2 Year3 Year4 Year5

Currently during the End Of Day utilizes the Server CPU by 40-50% and the IO by 90%. Probably the IO is the bottleneck.

0

20

40

60

80

100

120

Year1 Year2 Year3 Year4 Year5

Disaster Recovery Test and Development

Primary Site

$-

$,50

$1,0

$1,50

$2,0

$2,50

Millio

ns

Oracle License Costs

Hardware Costs

Daily load time can reach 5 days with the current HW

OR

Alternative To Batch Window

First OGG Differentiator Accessing directly transaction logs

After OGG Before OGG

Reduce source system overhead (and costs for stronger HW) by

70%


OR

Second OGG Differentiator Moving only committed

transactions

Alternative To Storage Replica

Begin, TX 1

Insert, TX 1

Begin, TX 2

Update, TX 1

Insert, TX 2

Commit, TX 2

Begin, TX 3

Insert, TX 3

Begin, TX 4

Commit, TX 3

Delete, TX 4

Begin, TX 2

Insert, TX 2

Commit, TX 2

Begin, TX 3

Insert, TX 3

Commit, TX 3

Begin, TX 2

Insert, TX 2

Commit, TX 2

Capture Checkpoint

Pump Checkpoint

Delivery Checkpoint

After OGG Before OGG

Reduce costs and efforts of data loss by 70%


Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp

New DB/ HW/OS/APP

Zero Downtime Upgrades & Data Migration

Fully Active Distributed DB

High Availability & Disaster Recovery

Application Offloading

Query & Report Offloading

Big Data, DW & Marts

Real-time BI, Hadoop Data Staging, Data Ingestion

Event Driven Architecture, SOA/JMS, Coherence

Message Bus & Data Grid

Data Synchronization Across the Enterprise

Global Data Centers

Real-time Analytics & Massive Parallelization

Data Streaming

GoldenGate

Real-time Data Delivery

11

Oracle GoldenGate For Big Data


• Accessing Directly Transaction Log

• Delivery Of Committed Transactions

• Stability Of Replica Line

Target DB Source DB

What Is So Special About OGG for Big Data?


1. Parameter setting only

2. No need to learn current and emerging big data technologies

13

OGG for Big Data OGGAA for JMS and Flat File

Flat File Java JMS

Hive HDFS HBase Flume

OGG for Big Data – Additional Value


• Name: Oracle GoldenGate for Big Data

• Separate Installer, available on eDelivery

• Includes base Java Adapter functionality, but no File adapter

14

Oracle

GoldenGate

Capture Database Transactions and Deliver to Big Data in Real-Time

JMS

Capture Trail Pump Route Deliver

HDFS (Files)

HBase (NoSQL)

Hive(SQL)

Flume (Streaming)

OGG for Big Data – Additional Value

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 15 Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

BIG DATA MOVEMENT &

TRANSFORMATION

BIG DATA GOVERNANCE


REAL-TIME BIG DATA

ORACLE BIG DATA

INTEGRATION







OLTP & ODS Systems Data

Warehouse, Data Mart

Oracle PeopleSoft, Siebel, SAP

Custom Apps

Files Excel XML

Enterprise Performance

Custom Reporting Packaged Applications

Business Intelligence

Analytics

Data Federation

Data Warehousing

Custom

Data Marts Data Access

Data Silos

SQL Java

Batch Scripts

Data Hubs

Data Migration

Data Replication

OLAP

Replacing Manual Coding


OLTP & ODS Systems Data

Warehouse, Data Mart

Oracle PeopleSoft, Siebel, SAP

Custom Apps

Files Excel XML

Enterprise Performance

Custom Reporting Packaged Applications

Business Intelligence

Analytics

OLAP

Oracle Data Integrator

Replacing Manual Coding

After ODI Before ODI

Reduce data transformation maintenance costs by 80% (hard to change, every script contains

special rules, code stored in many machines)


Journalize Read from CDC

Source

Load From Sources to

Staging

Check Constraints before

Load

Integrate Transform and

Move to Targets

Service Expose Data and Transformation

Services

Reverse Engineer Metadata

Reverse

Journalize

Load

Check

Integrate Services

CDC

Sources

Staging Tables

Error Tables

Target Tables

WS

WS

WS

SAP/R3

Siebel

Log Miner

DB2 Journals

SQL Server Triggers

Oracle DBLink

DB2 Exp/Imp

JMS Queues

Check MS Excel

Check Sybase

Oracle SQL*Loader

TPump/ Multiload

Type II SCD

Oracle Merge

Siebel EIM Schema

Oracle Web Services

DB2 Web Services

Sample out-of-the-box Knowledge Modules

Benefits

ODI Knowledge Modules

ODI Declarative Design

ODI Declarative Design

Define How : Built - in Templates

Define What You Want

Automatically Generate Dataflow

1 1 2 2




1 1 2 2




1 1 2 2



1 1 2 2 1 1 2 2

ODI E-LT

Staging Server

Data Warehouse

Second ODI Differentiator ODI Declarative Design and

ODI Knowledge Modules for reusing already written

down level SQL code

First ODI Differentiator Transformations using

the power of the Target Database – no

staging server

Why Is ODI Different?


Reduce ETL development

costs by 30% (no prebuilt code, need

to learn various languages, need to write and tune SQL)


Decrease the cost o of ETL HW by 100%


Business rules implemented in SQL

Source (MySQL)

ORDERS

LINES

CORRECTIONS

File

Target (Oracle)

SALES

ERRORS

Join

ORDERS.ORDER_ID =

LINES.ORDER_ID

…

Mapping

SALES =

SUM(LINES.AMOUNT) +

CORRECTION.VALUE

• SALES_REP =

ORDERS.SALES_REP_I

D

Constraints

ID is flagged not null

in the model. Unique

index UK_ID is declared

on the SALES table.

Filter

ORDERS.STATUS=CLOSED

…

Implementing The Rules


Target (Oracle)

SALES

ERRORS

Transform and

integrate

TEMP_

SALES

Check constraints/

Isolate errors

Source (MySQL)

ORDERS

LINES

CORRECTIONS

File

TEMP_1

Extract/Join/

Transform

TEMP_2

Extract/Transform

Join/Transform

1

2

3

4

5

Process Implementation Without ODI


1

2

3

4 5


Target (Oracle)

SALES

ERRORS

Source (MySQL)

ORDERS

LINES

CORRECTIONS

File

TEMP_1

Extract/Join/

Transform

TEMP_2

Extract/Transform

Join/Transform

Transform and

integrate

TEMP_

SALES

Check constraints/

Isolate errors

LKM

LKM

LKM

CKM

IKM

Proprietary Engine

- Specific Language

Process Implementation With ODI


Target (Oracle)

SALES

ERRORS

Transform and

integrate

TEMP_

SALES

Check constraints/

Isolate errors

Source (MySQL)

ORDERS

LINES

CORRECTIONS

File

TEMP_1

Extract/Join/

Transform

TEMP_2

Extract/Transform

Join/Transform

1

2

3

4

5

Process Implementation With ODI


Logical Design

Physical Design

Oracle

MySQL

Hive

Sqoop

Sqoop

IKM

LKM

LKM

Oracle

Hive

MySQL

Hive

Logical and Physical Design with ODI

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 25

Flume/Kafka Hive on MR, Tez, Spark

Logs

OLTP DB

SQOOP

OGG

Pig on MR, Tez, Spark

ODI

SQOOP Any DW

OGG

Spark

Oozie

OEDQ OEMM

Data Validation & Cleansing

Metadata Mgmt & Lineage

API/File/HDFS

Hive/HCat, HDFS,HBase

Hive/HCat, HDFS,HBase

NoSQL

Flume/Kafka

Load to Oracle

OLH/OSCH

Oracle DB Big Data SQL

Files

HDFS/File KM

Oracle Data Integrator (ODI) Capabilities for Big Data

http://hadoop.apache.org/


• ODI Is Writing Code

• SW Project Lifecycle + Data Semantic Related Functionalities

• Reusability Principle

• ODI Is Easy To Learn

What Is So Special About ODI for Big Data?


1. Code Generation for Spark

2. Code Generation for Pig

3. Execution using Oozie

4. No Installation on Hadoop

27

ODI Advanced BD Option – Additional Value


Hadoop Cluster

Spark

Sqoop Hive

Pig

ODI

Oozie

Sqoop

28

Hadoop Cluster

Spark Sqoop Sqoop

Hive

Pig

Manual Code Hadoop Cluster

ETL ETL HDFS

Hadoop Cluster

ETL ETL ETL

HDFS

1. Traditional ETL Tools (execute entirely outside of Hadoop)

2. ETL Tools with Native “on” Hadoop (require proprietary code on Data Nodes)

3. Manual Coding (ultimate flexibility, but very high cost)

4. ODI Native in Hadoop (no Engine & no Data Node footprint)

ETL

GG

generate manage

BEST

ODI Advanced BD Option – Additional Value


29


BIG DATA MOVEMENT &

TRANSFORMATION

BIG DATA GOVERNANCE


REAL-TIME BIG DATA

ORACLE BIG DATA

INTEGRATION







Information

Security

Master Data

Management

Policy Management

Enterprise Metadata

Reporting

Data

Integration

Data Quality

DATA GOVERNANCE

Data Governance Consolidates Strategies


Technology

Process

People

People • Governance Committee • Data Stewards

Process • Defined rules

& practices Technology

• Implements process and supports people

• Makes operation more consistent & cost-effective

• Basis for continuous process improvement

Data Governance – more than just technology


Data Originator

Data Originator

Data Originator

Data Originator

Data Originator

Data Consumer Data Consumer Data Consumer Data Consumer

Data Governance, Integration & Management

Transformation

Cleansing

Replication

Matching Merging

Aggregation Denormalisation

Separation Between Originators and Consumers


Customer ID Customer Name Address 1 Address 2 City State Zip Country Birth Date Gender

AD23298 Mr Peter Mayhew 9407 Main St Fairfax VA 22031-4001 USA 02/23/61 M

VS38611 Dr Ellen Van Der Heijde 144 E Grove St Kingston PA 18704 US 07/12/57

DC18223 Jalila Abdul-Alim (Do Not Call) 4548 Pennsylvania Ave Apt 205 Kansas City MO 64111-3349 USA 02/23/63 F

CO9387A Tayside Computers Inc. 4912 E 41st N Idaho Falls ID 83401 USA 31/03/2007 N/A

TZ35019 Mr Zachary P Jahn 98-1731 Ipuala Loop Aiea Hawaii 96701 1710 United States 06/12/86 Male

CB27843 Mrs Edith Y Baba Junior Baba Real Est. Corp. 209 Stony Point Trl Webster NY USA 11/17/1971 M

OX80306 Andrew & Mary Baxter 14 Oxbridge Way Milfrod NH 03055-4614 US 05/28/67 F

JP70210 Mr RJ & Mrs FB MacDonald 57 Hadleigh Close Westlea Swindon SN5 9BZ MA - USA - Y

RD48107 Mr Andy Baxter 14 Oxbridge Wy Milford NH 3056 USA 01/01/01 M

Inconsistent formats Abbreviations

(often ambiguous)

Attributes non-standard, missing

or invalid

Widespread

duplication

(often hard

to spot)

Compound Names

Embedded Additional Information

Mixed Business & Personal Names

Multiple Names

Mis-Fielded Data

Erroneous Data

International Date Formats

Default or Dummy Data

Why EDQ?

After EDQ Before EDQ

Avoid error costs (incorrect orders, inventory etc.) by 20%














• A Match Rule is simply the combination of comparison results

• Rules are evaluated in order and if one hits, we stop

• Rules can be ‘negative’ to eliminate pairs that are too different with a ‘No Match’ rule

• Rules can easily be turned on & off during the tuning process




ETL

BI Dashboards

App

ETL

ETL

How was sales figure calculated?

What will happen if I change this table?

What reports use the mainframe

data?

Sys Admin

Executive

BI Developer

Where did this data come

from?

Application User

Which reports use this customer

data?

CDC

Hadoop Data Lake

Data Steward

Can I trust the sources of this

customer data?

ETL

Developer

I want to design an experiment to measure the success of a

signup page. What data do I have?

Data Scientist

GG

Data Warehouse

OGG BI&DW Synchronization and Loading

ODI OEDQ

OEMM

After OEMM

Before OEMM

OPEX - Reduce data maintenance costs by 80% (not anymore.. hard to

change, long to find)

Value Of Metadata Management

After OEMM

Before OEMM

CAPEX - Reduce analytical project costs by 30% (not anymore ..post

remediation costs, unnecessary mistakes )


Metadata Repository

Any Web Browser

OEMM Architecture


Metadata Requirements Stack


1. Title - the title of the data object

2. Creator - the person or entity responsible for creating the data object

3. Subject - subject terms or keywords that describe the data object

4. Description - a brief description, or abstract, of the data object

5. Publisher - the entity responsible for making the data object available

6. Contributor - a person or entity who contributed to the creation of the data object

7. Date - data of creation, publication, or revision of the data object

8. Type - the type of object. For data this would typically be "dataset"

9. Format - a description of the format or file type(s) of the data object

10. Identifier - a permanent identifier used to locate and identify the data object

11. Language - the language(s) used within the data object (if applicable)

12. Source - a relational element describing the lineage of the data object

13. Relation - a relational element describing the relationship of this data object to other objects, collections, or entities

14. Coverage - describes the spatial and temporal context of the data object

15. Rights - describes any rights, restrictions, or terms of use

Metadata















Business Glossary Stewardship Workflow


Business Glossary Manages Definitions


Mapping of Business Term to Rules


Capture Business Rule Definitions


Semantic Linking of Business Rules


Allowable Values of Business Terms


Data Flow Architecture Views: End-to-End / Top-to-Bottom


Graphical Browser for Data Model Diagrams

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

BIG DATA MOVEMENT &

TRANSFORMATION

BIG DATA GOVERNANCE


REAL-TIME BIG DATA

ORACLE BIG DATA

INTEGRATION







Parse Click Stream Logs

Repair App Data

Classify Social Data

Structured

Unreliable

Unstructured High Velocity

Unstructured High Volume

SSN Credit Card Info

Supported Formats

Oracle Big Data Preparation Cloud Service


















h












Q&A

10

ISV Migration Center blog: http://blogs.oracle.com/imc ISV Migration Center email: [email protected]



mailto:[email protected]


• Oracle.com Partner Hub oracle.com/partners/goto/hub-ecemea

• Migration Center Team Blog blogs.oracle.com/imc

feeds.feedburner.com/oracleIMC

• Partner Webcast Recordings youtube.com/OracleIMCteam

• Partner Webcast Presentations slideshare.net/Oracle_IMC_team

• [email protected]

Oracle Partner Hub ISV Migration Center • twitter.com/OracleIMC

• plus.google.com/+OracleIMC

• facebook.com/OracleIMC

• linkedin.com/groups/Oracle-Partner-Hub-Migration-Center-4535240

http://oracle.com/partners/goto/hub-ecemea






http://www.slideshare.net/oracle_imc_team








https://plus.google.com/108612216492389783792




http://linkedin.com/groups/Oracle-Partner-Hub-Migration-Center-4535240














https://www.facebook.com/oracleimc


https://www.youtube.com/OracleIMCTeam