AgileAnalytics: Agile Real-Time BI with Oracle Business Intelligence, Oracle Data Integrator and...

AgileAnalytics: Real-Time BI with Oracle Business Intelligence, Oracle

Data Integrator and Oracle GoldenGate

Stewart Bryson

medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

About Red Pill Analytics

Capacity Analytics Agile Cloud Development for Oracle BI & Analytics

Checkmate DevOps for Oracle BI (Cloud-hosted or On-premises)

Single Doses Training + Strategy + Solutioning + Infrastructure + Support

Real Time BI with Kevin & Stewart

• 19 episodes over the last 2 years

• Featured guests from the Oracle BI & EPM industry

• Published on iTunes and YouTube

• Join us at the Thursday Deep-Dive session from 9-11am for a LIVE show that will be simultaneously broadcast online!

AgileAnalytics 1.0: Focus on Performance

Exadata: Extreme Performance

OBIEE: Extreme Metadata

Agile: Extreme Methodology

Exadata: Extreme Performance

OBIEE: Extreme Metadata

Agile: Extreme Methodology

AgileAnalytics 2.0: Focus on Integration

GoldenGate: Extreme Realtime

ODI: Extreme Transformation

Oracle Information Management Reference Architecture

Enterprise Performance Management

Pre-built & Ad-hoc BI Assets

Information Services

Data Ingestion

Information Interpretation

Access & Performance Layer

Foundation Data Layer

Raw Data Reservoir

Data Science

Data Engines & Poly-structured sources

Content

Docs Web & Social Media

Structured Data Sources

•Operational Data •COTS Data •Master & Ref. Data •Streaming & BAM

Immutable raw data reservoir Raw data at rest is not interpreted

Immutable modeled data. Business Process Neutral form. Abstracted from business process changes

Past, current and future interpretation of enterprise data. Structured to support agile access & navigation

Discovery Lab Sandboxes Rapid Development Sandboxes

Project based data stores to support specific discovery objectives

Project based data stored to facilitate rapid content / presentation delivery

Data Sources

Abstracts rate of reception from rate of

distribution

Facilitates “right-time” flow of data

Abstracts atomic data from the business process.

Process-neutral; Business logic agnostic

Process-specific

Imbued with business logic

SugarCRM (Our Sample Dataset)

Typical Waterfall Approach

• Interview users • Construct requirements • Create logical data model • SQL prototyping of source model

• Document source-to-target mapping

• ETL development • Front-end development • Performance tuning

Alternative: Iterative Waterfall?• Design iterations around smaller chunks

• Iteration 1: Interviews and user requirements • Iteration 2: Logical modeling • Iteration 3: ETL Development • Iteration 4: Front-end development

• Requires 4 iterations before we get any usable content

Manifesto for Agile Software Development

“We are uncovering better ways of developing software by doing it and helping others do it. !

That is, while there is value in the items on the right, we value the items on the left more.” http://agilemanifesto.org/

Manifesto Applied to AgileAnalytics• User Stories instead of requirements documents • User asks for content or functionality through a narrative

• Typically includes current version of the report

• Time-boxed iterations • Iteration has a standard length • Choose one or more user stories to fit in that iteration

• Rework is part of the game • There are no “missed requirements”... only those that haven’t been delivered yet.

Rapid Development Sandbox

Oracle GoldenGate can

bypass the Staging layer to trickle-feed the

Foundation layer

Staging layers exist to support batch

load scenarios

We use our Foundation layer

combined with any Access & Performance layer content to close stories immediately

Developers can use independent development

sandboxes to work on their issues in isolation

Are development sandboxes radical?

RPD Development requires a BI Server

RPD Development requires a query tool

RPD Development requires data visualization

Oracle GoldenGate Architecture

Loading Foundation: Standard ETL Approach

Source System

Staging Layer

Foundation Layer

Loading Foundation: AgileAnalytics Approach

Source System

Staging Layer

Foundation Layer

GoldenGate for AgileAnalytics: Loading the Foundation Layer

map SUGARCRM.ACCOUNTS, target SUGARFND.ACCOUNTS, KEYCOLS(ID, FND_SCN), INSERTALLRECORDS, colmap( USEDEFAULTS, FND_SCN=@GETENV("TRANSACTION" , "CSN"), FND_ROW_RANK=@TOKEN("TK_ROW_RANK"), FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"), FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE") );

Map SOURCE and TARGET

KEYCOLS defines matching criteria

USEDEFAULTS defines column for column mapping Built-in attributes from the

“token” and the “header”

GoldenGate for AgileAnalytics: Loading the Foundation Layer

map SUGARCRM.ACCOUNTS, target SUGARFND.ACCOUNTS, KEYCOLS(ID, FND_SCN), INSERTALLRECORDS, colmap( USEDEFAULTS, FND_SCN=@GETENV("TRANSACTION" , "CSN"), FND_ROW_RANK=@TOKEN("TK_ROW_RANK"), FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"), FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE") );

Map SOURCE and TARGET

KEYCOLS defines matching criteria

USEDEFAULTS defines column for column mapping Built-in attributes from the

“token” and the “header”

INSERTALLRECORDS makes this our Foundation Layer

GoldenGate INSERTALLRECORDS

NAME TICKER_SYMBOL

OWNERSHIP FND_SCN FND_COMMIT_DATE FND_DML_TYPERed Pill Analytics 2992758 08/20/2012 12:00 AM INSERT

NAME TICKER_SYMBOL OWNERSHIP

Red Pill Analytics

NAME TICKER_SYMBOL

Red Pill Analytics, LLC Private 2992760 03/13/2013 12:00 AM UPDATE

Red Pill Analytics, LLC Private

NAME TICKER_SYMBOL

Red Pill Analytics, LLC Private 2992760 03/13/2013 12:00 AM UPDATE

Red Pill Analytics, Inc. RPAI Public 2992762 05/02/2013 12:00 AM UPDATE

Red Pill Analytics, Inc. RPAI Public

NVL (!LEAD ( fnd_commit_date ) over ! ! ! ( PARTITION BY id ! !! ORDER BY fnd_commit_date ),!to_date( '12/31/9999','mm/dd/yyyy' )!)

NAME FND_SCN FND_COMMIT_DATE FND_EXPIRE_DATE FND_DML_TYPE FND_CURRENTRed Pill Analytics 2992758 08/20/2012 12:00 AM 03/13/2013 12:00 AM INSERT N

Red Pill Analytics, LLC 2992760 03/13/2013 12:00 AM 05/02/2013 12:00 AM UPDATE N

Red Pill Analytics, Inc. 2992762 05/02/2013 12:00 AM 12/31/1999 12:00 AM UPDATE Y

CASE!WHEN ( RANK() over (PARTITION BY id ! ! ! ORDER BY fnd_commit_date desc )) = 1!THEN 'Y'!ELSE 'N'!END

NAME FND_SCN FND_COMMIT_DATE FND_EXPIRE_DATE FND_DML_TYPE FND_CURRENTRed Pill Analytics 2992758 08/20/2012 12:00 AM 03/13/2013 12:00 AM INSERT N

Red Pill Analytics, LLC 2992760 03/13/2013 12:00 AM 05/02/2013 12:00 AM UPDATE N

Red Pill Analytics, Inc. 2992762 05/02/2013 12:00 AM 12/31/1999 12:00 AM UPDATE Y

Model-Driven Development•Start with thin requirements • Sample reports • SQL whenever possible •Annotated discrepancies

•Development against the Foundation Layer • Top-down approach • Build empty BMM containing logical “facts”, “dimensions” and “hierarchies” •Map physical sources to the BMM

•Develop permanent, high-impact dashboards •No work in this iteration is throw-away

Model-Driven Development

•Start with thin requirements • Sample reports • SQL whenever possible •Annotated discrepancies

•Development against the Foundation Layer • Top-down approach • Build empty BMM containing logical “facts”, “dimensions” and “hierarchies” •Map physical sources to the BMM

•Develop permanent, high-impact dashboards •No work in this iteration is throw-away

Model-Driven Development

Simplification of the Data ModelIntegration of Disparate Data Sources

Addition of Business Logic and Calculations

OBIEE is for star schemas

OBIEE is for star schemas False

Logical Table Sources (LTS) provide the

functionality for mapping transactional schemas as

dimensional models. !

Siebel initially acquired nQuire and the

technology that would become OBIEE to report

against Siebel CRM.

Model-Driven: Mapping Physical to Logical

Dragging and dropping tables from the Physical Layer is the only way to create logical tables in the Business

OBIEE Gestures

Model-Driven: Dimension Tables

Model-Driven: Fact Tables

Model-Driven: EDW Content (Date Dimension)

Model-Driven: Initial Dashboard

Dimensional Prompts

Performance Tiles

Operational MetricsTrending Information

ETL Development•We us GoldenGate to populate our Staging layer • Standard batch •Micro batch

•Use Oracle Data Integrator (ODI) to transform our data to meet business requirements •Need to decide what flavor of GoldenGate to use • Extend our current implementation •Use ODI journalization framework

Loading Access and Performance: Standard Approach

Staging Layer

Foundation Layer

Access & Performance

Loading Access and Performance: AgileAnalytics Approach

Staging Layer

Foundation Layer

Source System Access & Performance

GoldenGate handles Staging and Foundation

layers

Staging Layer

Foundation Layer

ODI from Staging for transactional table loading

GoldenGate handles Staging and Foundation

layers

Staging Layer

Foundation Layer

ODI from Foundation for complex calculations

and integration

Oracle Data Integrator (ODI)

•Data Integration tool within Oracle Fusion Middleware 12c •Robust sources and targets • Relational databases • Essbase • Flat file and XML •Web services • Big Data

•Declarative design • Logical implementation • Physical implementation

•ETL or E-LT •Database as ETL engine • Java Agents as ETL engine

Where Have We Seen This Before?

•Abstraction of physical sources using logical entities •Logic built into a model instead of a process •Metadata-based paradigm selection •“Do the work once”

Abstraction of physical sourcesLogic built into Models not Processes Metadata-based paradigm-selection

ETL Development: Joins with Declaration

ETL Development: Collateral from Model-Driven Development

WITH SAWITH0 AS (select count(T2196.ID) as c1, sum(T2196.AMOUNT) as c2, concat(concat(T3853.FIRST_NAME, ' '), T3853.LAST_NAME) as c3, T2196.LEAD_SOURCE as c4, T2196.SALES_STAGE as c5 from SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ , SUGARFND.USERS T3853 /* Users (Account Owner Role) */ , SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ , SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */ where ( T2041.ASSIGNED_USER_ID = T3853.ID and T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID ) group by T2196.LEAD_SOURCE, T2196.SALES_STAGE, concat(concat(T3853.FIRST_NAME, ' '), T3853.LAST_NAME)), SAWITH1 AS (select distinct 0 as c1, D1.c3 as c2, D1.c4 as c3, D1.c5 as c4, D1.c2 as c5, D1.c1 as c6 from SAWITH0 D1) select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, D1.c7 as c7, D1.c8 as c8 from ( select D1.c1 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c5 as c5, D1.c6 as c6, sum(D1.c5) over (partition by D1.c3) as c7, sum(D1.c6) over (partition by D1.c3) as c8 from SAWITH1 D1 order by c1, c7 desc, c3 ) D1 where rownum <= 65001 4

ETL Development: Finished Product

Build Now, Refactor Later

Front-end developers

aggressively close stories any

way they can

Complex calculations,

direct SQL, BI Publisher, Custom

JavaScript: Nothing is off

limits

A new “Refactor” Story is opened for the Metadata

Developer to consider this content for

inclusion in the RPD

Metadata developer decides if the complex

front-end content is viable

If viable, the developer aggressively

solves the Refactor Story using complex

calculations or opaque views

The original Refactor Story is reassigned to

the front-end developer to have the analysis point instead to the new metadata

A new Refactor Story is opened for the ETL

developer for any complex

transformations that may belong in the ETL

ETL Developer decides if the

transformations are viable, and if so, adds new transformations

The Refactor Story is then

reassigned to the Metadata

developer to rework the RPD

using new database attributes

Where did our Real-Time go?

Realtime Integration

ETL added latency Pure batch: significant Micro batch: medium

We lost our Realtime capability when we added our Access & Performance

Realtime Integration

ETL added latency Pure batch: significant Micro batch: medium

We need to a way to get that back utilizing the

Foundation layer

OBIEE Fragmentation

Realtime: Foundational Federation

WITH SAWITH0 AS (select T8974.OPPORTUNITY_COUNT as c1, T8974.AMOUNT as c2, T8939.INDUSTRY as c3, T8981.LEAD_SOURCE as c4, T8981.NAME as c5, T8981.OPPORTUNITY_ID as c6 from EDW.OPPORTUNITY_DIM T8981 /* Opportunity Dim (Standard) */ , EDW.ACCOUNT_DIM T8939 /* Account Dim (Standard) */ , EDW.OPPORTUNITIES_FACT T8974 /* Opportunities Fact (Standard) */ where ( T8939.ACCOUNT_KEY = T8974.ACCOUNT_KEY and T8939.INDUSTRY = 'Technology' and T8974.OPPORTUNITY_KEY = T8981.OPPORTUNITY_KEY and T8981.LEAD_SOURCE = 'Conference' and 'EDW' = 'Foundation' ) ), SAWITH1 AS (select cast(1 as INTEGER ) as c1, T2196.AMOUNT as c2, T2041.INDUSTRY as c3, T2196.ID as c4, T2196.LEAD_SOURCE as c5, T2196.NAME as c6 from SUGARFND.ACCOUNTS T2041 /* Accounts (Standard) */ , SUGARFND.OPPORTUNITIES T2196 /* Opportunties (Standard) */ , SUGARFND.ACCOUNTS_OPPORTUNITIES T2221 /* Accounts Opportunities (Standard) */ where ( T2041.ID = T2221.ACCOUNT_ID and T2196.ID = T2221.OPPORTUNITY_ID and T2196.LEAD_SOURCE = 'Conference' and case T2041.INDUSTRY when ' ' then 'Not Applicable' else T2041.INDUSTRY end = 'Technology' and TO_DATE('2013-08-15 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T2196.DATE_ENTERED ) ), !(continued...)

SAWITH2 AS ((select D0.c3 as c3, D0.c4 as c4, D0.c5 as c5, 'EDW' as c6, D0.c6 as c7, D0.c1 as c8, D0.c2 as c9 from SAWITH0 D0 union all select case D0.c3 when ' ' then 'Not Applicable' else D0.c3 end as c3, D0.c5 as c4, D0.c6 as c5, 'Foundation' as c6, D0.c4 as c7, D0.c1 as c8, D0.c2 as c9 from SAWITH1 D0)), SAWITH3 AS (select sum(D3.c8) as c1, sum(D3.c9) as c2, D3.c3 as c3, D3.c4 as c4, D3.c5 as c5, D3.c6 as c6, D3.c7 as c7 from SAWITH2 D3 !(continued...)

BI Server does a union between the

Foundation and Performance layers

Union is eliminated when the

“fragmentation column” is used as a

filter

AgileAnalytics: Real-Time BI with Oracle Business Intelligence, Oracle

Data Integrator and Oracle GoldenGate

Stewart Bryson

medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

AgileAnalytics: Agile Real-Time BI with Oracle Business Intelligence, Oracle Data Integrator and...

Technology

Oracle goldengate ogg

Oracle Goldengate

Installing and Configuring Oracle GoldenGate Veridata 12c ...€¦ · Oracle® GoldenGate Installing and Configuring Oracle GoldenGate Veridata 12c (12.2.1.2.0) E78470-04 December

Integrating Oracle GoldenGate for Big DataIntegrating Oracle GoldenGate for Big Data Release 12.3.1.1 E89478-02 September 2017. Oracle Fusion Middleware Integrating Oracle GoldenGate

Oracle GoldenGate high availability using Oracle … · Oracle White Paper—Oracle GoldenGate high availability with ... Oracle GoldenGate high availability using Oracle ... •

Oracle GoldenGate Best Practices: Configuring Oracle ... · PDF fileOracle GoldenGate Best Practices: Configuring Oracle GoldenGate Monitor 12c An Oracle White Paper August 2014 Oracle

Installing, Configuring, and Upgrading Oracle GoldenGate ... · Oracle® GoldenGate Installing, Configuring, and Upgrading Oracle GoldenGate Monitor Agent 12c (12.2.1.2.0) E79732-03

Oracle GoldenGate 12c - oracledwh.de€¦ · 1 Oracle GoldenGate 12c - Was ist neu seit 11gR2 ? - GoldenGate 12c Database & Middleware, Oracle 12c Multitenant GoldenGate 12c für

[1]Oracle® GoldenGate Installing and Configuring …Oracle GoldenGate Installing and Configuring Oracle GoldenGate for Sybase amount of RAM available to Oracle GoldenGate may significantly

Oracle GoldenGate Director

Mark Ritter · The Magazine for the Rocky Mountain Oracle Users Group • Vol 58 • Spring 2010 Mark Ritter Oracle GoldenGate and Oracle Data Integrator ... Active DataGuard Gain

Oracle Enterprise Metadata Management - nlOUG · Data Flow Lineage Model Versioning Model Browser ... Oracle Data Integrator, Oracle GoldenGate Oracle Business Intelligence Enterprise

The Forrester Wave™ - Oracle · 2016-10-03 · Oracle Data Integrator Oracle GoldenGate Oracle Big Data Preparaon Oracle Big Data SQL Oracle Big Data Connectors Oracle NoSQL Database

[1]Oracle® GoldenGate Using Oracle GoldenGate …1]Oracle® GoldenGate Using Oracle GoldenGate Monitor 12c (12.2.1) E60967-01 October 2016 This document contains background information

Oracle® GoldenGate Application Adapters for BASE24 · PDF fileIntroducing Oracle GoldenGate for BASE24 Oracle GoldenGate for BASE24 overview Oracle GoldenGate Application Adapters

Oracle GoldenGate Administering Oracle GoldenGate for ... · Oracle® GoldenGate Administering Oracle GoldenGate for Windows and UNIX 12c (12.1.2) E29388-06 November 2014 Documentation

Oracle GoldenGate on Linux for zzseriesoraclesig.org/2016presentations/11_Oracle_GoldenGate_Update.pdf · GoldenGate >=11.2 – Integrated Capture Source: Oracle GoldenGate Windows

Oracle GoldenGate Hands-on Tutorial - … 4 Oracle GoldenGate Hands-on Tutorial Tutorial: Basic Replication between two Oracle Databases _____ 19 Customizing GoldenGate …

Introducing Oracle Data Integrator and Oracle GoldenGate Marco Ragogna EMEA Principal Sales Consultant Data integration Solutions

Oracle GoldenGate Windows and UNIX Administrator’s … · Oracle GoldenGate Windows and UNIX Administrator’s Guide 7 PREFACE About the Oracle GoldenGate guides. . . . . . .