Real-Time Operational Intelligence

Presented by Nelson Petracek, Informatica npetracek@informatica.com

Oct. 17, 2013

DAMA NY: DAMA Day Oct. 17, 2013

The World has Changed…And So Must Business

The “New” World of Business

The takeaway is we’re now in a time of overwhelming

volatility, uncertainty, complexity, and ambiguity, where

forecasting becomes nearly impossible, and even

determining which of the risks confronting us are most

significant is almost unimaginable.

3

Stowe Boyd

https://www.odesk.com/blog/2013/08/futurist-analyst-stowe-boyd-dawn-work-placeforms/

• It is no longer sufficient to view

information “after the fact”.

• Business demands information

sooner, with more accuracy, in order

to meet competitive and regulatory

demands.

• Business needs to respond to

“threats” and “opportunities sooner.

• Reduce decision latency.

• Proactive alerts and notifications.

• Improve TTA (time to answer).

Changing Perspectives on Data

5

MORE

AGILITY

RIGHT

Time

PROACTIVE

vs.

REACTIVE

TRUST

Self-

Service

Fresh

Information

All Data

One

Place

Immediate

Response

Times

100%

Uptime

User Expectations

Traditional Data Delivery Process (BI)

Store

Analyze

Act

ETL EDW BI

Traditional Data Delivery Process (BI)

Store

Analyze

Act • Takes too long.

• Lots of “Wait” and

“Waste”

• No common and

trusted data access.

• Information is missing

or is stale / delayed.

• Too much “decision

latency”.

What Can Help?

• Solution approach that

complements and augments

traditional BI and reporting

solutions.

• Combines approaches and

techniques from various

technology areas, including:

• Data Integration

• Event Driven Architecture

• Event Processing

• Data Replication / Virtualization

• Reactive Applications

“Real Time Operational Intelligence”

Sense

Reason

Respond

Visualize

What is “Real Time”?

• Definition varies, depending on the

use case.

• Doesn’t always refer to situations where

event/data delivery/receipt is measured in

milliseconds or nanoseconds.

• One definition:

• Function is performed within a timeframe that

the user / application senses as immediate or

current.

• Often thought of as the processing latency

encountered after the capture of a “state

change” within the enterprise.

What is an Event?

• Events represent any change

in state throughout the

enterprise.

• Sensor Read

• Social Media Posting

• Location Change

• RFID Read

• Database Operation

• File Arrival

• Missing events are

themselves an event.

• Business process step “not” executed by a particular time.

• Event “not” received so many seconds/minutes/hours after a previous event.

• “No” readings received for a particular time period.

Data Integration

Traditional Grid Transactions,

OLTP, OLAP

Social Media, Web Logs

Machine Device,

Scientific

Documents and Emails

Analytics & Op

Dashboards

Mobile

Apps

Real-Time

Alerts

Archive Profile Parse Cleanse ETL Match

Event Driven Architecture

• An architecture in which the activity is driven by changes

in state within an environment.

• Events drive the execution of logic.

• Communication is typically asynchronous, decoupled.

Event Transport

Producers / Publishers

Consumers / Subscribers

Event Processing

• Solution approach that deals with making “sense” out

of events from multiple sources.

• Events may be combined with other sources of data to

detect “situations” of interest.

• Identify and respond to “threats” and “opportunities”.

Patterns

Alerts

Event Streams

Eve

nt

So

urc

es

(P

rod

uce

rs)

Actions

Dete

cte

d S

ituatio

ns

(Co

nsu

me

rs)

Devices

Systems

Applications

People

Inte

gra

tio

n /

Dir

ec

t Inte

gra

tion

/ Dire

ct

Pre-

Process

Process

Post-

Process

Data Replication / Virtualization

• Data Replication

• Data Virtualization

Transactional / Production

Applications

Merge / Apply,

Reports & Queries

Source System Target Systems

ODS / OLTP /

DW / Appliance /

Big Data Stores

• Event-enable source systems.

• Off-load transactional

systems.

• Provide freshest information.

Un/Semi structured

Data

Applications EDW Data Marts

Common Data Access Layer – Logical Data Object • Expose logical data objects to

external applications.

• Externalize transformations,

pre-compute data object

views.

• Hide source complexity.

Reactive Applications

• Describes design properties that apply across the

technology stack.

• Reactive applications are ones that are:

• Responsive

• Scalable

• Resilient

• Event-Driven

• Applications are designed to react to events, react to

load, react to failure, and react to users.

• Avoid contention.

• Allow recovery at all levels.

• Honor all response time guarantees.

Operational Intelligence (OI) Process

OI System

Action

• Proactive actions

instead of reactive.

• Users are “pushed”

the information they

need, when they need

it.

• Allows the business to

define the conditions

and rules.

Lack of “Operational Intelligence”?

“Big Data” Supply Chain Context

Business

Value

Big Data

P&L Goals

Generate

Insights

Inspire

Action

Validate

Hypothesis

Make

Operational

Prioritize

Goals

Data Scientist Analyst Engineer Business

Acquire &

Store

Explore &

Curate

Distribute

& Manage

Big Data Supply Chain

Refine &

Enrich

Data Management & Analytic Systems

Ops Dashboards &

Mobile BI Apps

Transactions,

OLTP, OLAP

Social Media, Web Logs

Machine Device,

Scientific

Documents and Emails

Data Warehouse

4. Collect real-

time events

3. Integrate data to

identify patterns & trends

2. Stream real-time

data

5. Correlate real-time

events with historical

patterns & trends

1. Replicate

change data

6. Deliver insights &

alerts in real-time

Data Virtualization

MDM

Real Time Operational Intelligence

“Lambda Architecture”

Adapted from “Runaway Complexity in Big Data”, Nathan Marz, Sept. 25/2012

Transactions,

OLTP, OLAP

Social Media, Web Logs

Machine Device,

Scientific

Documents and Emails

Batch Layer

Batch

View

Big Data Analytics + Real Time Streams

Speed Layer

Real Time

View

Serving Layer

Merged

View

Case Study: Healthcare

Silos of

Information

Fragmented

Service

Delivery

“After the

Fact”

Reporting

Reactive

Decisions

Disjointed

Communication

Limited Cross-

Department

Collaboration

Proactive vs. Reactive

Imp

act

Time

“Preventable cause of death

identified retrospectively”

“Which congestive heart failure (CHF) patients

are most likely to be readmitted?”

"Patient XYZ was discharged 10 days ago with chronic obstructive pulmonary

disease (COPD) and there is no record of an appointment scheduled with her PCP.

Please call her."

Immediate Action for High Impact

Architectural Implications

• Shift to modeling events as enterprise assets. Events become first-class citizens.

Information and Data

Architecture

• Shift in thinking from procedural to declarative development.

• Focus on allowing users to control aspects of the system.

Application Architecture

and Development

• Shift from centralized, database-centric client-server applications to distributed systems.

Infrastructure Architecture

Summary

• Augment traditional BI solutions with real-time

operational intelligence to:

• Reduce decision latency.

• Improve decision making and responsiveness.

• Increase visibility.

• Improve “time to answer” and “fail faster”.

• Build upon the principles of an event-driven architecture

and reactive applications.

• Incorporate solutions that are event driven, scalable,

resilient, and responsive.

Questions? www.operationalintelligence.me

26