Real-time Data Integration for Business Activity Monitoringjs/download/bpim_workshop.pdfSource Systems ERP... Database External Sources WFMS Legacy Systems Message Driven EJBs. Watson

Watson Research Center

Real-time Data Integration Real-time Data Integration for for Business Activity Business Activity

MonitoringMonitoring

Josef SchieferIBM Watson Research

Workshop in Adaptive Business Operations02-25-2003

>> A Container Based Approach>> A Container Based Approach


Gaining Insight into the Business

Tech

nolo

gy E

volu

tion Data Warehousing

Business Process Warehousing

Real-Time Business Monitoring

Adaptive Decision Support Systems

Technology Evolution Technology Evolution forforGaining Insight into the BusinessGaining Insight into the Business


Wor

kflo

w M

anag

emen

t Sys

tem Automatic Feedback

Input: Event DataOperation: Automatic Comparison, EvaluationResponse: Adjustment of available quantities(e.g. workload-based staff assignment, changing priorities)

Strategic Management & Control

Input: High-level Quality ParametersOperation: Manual Comparison, AlarmsResponse: Adjustment of process and environment

Operative Management & ControlInput: Process-specific Quality ParametersOperation: Manual Comparison, AlarmsResponse: Adjustment of available quantities (e.g. schedulingadditional staff members, changing inventory levels) Filter

FilterOperative System

Management System

Instant (real-time) response time for decision making can be guaranteed (seconds); theperiod of time during which the decision has an effect is very short. Decisions are made

very frequently and instantly when the events are raised.

Time horizon: months – years. A decision is made often only once.Historical data is very critical. Includes high-level business metrics.

Leve

ls o

f Le

vels

of

Res

pon

sive

nes

sR

espo

nsi

ven

ess


Task 1 Task 2 Task 3 Task 4 Task 5 ProcessOutcomes

Customer

Sub Process 1Sub Process 2

M3

M4 M4

M3

M4 M4 M4

M3 M1M1

Internal

ExternalM2

Manager ObjectivesM3

- Reduce service costs- Reduce order processing time- Improve service quality- Optimize resource assignments

Workgroup ObjectivesM4

- Reduce defects/waste/delays- Improve customer rating- Automate processing steps- Early error detection

Vice Pres./Director ObjectivesM2

- Shorten delivery and supply time- Increase Return on Investment- Increase delivery performance

Executive ObjectivesM1

- Fully exploit customer potential- Improve profitability- Customer satisfaction

Process Monitoring LevelsProcess Monitoring Levels


Data Integration ChallengesData Integration Challengesfor BAMfor BAM

Data Propagation- Near real-time- Continual- Complex ETL processing- Many, but small data extracts

Adding Business Process Context- Context consists of

* Process Description* Organizational Context* Business Data* Causal Context Data

- Context information must becollected from several data sources

- Business Metrics on various abstraction levels

Configurability / Manageability- Configuration Changes in Runtime- Deployment in existing DWH environments- Reflection of data propagation status

Evaluation of Business Metrics- Evaluation of SLAs- Response Mechanisms- Notifications

Solution Management- Solution Templates- Reusable, pluggable ETL components- Adaptive Architecture


J2EE ETL EnvironmentJ2EE ETL Environment

Pro

prie

tary

Con

nect

ors

EJB Container

J2EE

ETL

Env

ironm

entJava Services

JAAS, JNDI, JMS,JAXP, JCE, JTA,

JAF, ...

ETL Container

Res

ourc

e A

dapt

ers

Evaluators

RangeEvaluators

CustomEvaluators

CompareEvaluators

Event Adapters

JMS EventAdapter

JCA EventAdapter

MQ SeriesEvent Adapter

JCA EventAdapter...

ETLets

Event DrivenETLets

ScheduledETLets

ExceptionETLets

Session Beans

ConversionEJBs

AssemblyEJBs

ParsingEJBs

CleansingEJBs...

CMPEntity Beans

BMPEntity BeansE

AI/B

PI

Con

nect

ors

Dat

abas

eC

onne

ctor

sJ2

EEC

onne

ctor

sJM

SC

onne

ctor

s

Sour

ce S

yste

ms

ERP

...

Database

External Sources

WFMS

Legacy Systems

Message DrivenEJBs


ETL Container - PurposeETL Container - Purpose

Extracting raw event data from various source systems

Calculation of business metricsPersisting event data and business metrics

Coordination of ETL processing/transformation (--> processing flow)

Evaluation of business metricsAllows flexible (re-)configuration/deployment of ETL components

Separates extraction, transformation, and evaluation logic(--> pluggable components)

Provides a configurable, scalable and high-performance ETL environment for a large number of events


Batch vs. ContinuousBatch vs. ContinuousData IntegrationData Integration

OperationalSource

OperationalSource

OperationalSource

OperationalSource

WFMS

Batch Data IntegrationLoading

TransformationExtraction

EnterpriseData Warehouse

ProcessData Store

ProcessWarehouse

Business ProcessMonitoring

ETL Container

EventAdapters

ETLets Evaluators

ProcessInformation

Factory

WorkflowEvents

Response

Notifications

Workflow Metrics


Event Adapters: Dequeue events and transform them into a standard event formatETLets process the standardized events and calculate the business metricsEvaluators evaluate the calculated business metrics

Real-time Data Integration Real-time Data Integration with an ETL Containerwith an ETL Container

(Event Processing, Calculation & Evaluation of Business Metrics)(Event Processing, Calculation & Evaluation of Business Metrics)

Cleansing(Parsing, Correction,

Standardization,Completion)

Matching Consolidation,Transformation

MetricCalculations

ProcessWarehouse

Message Queue

J2EE Connector

JDBC Connector

EventAdapters ETLetsStandardized

XML Events

ProcessData Store

EvaluatorsMetric

Evaluation

Rule EnginesETL Container

...


Real-Time Event ProcessingReal-Time Event Processing

Managed straight-through ETL processing without intermediary storage!!

ETL Container

Eve

nt A

dapt

er Thread 1

Thread 2

Thread 3

EventHandler

ETLetService

ETLet 1

ETLet 2

ETLet 3

...

EvaluationService

Evaluator 1

Evaluator 2

Evaluator 3

MetricHandler

MetricDispatcher

Event 1

Event 3

Event 2

Eve

nt D

ispa

tche

r -- Workflow Event Processing --E

vent

Ada

pter

Eve

nt A

dapt

er

ProcessData Store

ProcessWarehouse

...


BPSM ArchitectureBPSM Architecture

Policy Managem

ent

EventAdapters

ETLets Evaluators

Pro

cess

Eve

nts ETL Container

ProcessWarehouse

ProcessData Store

Message Bus

BusinessMetrics

Message Bus

ProcessInformation

Factory

BI Agent Layer

Dashboard Facade

IntegrationLayer

ReactiveAgent

DeliberateAgents

ReflectiveAgents

Policies

SensingAgents

ResponseAgents

WFMS

Process 2

Process 3

Process 1 BusinessApplications

Service BeansUser BeansManagement Widgets

Dashboard

Presentation Layer


ETL Container - StrengthsETL Container - Strengths

Lightweight, near real-time ETL processing (Threads)

Support of complex transformations / processing

Evaluation capabilities of calculated business metrics

Usage of existing J2EE infrastructure

Container managed, optimized, reusable, configurable ETL components

Clean separation of Extraction logic (EventAdapters)

Transformation logic (ETLets)

Evaluation logic (Evaluators)

Event handling is separated from the database

Event-driven actions linked to the near real-time data integration

Development methodology very similar to Web Applications (with Servlets)


When NOT to use Near Real-Time Data Integration:

Near real-time data propagation is NOT a requirement

Existing data replication solution are sufficient

Very large data volumes (--> bulk loading)

When NOT to use an ETL container:Transformations can be done with SQL or even not necessary

Configurability is provided by ETL tools

No Java development skills available

When NOT to use Near Real-Time When NOT to use Near Real-Time Data Integration / ETL ContainerData Integration / ETL Container


Future Work: Future Work: Adaptive BAM SolutionsAdaptive BAM Solutions

Process 2

Process 3

Process 1

BPEL

BPEL

BPEL

WFMS ETL Container

Even

t Ada

pter Thread 1

Thread 2

Thread 3

EventHandler

ETLetService

ETLet 1

ETLet 2

ETLet 3

...

EvaluationService

Evaluator 1

Evaluator 2

Evaluator 3

MetricHandler

MetricDispatcher

Event 1

Event 3

Event 2

Eve

nt D

ispa

tche

r -- Processing of Workflow Events --

Even

t Ada

pter

Eve

nt A

dapt

er

ProcessData Store

ProcessWarehouse

PIF Builder

SchemaDefinition/

Composition

SchemaGeneration

SchemaDefinitions

Configuration

WorkflowEvents

ETLConfiguration

Process Start Time

Customer

Coverage

Process

activity_time_keypolicy_keycustomer_keyagent_keycoverage_keycoverage_item_keyprocess_keyprocess_outcome_keyamountsum_waiting_timeprocessing_timeprocess_runtimeprocess_runtime_devprocess_state_times…

Policy

Agent

Covered Item

Process Outcome

Process Compl. TimeProcess Start Time

Customer

Coverage

Process

activity_time_keypolicy_keycustomer_keyagent_keycoverage_keycoverage_item_keyprocess_keyprocess_outcome_keyamountsum_waiting_timeprocessing_timeprocess_runtimeprocess_runtime_devprocess_state_times…

Policy

Agent

Covered Item

Process Outcome

Process Compl. Time

1 2 3

...

BPCL


Thank You!Thank You!

Questions?Questions?


Backup Slides

Backup

Backup Slides


Integrated data foundation for the monitoring & analysis of business processes. The PIF adds the process context to existing DWH environments

Enables process-oriented view of all business data

Provides new process key performance indicators

Near real-time integration of workflow events

Process Information Factory (PIF)Process Information Factory (PIF)with Container Managed Data Integrationwith Container Managed Data Integration

Workflow Data

Business Data Integrated process-orienteddata foundation for a

process-oriented monitoring/analysis


Transaction Processing vs.Transaction Processing vs.Informational ProcessingInformational Processing

Level 1 - Transactional Processing

OrderProcessing

Level 2 - Informational Processing

Fulfillment Billing Payment

...

TR-75

Miller

Singer

List

Risher

TR-75

Miller

Singer

List

Risher

531 539 652 683 867531 539 652 683 867

Jun.May.

Apr.Mar.Feb.Jan.

Jun.May.

Apr.Mar.Feb.Jan.

Sale

sA

ssis

tant

Sale

sM

anag

er VI

DEO

all

TraditionalSale

InternetSale

1. quarter

2. quarterHJ1

1. quarter

2. quarterHJ1

Participant

i me

P r o c e s s

Jones

Sale

Uni

t

2727

531Jan.

Miller

Singer

List

Rahl

T

O r g a n i z a t i o n

Real-tim

e InformationR

eal-t

ime

Dat

a In

tegr

atio

n

ProcessWarehouse

ProcessData Store

EnterpriseData Warehouse


Comparison:Comparison:DWH, ODS, PDS, PWHDWH, ODS, PDS, PWH

Criteria:Basic orientation:

Business view:Process information:

Time references:Access:

Aggregation level:Integration level:

Accessibility:

Legend: Data managed by transaction systemsData managed by operational data storesData managed by traditional data warehouse systemsData managed by process data storesData managed by process warehouses

transactionlocal

no informationactual

read/writedetailedisolated

real-time

subject-orientedenterprise-widefull informationactual + historyread-onlyaggregatedintegrateddelayed


Metric Handler( calls the metric evaluators)

Binding between ETLets andMetric Evaluators are defined in the deployment descriptor

Metric Handler( calls the metric evaluators)

Binding between ETLets andMetric Evaluators are defined in the deployment descriptor

Implementation: ETLets, EvaluatorsImplementation: ETLets, Evaluators

Documents

Real-time Data Integration for Business Activity Monitoringjs/download/bpim_workshop.pdfSource Systems ERP... Database External Sources WFMS Legacy Systems Message Driven EJBs. Watson