CEP Overview v1 2 for public use

© Copyright 2000-2012 TIBCO Software Inc.

CEP 101

Complex Event Processing

Paul Vincent

TIBCO Software


PRESENTER

Paul Vincent, CTO Business Rules and CEP ◦ Part of TIBCO QL product team

◦ Contributor to OMG DMN Decision Model and Notation standard

◦ Contributor to http://www.thetibcoblog.com category on CEP

TIBCO Software Inc.: ◦ Software tools+solutions for “event enabled enterprises”

◦ History: pub-sub middleware (eg RV), standard event processing (eg BW)

◦ Recent innovations, news —Hardware Logging and JMS servers

—Statistical server for R and S+ analytic languages

—Event processing combined with decisions and BPMN

◦ Headquartered in Palo Alto, California

◦ Over 3,000 customers and offices in 40 countries

http://www.thetibcoblog.com/




TOPICS

Intro to Complex Event Processing

IT perspective and Technologies

Examples

CEP Market, Tools Available


2-3 DECADE’S CHANGES IN BUSINESS LATENCY

Perform Trading Analytics

Document Transfer

Handle Call Center Inquiries

Settle Trade/Transaction

Build-to-Order Product

Refresh Data Warehouse

Airline Operations

Track Financial/Risk Position

Supply Chain Updates

Phone/Service Activation

1 month

3 days

6 weeks

5 days

30 minutes

20 minutes

8 hours

1 day

1 day

3 days

1 hour

Mail… express… fax… e-mail 20 seconds

1 day

1 day

5 seconds

30 seconds

10 seconds

5 minutes

15 minutes

1 hour

Process Cycle-Time Improvements

Source: Gartner


REAL-WORLD EVENTS

Customer

Logon Fed

Base Rate

Increase

Production

Item

Arrives at

Store

Customer

Checks

“Close Account”

Web Page

New

Liability

Added

Rental

Car

Returned

New

Order

Contract

Submitted

Contract

Returned

thru EDI Rental

Car

Crashed

Mobile Call

from CT

@11.13


Customer

Logon Fed

Base Rate

Increase

Production

Item

Arrives at

Store

Customer

Checks

Close Account

Web Page

New

Liability

Added

Rental

Car

Returned

New

Order

Contract

Submitted

Contract

Returned

thru EDI Rental

Car

Crashed

Mobile Call

from CT

@11.13

WHERE IS THE INFORMATION?


7

Fraud

Risk! Risk of

Customer

Defection

Employee

Over hours

Customer

CrossSell

Opportunity

Compliance

Limit

Approached

Customer

now rated

Gold

Change in

Product Sales

Trend

Contract

Validated

Contract

Valid Rental

Contract

Complete

Cell phone

fraud alert

WHERE IS THE INFORMATION?


COMPLEX BUSINESS PROBLEMS

Fraud / Theft ◦ Thousands-to-millions of high-value small-size product

items or transactions ◦ How do you identify known patterns of “suspicious”

behavior?

Logistics / Scheduling ◦ Raw material, production & delivery scheduling and

resources are complex and prone to change ◦ How do we reallocate resources to handle business and

production changes?

Activity Monitoring ◦ Complex production and supply process with multiple

actors ◦ How to measure and action Key Performance Indicators?

Product location

change

Relevant event of interest

Resource,

requirement change

Process data change


TYPES OF “EVENT”

Change of state in some entity

◦ Customer call

◦ Bank debit

◦ Aircraft movement

Observation of some entity

◦ CRM record of a customer call

◦ ATM report of debit transaction success

◦ Radar plot update of an aircraft

IT Message

◦ Queued point-to-point message

◦ Publish / subscribe message

Incident

Observation

IT Message


ASSOCIATED EVENTS

Positive Events ◦ Product item X arrives at Production station S from Store T

◦ Production worker Y arrives at Production station S

◦ Production contract for item Z by time T is posted

Negative Events ◦ Product item X has been in transit to Store T for >15 minutes

◦ Subcomponent Y hasn’t arrived at the Production station by the ETA

◦ Delivery of contract Z has not taken place

Sets of Events ◦ 5+ items of Product item type Y failed to arrive at destination

◦ Supplier Y was 5 mins late for 1 delivery, but made it early to the next

◦ Return rate on component Z exceeds SLA %


SIGNIFICANT FEATURES OF THESE EVENTS

Time Sensitivity ◦ A thief may leave the building at the same time as stolen

product

◦ A product should take 40 minutes to travel a given production line segment

Distributed Event Sources ◦ A series of produced items fails at various QA stages, and their

common attribute was a storage location

◦ Multiple suppliers for a subcomponent are reporting delivery delays

ERP Supplier

t


12

THE EVENT CLOUD

Can we infer important business events by correlating events automatically + earlier, regardless of source / type?

RFID events

Timeout events

Supply chain events

Delivery events

Personnel events


“The events we have access to are not always

tailored to the problems we are trying to solve.

Therefore, we need a technology that enables

us to progress in stages.

1: The first stage is recognizing relevant patterns of

events in the sources of events we do have access to

and can monitor.

2: The second stage is aggregating information in those

events to build up information that is needed to solve

our problems.”

Dr. David Luckham - Stanford University

Author, The Power of Events

POWER OF EVENTS


EPTS ON DERIVED AND COMPOSITE EVENTS

Derived event (also synthesized event): an event that is generated as a result of applying a method or process to one or more other events.

◦ Example: —A “Company B is a good buy” event may be derived from “Company B price has dropped

where Company A price has risen within 5 minutes”

◦ Note: The absence of an event, say in a given time interval, can lead to a derived event for the missing event.

Composite event: a derived, complex event that is created by combining base events using a specific set of event constructors

◦ Operators include disjunction, conjunction, sequence, etc.

◦ A composite event always includes the base (member) events from which it is derived.

◦ Note: A derived event may not be a composite event if its method of derivation lies outside a specified set of allowed constructors.

Note: The term aggregate event is sometimes used for some forms of composite or derived event.


EPTS ON COMPLEX EVENTS

Complex event: an event that is an abstraction of other events ◦ Abstracted events are the members of the complex event

◦ Examples: —the 1929 stock market crash (an abstraction denoting many thousands of member events),

—the 2004 Indonesian Tsunami (an abstraction of many natural events) ,

—a completed stock purchase (an abstraction of the events in a transaction to purchase the stock).

—a successful on-line shopping cart checkout (an abstraction of shopping cart events on an on-line website).

◦ Note: a complex event denotes or signifies the set of its member events. This may or may not include references to individual member events.

A composite event or a derived event is a complex event. The converses are not necessarily true.


EPTS ON EVENT PROCESSING

Event processing: computing that performs operations on events, including reading, creating, transforming and deleting events.

◦ Note: The overloaded meaning event object processing is intended in this context.

Complex-event processing (CEP): Computing that performs operations on complex events, including reading, creating, transforming or abstracting them.

◦ Note: CEP results in creating complex events even if some or all of the source events are simple events


ALTERNATIVE TERMS

CEP (technology) applies pattern detection & filtering to the event clouds & streams and their histories

Multiple modelling / execution paradigms are available for pattern detection

Related terms: real-time analytics, continuous intelligence, event processing, …

Event Streams

CEP

RFID events

Timeout events

Supply chain events

Delivery events

Personnel events

Event Cloud


WHAT DOES CEP SOLVE? P

ote

nti

al

Bu

sin

ess

Va

lue

Business Event

Resulting Complex Event Measured

Root Cause / Correlation

Corrective Decision Made

Action Taken

Action time

warnings precede threats

CEP allows fast

responses to detected

issues

… the “ Latency Problem“


TOPICS



Examples



USE CASE PATTERNS

“Situational Awareness”

“Sense and Respond”

“Track and Trace”


CEP’S ROLE IN I.T.


COMPLEX EVENT PROCESSING

Processing type:

Decision

Processing

Event

Processing

-- TIBCO Reference Architecture by Alan Lundberg

2005


EVENT-DECISION ARCHITECTURE

EVENT

SOURCES

EXTERNAL

.

.

.

User

Interface

&

Systems

Event-Decision Architecture

DISTRIBUTED

LOCAL

EVENT

SERVICES

.

.

EVENT

PROFILES

.

.

DATA

BASES

.

.

OTHER DATA

EVENT PRE-

PROCESSING

LEVEL ONE

Event Tracking &

Refinement

EVENT + DATA STORE

Historical

Data

Profiles &

Patterns

LEVEL TWO

Situation Detection

& Refinement

LEVEL THREE

Impact Assessment

& Predictive

Analysis

LEVEL FOUR

Process Refinement

& Adaptation

-- Adapted from JDL Data Fusion Model by Tim Bass

Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001


EPTS FUNCTIONAL REFERENCE ARCHITECTURE

-- EPTS Reference Archjitecture

by EPTS RA Working Group

editors Alex Alves, Adrian Paschke,

Catherine Moxey, Paul Vincent

Proceedings of the 6th ACM

International Conference on Distributed

Event-Based Systems, ACM, 2012


STANDARD OR SIMPLE EVENT PROCESSING

BPM: business

process

SOA:

IT services

Data Persistence (service)

new order fulfilment

Synchronous Events

Fixed to specific event types

Fixed to specific event types


MULTIPLE SIMPLE EVENT PROCESSING

Multiples of

Processes

Multiples of

Services

Multiples

of Database Views


new supplier

change order

delivery

purchasing

rectification

billing


MULTIPLE SIMPLE EVENT PROCESSING (THEORY)

Multiples of

Processes

Multiples of

Services

Multiples

of Database Views


new supplier

change order

delivery

purchasing

rectification

billing


COMPLEX EVENT PROCESSING

BPM: business

process

Data Persistence (service)

SOA: IT

services

source

events

Continuous

identification of event

patterns

CEP: complex

event processing

State store

business events

Asynchronous Events

Flexible to any event types


REQUIREMENTS FOR CEP TECHNOLOGY

Access and Monitor the “Event Cloud”

◦ Instrumentation (for state changes)

◦ JMS, RV, MQ, TCP/IP, etc…

◦ Timers to detect lack of events

Match Patterns (+ Apply Business Logic)

◦ Detect events

◦ Detect event patterns

◦ Maintain State and Facts over time

◦ Update Detection algorithms as events change

Pattern Detection Model

for recognizing patterns,

defining actions

Temporal Model

for determining

time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable

message delivery

His

tory

an

d S

tate


CEP TECHNOLOGIES

Event

Sources

Event

Consumers

Event

Bus

Types of CEP Processing

Rule-Driven CEP

Rule Engine Rulebase

Event Stream Query CEP

Query Engine Stream Queries

Neural Net CEP

Neural Net Network

Orchestrated Event Stream CEP

Flow Engine Flow Model


State Model

CEP BEHAVIOR: STATE-ORIENTED

State 1 State 2

New

Order New

Order New

Order

Condition /

TimeEvent


STATE MODEL CEP AGENT FEATURES

1. Visual modeling metaphor ◦ State diagram / flow diagram is simple to follow

2. State / flow transitions can be time-related ◦ Can model missing events through time-outs etc


Rule

CEP BEHAVIOR: RULE-ORIENTED

Complex

Patterns

New

Order New

Order New

Order

(Lots of

Events)

Reaction


INFERENCE RULE CEP AGENT FEATURES

1. High performance pattern matching ◦ Rete (or other) algorithm determines what rules are executable based on

underlying data changes

2. Declarative + Inferencing ◦ Rules defined in terms

of classes: can be relevant for any # instances

◦ Rules’ actions can cause other rules to fire automatically

3. In-memory ◦ Limited only by

JVM / process memory


Query

CEP BEHAVIOR: QUERY-ORIENTED

New

Order New

Order New

Order

(Lots of

Events)

Results Event Complex

Query


QUERY CEP AGENT FEATURES 1. Common query language

◦ Usually SQL-based – widely used language

◦ May be in-memory, in-file or both

◦ Can include query optimizers

2. Continuous ◦ Extensions usually support time windows for the query to operate over


Usually in-memory for simple apps like trading

Cache / DB memory stores used where failover / DR required, or scalable distributed processing

Other options too (e.g. transactional shared memory systems)

CEP MEMORY ARCHITECTURES


Pattern

Detection

CEP TOOLS EVOLVING – THE “EVENT SERVER”

Event & State store

events Event

Transport &

Distribution

Decisions Reactions


TOPICS



Examples



EVENT-DRIVEN MARKETING USING CEP

#Campaign and Offers pa 500 Campaigns + 4 M Offers

#Target Customers >4M

Transaction Volume >240 Msgs/Sec

Revenue Growth Campaign Performance

1. Campaign 1: 235% Over Campaign Revenue Target

2. Campaign 2: 160% Over Campaign Revenue Target

3. Campaign 3 270% Over on Additional Spending Target

4. Campaign 4 250% Over Campaign Revenue Target

Time to Market per Marketing

Campaign

Concept to Execution:

Before: 3 – 6 Months

After: 7 – 8 Days


Shipment container track and trace:

◦ Monitoring shipment milestones and invoking appropriate processes

54 million active milestones for shipments in-situ at any one time

Implemented 3 times to compare technology approaches

SHIPPING TRACK AND TRACE USING CEP

Approach Cost Cost vs Java

Java programming 64 1x

BPM for events 400 6x

Event-driven rules 28 overall 0.5x

Event-driven rules 12 best case 0.2x


MANUFACTURING QUALITY CONTROL USING CEP

100% testing?

Test results data vs Error lifecycle

Manual, expensive tests

Example: after a restart

production station is

75% reliable for first 4 uses

100% reliable for the next 11

days

80% reliable thereafter

Manufacturing tasks per day 24,000

Number of process steps per unit Several 100s

Solution results 10-15% avg. reduction in manual test costs

“Significant reduction” in overall costs


TOPICS



Examples



FORRESTER NOTE ON BR PLATFORMS, JULY 2011


GARTNER PREDICTIONS, OCT 2011


CEP MARKET PLAYERS

Different

technologies

for different

CEP subtypes


Documents

CEP Overview v1 2 for public use