Dr. Opher Etzion, STSM Lead Architect, Event Processing Technologies IBM Software Group

Software Group – Event Processing Technology and Architecture

| 2006 © 2006 IBM Corporation

Event Processing – Vision and RealityBPM-BAM-CEP ConferenceRegensburg, June 19, 2006 Keynote Address

Dr. Opher Etzion, STSMLead Architect, Event Processing TechnologiesIBM Software Group

Software Group – Event Processing Architecture and Technologies

An Evolving Thinking About Event Processing © 2006 IBM Corporation2

Outline

Towards an event processing Community

Event Processing – market view

Terminology and architecture

Event Processing - segments



Event Processing – Market View



What drives this area ? From the bird eye’s view:

Nothing is really new We process events for many years (e.g. exception handling in OS).

However, recently: Significant amount of events – types, sources, instances Variety of application need to process events Some traditionally stand-alone applications need to be integrated with regular

information systems (simulation, real-time) Functionality requires sophistication – e.g. temporal capabilities, spatio-temporal

capabilities.These all contributed to COTS tools

It is not cost-effective to develop this functionality for a single application It is recognized as middleware level capability, thus customer preference for COTS. Drive for standards is next step

.



Event Processing - Analyst view

Event-Based Application PlatformsDefinition: Application platforms that offer a programming model for event-driven computing. Business application vendors will need complex event processing capabilities as part of their evolving service-oriented architectures (SOAs) to deliver adaptability and flexibility of their platforms.Justification for Hype Cycle Position/Adoption Speed: Some leading platform vendors have proposed, but none has strategically committed to, an event model as primary for business application design. JavaBeans is the early, and nearly failed, attempt at this model. The model has an intrinsic power that makes its gradual adoption likely, but not in the near term.Business Impact Areas: Event-based application platforms see the software environment as a relationship of events, as opposed to a relationship of programs in prevailing models. Business application vendors will design and develop event-driven principles into their traditional SOAs. This evolution will make unique application styles possible.

Benefit Rating: High.Market Penetration: Less than 1 percent of target audience.Maturity: Embryonic.



More advanced domain: Algorithmic Trading

Complex Event Processing for Trade FacilitationDefinition: A component of business activity monitoring, complex event processing (CEP) entails the receipt or extraction of events as they are occurring in real- or near-to-real time, processing (analysis, filtering, aggregation and cleansing) to determine if the event requires action, and producing an output that can be used to trigger an appropriate action.Justification for Hype Cycle Position/Adoption Speed: Current implementations typically involve a limited number of feeds and relatively simple filtering or analysis. CEP is one component of the enterprise nervous system concept, and to straight-through processing and real-time operations.Business Impact Areas: Applications include algorithmic trading, quality of service monitoring, best execution assessment.Benefit Rating: High.Market Penetration: Five percent to 20 percent of target audience.Maturity: Adolescent.Example Vendors: Atsmai, iSphere, Progress Software (acquired Apama), Streambase.Recommended Reading:• “Clarifying the Terms 'Event-Driven' and 'Service-Oriented Architecture'”• “Innovative Vendors in Business Event Management”



Circles of players

Event Processing as part of full-

fledged middleware

General Event Processing

products that can work with multiple

platforms

Event Processing capabilities

embedded inside solution /

applications (sold as products also)

Tibco

Oracle

Progress

StreamBase

AptSoft

RuleCore

Coral8

Aleri Labs

RedRabbit

Actimize

Leanway

G4



IBM complete Roles

Event-Driven Applications

Industry Solutions

Application Development

Tools

IBM Middleware

Development tools provider

Middleware Producer

Solution Provider

Services Provider

MiddlewareBased

EmbeddedBase (ISV) Integration

platformIBM software

General

Practice

IBM SW services



Event Processing – segments



Major Segments

EPBL - Event processing as part of business logic

EPPD - Event processing as part of problem determination

EPID - Event processing as part of information distribution

EPPS - Event processing as part of predictive systems

EPBO - Event processing as part of business observation



EPBL - Event Processing as part of business logic

Types: Application integration through event processing

Event-driven business policies/rules Application examples:

Adaptive workflows (Telco and others)

Trade management (FSS)

Automated shipping and receiving (Retail)

Just-in-time car rental allocation (Travel and Transportation)

Time-critical targeting (military)



EPBL – characteristics

Entire spectrum of mediators: transformation, routing, validation, enrichment.

Pattern detection: varies, requires rich language. Non functional requirements:

All mission critical systems requirements (integrity, transaction support, recoverability, high availability, failover, clustering, security, audit)

May have a need for high throughput

predictability is needed in some applications

Time-out management is important.

Require end-user ability to maintain rules/policies.



EPBO - Event processing as part of business observation

Business activity monitoring KPI – performance management

Exception management in BPM. Sense and respond

Time-constrained Decision processes and analytics may be needed

A loop that contains actuators and monitoring the actuators results. Applications:

RT compliance/ Fraud detection/AML (FSS, Telco)

Manufacturing S&R (industrial)

Adaptive policy setting (operational resilience)

Business health assessment (cross-industry)

Promotion evaluation (Retail)

SLA management of a process (cross industries)



EPBO – charachterisitcs

Languages: Aggregation-land: trends, time series processing, statistical

measures etc…

Data-base connection – retrospective processing

Send and response may need:

o Predictabilityo Embedded analytics/decision modules

Business performance may need:

o Metrics and KPI managemento Derived data (“active data warehousing”)o Causality model.

Require business analysts interfaces.



EPPD – Characteristics

Main functionality: Eliminate redundant events – filtering, remove duplication

Event correlation to determine symptoms

Problem/symptoms relations modelingBoth real-time and post-mortem (log files) May need high throughput Time interval typically short Assumption that events can be lost:

No fault tolerance is required.

Partial pattern satisfaction may be useful. Pattern language is focused on – sequence within short time-

frames, thresholds, filtering condition. Typical users: system administrators, developers, product support

team.



EPPD – Characteristics

Main functionality: Eliminate redundant events – filtering, remove duplication

Event correlation to determine symptoms

Problem/symptoms relations modelingBoth real-time and post-mortem (log files) May need high throughput Time interval typically short Assumption that events can be lost:

No fault tolerance is required.

Partial pattern satisfaction may be useful. Pattern language is focused on – sequence within short time-

frames, thresholds, filtering condition. Typical users: system administrators, developers, product support

team.



EPID - Event processing as part of information distribution

Enable individuals/applications to subscribe to personalized derived events rather than raw events

Customer alert systems (banking)

Content-based geo-spatial subcriptions



EPID – charachterisitcs

Personalized and specialized form of the publish/subscribe paradigm

Typically loosely-coupled from the operational system, Subscription can be by individuals, “communities of

interests” or applications – routing decisions and management of subscriptions is important

Subscription can be to derived events and not only to raw events, which requires pattern detection – trends, content filtering combined with basic relations (conjunction, disjunction, negation, sequence etc..).

Requires end-user interfaces for subscriptions in some cases, development interfaces in cases of applications’ subscriptions.



EPPS - Event processing as part of predictive systems

Impact analysis: prediction of consequences of already observed events towards future events, or entity states.

May be in response to future change proposals (“change management”)

May be combined with “sense and respond” to mitigate or eliminate this impact.

Applications: Change management in dependent cases (automobile, industrial)

Impact of IT problems on business health (cross-industry)

o Includes ITSM Availability, Incident, Problem Management On-line/real-time detection of fraud, money laundering.

On-line risk management.



EPPS – characteristics

Requires predictive tools embedded in event processing Requires dependency model of events and entities Requires mining tools to find causality relations requires rich causality model with possible uncertainty

handling (e.g. probabilistic engine) Rich pattern language – different type of condition

causality.



Event Processing - Platform Independent Architecture and Model



Event Driven Applications

Event Processing

Event-drivenActions

Event Creation



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event

Event Processing Mediator Event

Actor

Event Processing Mediator

Event Consumer

EventCloud

EventConsumer

ArchitectureGlossary

Event:

(1). Something of interest that occurred in reality (state changed, fact becomes true)

(2). The computerized message to report the event

Event Producer:

An entity (e.g. software artifact) that creates events and reports them (e.g. workflow)

Event Sensor: An entity that senses events and reports them



Event Creation

Produced or sensed Push, periodic pull, on-demand pull. challenge: Increase the “event scope”

Events extracted from video streams

Events from RSS feeds

Events extracted from textual information



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Actor


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Event Topic:

A data-type channel for transporting events of a certain type

Event Consumer:

An entity which is reported about event creation

Event Actor:

An entity which produces automatic actions in response to an event



Event-driven Action

Notify/Store – for consumer Trigger, Orchestrate – for actor (BPM event processing)



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Actor


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Context: A set of criteria to partition the space of events according to temporal (e.g. time window), spatial (e.g. space window) and partitioning entity (e.g. platinum customers)

Event Stream: A collection of events that arrive to a single consumer over a single event topic within a single context

Event cloud: A collection of all event streams that a single consumer receives



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Actor


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Event Processing Mediator:

A software artifact that gets an “event cloud” as an input, and produces a collection of events as output.



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Actor


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Pattern Detector: A software artifact that obtains an event cloud as an input, and creates a “complex event” –Example: the stock quote is monotonically decreasing within 2 hours. Return the collection of stock quotes.

Complex event: An event that contains reference to all events that participate in the pattern



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Actor


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Derived Event: An event that is calculated as a function of other events.

Event Processor: A software artifact that obtains a composite event and creates collection of derived events using some function on the input events, such as: Enrichment, Transformation, Aggregation, Split.



Event Processing Mediators

Event ProcessingMediators

EventValidation

EventRouting

EventTransformation

Event Enrichment

Event Pattern Detectors

Event Emitters



Event Producer

Event

EventTopic

Emitter

Event Sensor

EventTopic

Event

processor

Pattern Detector

Complex event

Derived Event


Consumer


Event Consumer

EventCloud

EventConsumer

Architecture

Glossary

Emitter:

A software artifact that makes routing decisions for the derived events to consumers and actors: subscribers, itinerary based or intelligent routing



Several assertions

There are several different approaches to event processing. Rule-based approach Extended SQL-based approach (stream and active database) Script oriented approach Other approaches

Our aim is to have a model-based approach that is in a “platform independent” level, have appropriate tooling, and consistent with standard approaches.

Probably there is no “one fits all” solution in the run-time level, but from the model and tooling level everything should be seamless

We should strive for open architecture with: Separation of concerns:

o each function can be implemented by various run-time artifacts, each of them satisfies a subset of the appropriate functions, and given set of non-functional requirements.

o We should own the “integration platform” for event processing: the biggest market for event processing will be in “application integration through event processing”.

o We should have our own run-time artifacts, but allow easy integration with other run-time artifacts, mainly specialized ones.



EDA as part of SOA

Enterprise Service Bus

Portal Service

WorkflowBusiness Activity

Business-to-Business Interactions

Enterprise Information System Adapter

Script, POJO, Stateless Session Bean

DistinguishedServices

DistinguishedServices

Information MgmtXML Database

Information MgmtXML Database

This is the famous SOA picture

Each service can emit events – all the time, periodically, or by request

The events are being processed Using event processing mediators (e.g. enrich)

The processed events notify or orchestrateThese building

blocks are not tightly coupled to the ESB

Event Processing can also be retrospective



Event Processing Network

EPN consists of a set of nodes (event processing mediations AKA as event processing agents, service, and event processing endpoints)

Following Separation of concerns principle

Each processing node has an “event pattern detector” part and “event emitter” part , which are the same for all mediator types.

This is a design pattern, but can also be run-time artifact, providing distributed set of lightweight run-time artifacts.

Event Pattern Detector

Event Processor


Event Emitter



Retrospective processing of events

We need to do all event processing in retrospect A naïve way is to select all the raw events from the database

and run them through the “memory” event processing.

This may not be a cost-effective way to do it when there is a large quantity of raw events…

Solutions: Built-in database capabilities that can optimize to this type of

processing

Converting this functionality to plain SQL (probably the short term game)

Time



Towards an Event Processing Community

Conclusion



Activities in progress

Establishing international community: The kick-off meeting of the community has been the first event

processing symposium in March 2006

o The steering committee consists of representatives of IBM, Tibco, Oracle, Progress, SAP, Streambase, Gartner and some academic people.

o Follow-up symposium in November 2006 Several community work-groups:

o Terminologyo Reference Architectureo Interoperability

ACM SIG in construction Starting work with OMG on standards More workshops and conferences

Vendors announcements – SOA 2.0

Documents

Dr. Opher Etzion, STSM Lead Architect, Event Processing Technologies IBM Software Group