Esper - CEP Engine

JTeamInnovating J2EE Development

EsperT h e P o w e r t o P r e d i c t

Wednesday, April 1, 2009


Agenda

• Predictive Business

• Complex Event Processing

• Esper

• Demo

• Questions JTeamInnovating J2EE Development



Predictive Business


"In the future, every company must be able to recognize what is coming early enough to take evasive action or steer towards an emerging opportunity."[Frederick W. Smith-CEO FedEx]



Business Demands


High throughput

Low latency

High AvailabilityComplexity

Insight



Conventional Models


• RDBMS

• Adds delays into data storage

• Uses Polling Model

• Wasteful Use of Resources



Conventional Models


• Messaging Systems

• Process Unitary Events

• No Correlation support



Agenda



• Esper

• Demo




Complex Event Processing




Concepts - Event


An event is a piece of data that represents that something happened in the real world, or in software system.

e.g. Stock Tick, Weather, UserClick



Concepts - Complex Event


An event that could only happen when lots of other events happen.

e.g. Housing Price Drop



Concepts - Event Stream


A linearly ordered (often by time) sequence of events forms Event Stream.

e.g. Click Stream



Concepts - Event Cloud


A partially ordered set of events form Event Cloud

e.g.

Set of all stock trades for GOOG within a 5 minute time window is an Event Stream. While all Stocks sold in a business day is an Event Cloud.



Concepts - Event Cloud




ESP and CEP


Event Stream Processing is focused more on high-speed querying of data in streams of events and applying mathematical algorithms to the event data.

Complex Event Processing includes event data analysis, but places emphasis on patterns of events, and abstracting and simplifying information in the patterns.



CEP Engine


An event processor is an application that performs operations on event objects, including creating, reading, transforming, aggregating, correlating or removing them.



CEP Engine




CEP Engine Characteristics


1. Keep the data moving

2. Query on streams

3. Handle Stream Imperfections

4. Integrate Stored and Streaming Data

5. Guarantee Data Safety and Availability



CEP Engine Characteristics


6. Partition and scale applications automatically

7. Process and Respond instantaneously

8. Interoperability



CEP in Application Architecture




Commercial CEP Products


• Weblogic Event Server

• Coral8

• TIBCO

• IBM

• StreamBase

• Progress



CEP/EDA and SOA [1/3]


EDA is SOA 2.0

EDA is Proactive SOAEDA and SOA Complementary











Real World Applications


Capital Markets

BAM - Business Activity Monitoring

Supply Chain Management

Flight Operations Monitoring

Fraud Detection



Agenda



• Esper

• Demo




Esper


• Dual Licensing model

• Open Source GPL

• Esper Tech

• Q2/2007 BEA Partner



Good Documentation


• Nice reference document

• Patterns catalogue available online

• Lots of examples



Lightweight


• ANTLR

• CGLIB

• LOG4J



Ohloh Analysis Summary




Pluggability


• Works with any Java Process

• OSGI Enabled



Architecture


ESPER Engine

EPServiceProvider

Event Objects

Listeners

Configuration

EPL Statements JDB

C Adapter

Subscribers



API Overview


• EPServiceProvider

• Acts as Engine

• EPStatement

• Queries written in EQL

• UpdateListener



Event


• POJOs

• Key value pairs (java.util.Map)

• XML Documents



Event




Event Properties


Type Syntax Example

Simple name sensor

Indexed name[index] sensor[0]

Mapped name(key) sensor(‘light’)

Nested name.nestedName

sensor.value



UpdateListener


Receives updated data as soon as it is processed for the statement.



Subscriber


• Must provide a method by name update to receive insert stream events row-by-row

• The number and types of parameters declared by the update method must match the number and types of columns as specified in the select clause, in the same order as in the select clause

•Wednesday, April 1, 2009


Subscriber


select orderId, price, count(*) from OrderEvent

public class MySubscriber { ...

public void update(String orderId, double price, long count) {...}

}



Esper Engine Integration


• Obtain an Esper engine instance

• Create a statement (using the Esper query language)

• Register the statement with the engine

• Create a listener (by implementing a Java interface that will be triggered when the statement evaluates to true) and attach it to the statement



Esper Engine Integration


//Register events Configuration configuration = new Configuration(); configuration.addEventType("TempEvent", BodyTemperatureEvent.class.getName());

// Get Esper CEP Engine

EPServiceProvider cepEngine = EPServiceProviderManager.getProvider("esper-sports-poc", configuration);

// Create EPStatement

PlayerBMIStatement playerBMIStatement = new PlayerBMIStatement(cepEngine.getEPAdministrator());

playerBMIStatement.addListener(new PlayerBMIListener());

// Dispatch Event

cepEngine.getEPRuntime().sendEvent(eventInstance);



EPL


Similar to SQL

Name BPRoss 110

Rachel 120Joe 100

Stream

Event



Sliding Window Support




Streams


Input stream denotes new events arriving, the term remove stream denotes events leaving a data window.



Windows


• win:length(4)

• win:time(4 sec)

• win:time_batch(4 sec)

• win:length_batch(4)



Batch Time Window




Named Windows


• Analogous to database views

• Created using SELECT queries

• Define window (view) first and insert datacreate window OrdersTimeWindow.win:time(30 sec) as select symbol, volume, price from OrderEvent



Filters


select * from Withdrawal(amount>=200).win:length(5)



Aggregations


• sum

• avg

• count

• min/max

• stddev



Output Rate Limiting


Engine dispatches to Listener when output condition occurs.

e.g.select irstream symbol, volume, price from MarketData.win:time(5.5 sec) output every 1 seconds



Event Patterns


Pattern Expression = Pattern atoms + Pattern Operators

every (spike=ServiceMeasurement(latency>20000) or error=ServiceMeasurement(success=false))

select s from pattern [every s=SubmitOrderEvent -> (timer:interval(30 sec) and not OrderAcceptedEvent(theOrder.id=s.theOrder.id))]



RDBMS Integration




RDBMS Integration


select name, healthy_temp from TempEvent.win:time(60 sec) temp, sql:mydb3 [' select name,healthy_temp from Player where name = ${temp.playerName} and ${temp.temperature} > healthy_temp ']



Non Event Based Data Integration


public class InstrumentRepository {

public static Exchange[] getExchangesForSymbol(String symbol) {

...

}

select ord, eligibleExchanges from TheOrder as ord,

method:com.ociweb.jnb.esper.InstrumentRepository.getExchangesForSymbol(ord.symbol) as eligibleExchanges

}



CEP Design Patterns (1/9)


Filtering:e.g. Click-stream analysis: a filter may be used to capture the trades that originate from a certain set of IP addresses.





In Memory Caching:e.g. Click-stream analysis: a typical application may hold the recent clicks and searches performed by the users, coupled with the relevant historical and reference information.





Aggregation over Windows:e.g. Click-stream analysis: it is often useful to compute the number of visitors who click on a particular link within a specified time interval.





Database Lookups:e.g. Trading: a trading application may look up historical price for a stock, or certain information about an order, or certain rules and regulations stored in a database.





Database Writes:e.g. Click-stream analysis: storing the raw click-stream history, together with derived data, in the database





Correlation (Joins):e.g. Trading: correlating information from multiple exchanges to find arbitrage opportunities.





Event Pattern Matching:e.g. Fraud detection: fraud patterns are often described as a sequence of events, in one or more streams.





Hierarchical Events:e.g. Order Processing: Purchase Order can have multiple items which needs to be processed differently.





Dynamic Queries:e.g. Trading environments: every trader can enter their subscriptions like X=MSFT, Y=.1%, Z=10 am.



EsperIO


Provides adapters support:

• CSVAdapter

• JMSAdapter

• OpenTick Adapter



Demo




Performance Tips


// Better performance

select * from RFIDEvent

// Less good performance

select assetId, zone, xlocation, ylocation from RFIDEvent



Conclusion


Do I really need it?

Do I have the right infrastructure?



Conclusion


• Learning Curve of EPL

• No Standardization of EPL

• EPL is still evolving



Questions?




CEP in Dutch Market


• KLM - TIBCO

• Boekhandels Groep Nederlands (BGN)



Resources


Esper http://esper.codehaus.org/

CEP

http://complexevents.com/

CEP http://www.cepbusinesscast.com/CEPOpen Source SOA


http://esper.codehaus.org

http://esper.codehaus.org

http://complexevents.com

http://complexevents.com

http://www.cepbusinesscast.com

http://www.cepbusinesscast.com

http://www.manning.com/davis/

http://www.manning.com/davis/

Technology

Esper - CEP Engine