Complex Event Processing with Esper

Real-time Event Stream & Complex Event Processing

tedwon

1. Church bells ringing...

2. Appearance of a man in a tuxedo with a woman

in a flowing white gown...

3. Flower flying through the air.

Concept of CEP

Concept of CEP● From these events CEP may infer a wedding.

● CEP as a technique helps discover complex events as a pattern by analyzing

and correlating other events

● http://en.wikipedia.org/wiki/Complex_event_processing#Conceptual_description

● Algorithmic Stock-Trading

● Real-time Stock Tick monitoring

● Detection of credit-card fraud

● Real-time ETL processing

● Business activity monitoring

● Real-time Mobile Targeting Ad & Recommendation

CEP Use Cases

CEP is● based EDA(Event-driven architecture)

● not Request & Response style system

● Detection & Reaction style system

● not Pre-save and Post-process paradigm

● Pre-process and Post-save paradigm

● Loosely-coupled system

● Asynchronous style processing paradigm

● Intelligence system

Event Driven Architecture● Software architecture pattern promoting the production, detection,

consumption of, and reaction to events.

eye,ear...sensory organs think,decision hand,foot...reaction

EDA Event Process 1. SEP - Simple Event Processing

a. JMS(Java Message Service)

b. ESB(Enterprise Service Bus)

2. ESP - Event Stream Processing

3. CEP - Complex Event Processing

ESP vs. CEP● ESP

○ Monitor streams of event data, analyse those events, and act upon opportunities

○ Filtering, Aggregation, Join

○ Average of Google stock over the last (moving) 30min

● CEP○ Detecting patterns among events

○ If this Google stock increased more than 5% two times followed by Apple stock

decreased more than 10% then…

CEP Open Source Projects● JBoss Drools Fusion

○ Rule Script DSL○ Java

● EsperTech Esper○ SQL-like Script DSL

○ Java

EsperTech Esper

Open Source Esper● Provide ESP/CEP Engine

● Provide SQL-like EPL(Event Processing Language)

● Well-documented reference

● Lightweight & Embeddable

Esper Architecture

Esper Architecture

Esper Architecture● Real-time ETL application

ETL EPL(Extract, Transform, Load)

Hadoop FileSystem API ImplThrift Server Impl

Esper with Runtime● Esper + Java EE Platform

● Java Standalone Instance

● Esper + OSGi Server

Esper EPL Features● Event filtering

● Sliding data windows

● Aggregation

● Pattern matching

● Joins and Outer Joins

● Subquery

● Reference historical data

Esper EPL Example

Order Top10 by item over last 30 min

OrderEvent.win:time(30 min)fromgroup by itemId

select itemId, count(*) as cnt

limit 10

Esper EPL Example

Continuous query

OrderEvent.win:time(30 min)fromgroup by itemId

select itemId, count(*) as cnt

limit 10

Esper EPL Features

MySQL native query

EPL Join

06 EPL과 Adapter 개발

where users visit in Gangnam district over last one hour

“Place Name!!”

from LR.std:unique(assetId).win:time(1 hour) as lr,sql:db[select zone_name from ZoneName

where zone=${lr.zone}]

select assetId, zone_name

where zone in (‘gangnam code’)

Esper Pattern EPL Example

select Part.zone from pattern [ every Part=CountZone(cnt in (1, 2)) -> ( not CountZone(zone=Part.zone, cnt in (0, 3)) and timer:interval(10 sec) )]

Quick Start Esper

Esper Quick Start Project

Esper Quick Start Project

Thank you

References1. https://en.wikipedia.org/wiki/Event-driven_architecture 2. http://en.wikipedia.org/wiki/Complex_event_processing3. http://www.espertech.com/esper/quickstart.php 4. https://github.com/espertechinc/esper 5. http://www.hawkular.org/blog/2017/01/13/events-aggregation-extension.html 6. https://github.com/tedwon/cep-esper-quick-start7. https://www.slideshare.net/matthewmccullough/complex-event-processing-with-esper