Embed Size (px)
Citation preview
Esper• Open Source CEP and ESP engine • Available for Java as Esper, for .NET as NEsper • Developed by “Codehaus” • http://esper.codehaus.org/ (write “esper complex
event processing” on Google)
“Esper enables rapid development of applications that process large volumes of incoming messages or events. Esper filters
and analyzes events in various ways, and responds to conditions of interest in real-time”
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
EsperHow does it work
Works as real time engine that triggers actions when event conditions occur among event streams
An Event Processing Language allows registering rules (sql-like) in the engine,
using Java objects to represent eventsMid
dle
war
e L
abo
rato
ryM
IDL
AB
M I D L A B M i d d l e w a r e L a b o r a t o r y
DB#
Queries
Data
Alert! query satisfied
Reversed DB concept
DB#DB#
Esper ArchitectureM
idd
lew
are
Lab
ora
tory
MID
LA
B
Outline
• Events representation
• Esper input
• Esper logic - Processing (queries and event processing language)
• Esper Output
• PerformanceMid
dle
war
e L
abo
rato
ryM
IDL
AB
Events representation
• Each event is a Plain Old Java Object (POJO)
• The instance varables of the object are the properties of the event. The object must have the getters.
• The class specifies the event type (stream).
How the events are represented inside the engine?
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Example:
Mid
dle
war
e L
abo
rato
ryM
IDL
AB Through the POJOs, Esper can
take the properties of the events by name.
Events representation
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Other representations:
• Java Map: key-values pairs, can contain objects, further maps, arrays... • XML: Document Object Model (DOM) • XML: Streaming API for XML (StAX) !
•Adapters (csv, spring jms template, db output, http...)
Multiple inner streams can have different representations - this is transparent to the processing
engine
Events representation
Esper Input
• Esper sockets
- optimized java sockets (very high throughput)
How the events are given to the engine?
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
• Send event “inside” the engine
- Through the reference of the engine, whithin the java application.
- useful if you integrate esper in your application
Esper logic
Almost all CEP systems have a language (or something equivalent)
in order to specify the correlation rules
No DataBase to interrogate, the data “flow through” the queries (if SQL extention)
(that become “long running queries”)Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Esper EPL language Esper comes with their “SQL-Like” language, called
Event Processing Language (EPL)
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Very similar to SQL: select, where, having, group by and so on, but with some important improvements
(later).
In EPL the queries are called “statements” Statements are given to engine in form of string
parameters
EPL vs SQL M
idd
lew
are
Lab
ora
tory
MID
LA
B
Each event belonging to an event stream is like a tuple (a row of the table). The event stream is like the table.
The properties of an event are the attributes of the tuple.
Name Surname Age
Mario Rossi 24
Gianni Bianchi 56
Roberto Verdi 23
People
Gianni Bianchi 56 Mario Rossi 24 Roberto Verdi 23
Time
Stream name: People
NO ORDERING, it is a set! ORDERING, it is a stream!
EPL vs SQL M
idd
lew
are
Lab
ora
tory
MID
LA
B
Name Surname Age
Mario Rossi 24
Gianni Bianchi 56
Roberto Verdi 23
People
Gianni Bianchi 56 Mario Rossi 24 Roberto Verdi 23
Time
Stream name: People
Select Name, Surname from People Where (Age<24)
If you don’t have to capture the concept of time the EPL and SQL queries are equal!
The output will be provided in a different way.
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
Generally the Stream Processing languages are more powerful then SQL ‘cause they can capture the concept of time. EPL can also modify magic numbers at runtime,
can execute user defined functions and can capture relations among events.
•Windows •Pattern •Variables •User Defined Functions
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
A
B A A C
D
A
Windows
Time
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
batch-space: “wait for n events, calculate among n events” fixed in size - variable in time - one calculation each n events, among n events.
batch-time: “wait for n seconds, calculate among the events arrived in the n seconds” fixed in time - variable in size - one calculation per n seconds
A B A A C D A
A B A A C D A
Windows
select ... from MyEvent.win:length_batch(3)
select ... from MyEvent.win:time_batch(1 sec)
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
Windows
sliding-space: “wait for the firsts n events and after calculate among the last n events as soon a new event arrives” moving, fixed size window - one calculation per event among n events
A B A A C D A
sliding-time: “when a new event arrives, calculate among the events arrived in the last n seconds” moving window fixed time, one calculation per event.
A B A A C D A
select ... from MyEvent.win:time(2 sec)
select ... from MyEvent.win:length(3)
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
Pattern
A B A A C D A
In a given window, pattern captures relations among (at least 2) events:
•e.g. A => B - an A event after a B event •e.g. every A =>B - every A event after a B event
select * from pattern [(every A -> B).win:length(10)]
The window is needed to avoid excessive memory consumption
“I want to be alerted when an A event is followed by a B event in the last 10 events”
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
select * from pattern [every (A or B).win:length(10)]
select * from pattern [(A until B).win:length(10)]
select * from pattern [(a.prop1 = true -> a.prop2 = true). win:time(10 sec)]
Several cases available, e.g. :
Pattern
“I want to be alerted when an A or a B event arrives in the last 10 events”
“I want to be alerted each time an event is between an A event and a B event in the last 10 events”
“I want to be alerted each time an event with a given property satisfied is followed by an event with another property satisfied in the last 10 seconds”
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
A B A A C D A
Pattern can also be used to correlate different streams
Select * from pattern [every ( a=stream1(idS = 49) -> b=stream2(idR = 49) )].win:time(30 sec)
A BA C D A
stream1
stream2
Pattern
“I want to be alerted each time an event of the stream1 having id 49 is followed by an event of the stream2 having id 49”
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
The EPL queries can be very “long running” queries. The variables allow to
change numerical values at runtime.
select symbol, sum(price) from TickEvent group by symbol having sum(price) > var_threshold
Variables
EPL improvements M
idd
lew
are
Lab
ora
tory
MID
LA
B
User Defined Function
Within an EPL statements can be executed a user defined (static) Java method and can be used the
returned value.
select 3 * com.mycompany.MyClass.myFunction(price, volume) as myValue from StockTick.win:time(30 sec)
Esper output
• Esper triggers a listener or a subscriber bound to the statement;
• Esper inserts into another stream the result of the statement;
• both of them.
What happens when a query is satisfied?
three possible solutions:
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Esper ListenersWhen you register a new statement (query) to an esper engine you can specify a related
listener or subscriber
A Simple java object that is instantiated the first time that a query (registered to him) is satisfied and
that contains an “update” method invoked each time the registered queries are satisfied.
The attributes of the event are passed via parameters (subscriber) or via an “eventBean” object (listener)
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Esper “insert into”When a statement is satisfied, it returns events or
properties that can be used by other statements creating a logical network of streams.
The “insert into” clause acts like a dispatcher separating meaningful events among others.
Insert into replies select * from eventStream
where type = ‘reply’
This statement creates a new events stream containing only replies. This Stream can be used
by the other statements in the from clause or in the pattern clause
Esper “insert into”
Rq Rp Rq Rp Ack Rq Rp
eventStream
replies
Select * from replies where cond
Insert into replies select * from eventStream
where type = ‘reply’
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Rp Rp Rp
Esper “insert into”
Join of streams is supported (also by sql) - multiple inner stream are also supported
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
Esper “insert into” example
Insert into replies select * from eventStream
where type = ‘reply’
Insert into requests select * from eventStream
where type = ‘request’
select (replies.timstamp - request.timestamp) as XXX from replies, requests
where requests.requestID = replies.replyID and requests.dest = replies.source
Esper performanceM
idd
lew
are
Lab
ora
tory
MID
LA
B
Esper performanceM
idd
lew
are
Lab
ora
tory
MID
LA
B
2 Ghz dual core laptop
ConclusionM
idd
lew
are
Lab
ora
tory
MID
LA
B
✓ Pro &
❖Cons
✓ Very easy to use ✓Open Source, GPL license (download and try it!) ✓Very well supported (API and documentation) ✓Impressive performance
!
❖Centralized System (but can be distributed)
ConclusionM
idd
lew
are
Lab
ora
tory
MID
LA
B
What we are doing with it?
Online Failure Prediction in Air and Naval traffic control systems
Collaborative Security in financial critical infrastructure
thanks
Mid
dle
war
e L
abo
rato
ryM
IDL
AB
