WSO2 Complex Event Processor 3.0.0
An overview of upcoming features
By S. Suhothayan
Associate Technical Lead, Team Lead CEP.
About WSO2
• Providing the only complete open source componentized cloud platform – Dedicated to removing all the stumbling blocks to enterprise agility – Enabling you to focus on business logic and business value
• Recognized by leading analyst firms as visionaries and leaders – Gartner cites WSO2 as visionaries in all 3 categories of application
infrastructure – Forrester places WSO2 in top 2 for API Management
• Global corporation with offices in USA, UK & Sri Lanka
– 200+ employees and growing
• Business model of selling comprehensive support & maintenance for our products
150+ globally positioned support customers
Outline
! Scenarios of Event Processing ! WSO2 CEP Server & SOA integrates ! The Siddhi Runtime CEP Engine. ! High availability, Persistence and Scalability of
WSO2 CEP ! How CEP can be combined with Business
Activity Monitoring (BAM). ! Demo
CEP Is & Is NOT!
! Is NOT! o Simple filters
• Simple Event Processing • E.g. Is this a gold or platinum customer?
o Joining multiple event streams • Event Stream Processing
! Is ! o Processing multiple event streams o Identify meaningful patterns among streams o Using temporal windows
• E.g. Notify if there is a 10% increase in overall trading activity AND the average price of commodities has fallen 2% in the last 4 hours
WSO2 CEP Server
! Enterprise grade server for CEP runtimes ! Supports several transports (network access) ! Supports several data formats ! Support for multiple CEP runtimes ! Governance ! Monitoring ! Tools (WSO2 Dev Studio)
WSO2 CEP Architecture
Siddhi CEP Runtime ! Apache License, a java library, Tuple based event model ! Supports distributed processing ! Supports multiple query models
• Based on a SQL-like language • Supports
! Partitions ! Filters ! Windows ! Joins ! Ordering ! Output Rate Limiting ! and others
CEP Event Adaptors
! Is an adaptor for receiving and publishing events ! Has the configurations to connect to external endpoints ! Its many-to-many with CEP engine
CEP Event Adaptors Support for several transports (network access) and data formats ● SOAP/WS-Eventing
XML messages ● REST
JSON messages ● JMS
Map messages XML messages Text messages JSON messages
● SMTP (Email) Text messages JSON messages XML messages
● Thrift - WSO2 data format High Performant Event Capturing & Delivery Framework supports Java/C/C++/C# via Thrift language bindings
WSO2 Event
CEP Event Adaptors ● Cassandra (from CEP 3.0.0)
Map messages ● Fix (from CEP 3.0.0+)
Map messages ● MYSQL (from CEP 3.0.0)
Map messages ● HBase (from CEP 3.0.0+)
Map messages
& Event adaptors are pluggable !
CEP Event Builders
• Event builder converts different input event types to a type compatible with the execution plan.
• Subscribes to an Input Event Adaptor to listen for events and sends a converted WSO2 Event or Basic Event to the Execution Plan
• Receives events in different formats and exposes those input streams as stream definitions
• Has a one to many relationship with execution plans in execution plan.
CEP Execution Plan
● Is an isolated logical execution unit
● Each execution plan has a set of Queries Input & Output stream mappings.
● Its one-to-one with a CEP Backend Runtime Engine
● It deals with Siddhi processing engine.
CEP Event Formatter
Event formatter does the inverse – listens to events coming from event processor and sends converted events to Event adaptors.
There are 5 types of output mapping types are available
● Map ● Text ● WSO2Event ● XML ● JSON
Monitoring (Event Tracer & Event Statistics) ! Provides real-time statistical visual illustrations of
request & response counts per time based on CEP server, execution plan, transport adaptor, event builder and formatter.
Writing CEP Queries (using
Siddhi Runtime)
Siddhi Queries ! Filters and Projection ! Windows
o Events are processed within temporal windows. (e.g. for aggregation and joins)
Time window vs. length window. ! Joins - Join two streams ! Event ordering - Identify event sequences and
patterns ! Event Partitions ! Event Tables
Filters
! Filters the events by conditions, use to detect simple condition
! Conditions o >, <, = , <=, <=, != o contains, instanceof o and, or, not
! Example
from <stream-name> [<conditions>]* select <attributes>
insert into <stream-name>
from cseEventStream[price >= 20 and symbol==’IBM’] select symbol, volume insert into StockQuote
Window from <stream-name> [<conditions>]#window.<window-name>(<parameters>)
select <attributes> Insert into <stream-name>
Types of Windows ● (Time | Length) (Sliding| Batch) windows ● Type of aggregate functions ● sum, avg, max, min
Example from cseEventStream[price >= 20]#window.lengthBatch(50) select symbol, avg(price) as avgPrice group by symbol having avgPrice>50 insert into StockQuote
Join
! Use to join two streams based on a condition. There must be at least one window defined
! Unidirectional – event arriving only to the unidirectional stream triggers join
! Example
from <stream>#<window> [unidirectional] join <stream>#<window> on <condition> within <time>
insert into <stream>
from TickEvent[symbol==’IBM’]#window.length(2000) join NewsEvent#window.time(5 min)
on TickEvent.symbol==NewsEvent.company select * insert into JoinStream *
Pattern
! Use to Check condition A happen before/after condition B.
! Can do iterative checks via “every” keyword. ! Here with “within <time>”, SIddhi emits only events
that are within that time of each other ! Example
from [every] <condition> → [every] <condition> … <condition> within <time> select <attributes> insert into StockQuote
from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and a1.cardNo==a2.cardNo] within 1 day select a1.cardNo as cardNo, a2.price as price, a2.place as place insert into potentialFraud
a1 x1 k5 a2 n7 y1
Sequence
! Regular Expressions supported o * - Zero or more matches (reluctant). o + - One or more matches (reluctant). o ? - Zero or one match (reluctant). o or – either event
! Here we have to refer events returned by * , + using square brackets to access a specific occurrence of that event
from <event-regular-expression> within <time> select <attributes> Insert into <stream>
from a1 = requestOrder[action == "buy"], b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+, b2 = cseEventStream[price <b1.price] select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice insert into purchaseOrder
a1 b1 b1 b2 n7 y1
Event Paritions
Partition types can be one of two types • Variable Partitions - Partitions are created by
the discrete values that are encountered for a variable
define partition StockSymbol by StockStream.symbol
define <partition-id> by <partition-type> (,<partition-type>)*
• Range partitions - Partitions are created according to predefined ranges of variables
define partition stockVolume by range volume < 10 as 'SMALL', range volume > 10 and volume < 100 as 'MEDIUM', range volume > 100 as 'LARGE'
Event Tables define table <table-name> (<attribute-name> <type> {, <attribute-name> <type>}*) ( from <table-type>.<datasource-name>:<database-name>.<table-name>)?
Event tables can be used in the same manner as an event stream, with the difference being that events sent to an event table being persisted to a data source. CEP supports event tables for
• In Memory • Relational
o MySQL o H2
define table cseEventTable(symbol string, price int, volume float) from MYSQL.cepDataSource:cepdb.cepEventTable0
Working with Event Tables
from <stream> (select <attribute-name> (,<attribute-name>)* )? insert into <table-name>
Inserts the selected attributes from the input stream into the event table.
from cseEventCheckStream[symbol==cseEventTable.symbol in cseEventTable] insert into outStream;
For update and delete
from <stream> update <table-name> (on <condition>)?
from <stream> delete <table-name> (on <condition>)?
! We compared Siddhi with Esper, the widely used opensource CEP engine
! For evaluation, we did setup different queries using both systems, push events in to the system, and measure the time till all of them are processed.
! We used Intel(R) Xeon(R) X3440 @2.53GHz , 4 cores 8M cache 8GB RAM running Debian 2.6.32-5-amd64 Kernel
Performance Results
Simple filter without window
Performance sending event within same JVM
from StockTick[prize >6] return symbol, price
State machine query for pattern matching
Performance sending event within same JVM
From f=FraudWarningEvent ->
p=PINChangeEvent(accountNumber=f.accountNumber) return accountNumber;
Performance Sending Events over the network ! Here we publihsed data from two client publisher
nodes to the CEP Sever node and sent the triggered notifications of CEP to a client subscriber node.
! To test the worsecase sinario, 100% of the data published to CEP is recived at the subscriber node after processing (No data is filtered)
! We used Intel® Core™ i7-2630QM CPU @ 2.00GHz, 8 cores, 8GB RAM running Ubnthu 12.04, 3.2.0-32-generic Kernel, for running CEP and used Intel® Core™ i3-2350M CPU @ 2.30GHz, 4 cores, 4GB RAM running Ubnthu 12.04, 3.2.0-32-generic Kernel, for the three client nodes.
HA/ Persistence ! Ability to recover
runtime state in the case of a failure.
! Enables queries to span lifetimes much greater than server uptime.
! Takes periodic snapshots and stores all state information to a scalable persistence store (Apache Cassandra).
! Supports pluggable persistent stores.
Scaling ! Vertically scaling
o Can be distributed as a pipeline ! Horizontally scaling
o Queries like windows, patterns, and Join have shared states, hence hard to distribute!
o Use distributed cache (Hazelcast) to achieve this • shared memory and batch processing
Event Recording
! Ability to record all/some of the events for future processing
! Few options o Publish them to Cassandra cluster using WSO2 data
bridge API or BAM (can process data in Cassandra with Hadoop using WSO2 BAM).
o Write them to distributed cache o Custom thrift based event recorder
Integration with WSO2 BAM
Data Receiving Data Analyzing Data Presentation
Data Publishing
CEP Role within WSO2 Platform
DEMO
Scenario ! Monitoring stock exchange for game changing
moments ! Two input event streams.
o Event stream of Stock Quotes from a stock exchange
o Event stream of word count on various company names from twitter pages
! Check whether the last traded price of the stock has changed significantly(by 2%) within last minute, and people are twitting about that company (> 10) within last minute
Input events ! Input events are JMS Maps
o Stock Exchange Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("symbol", "MSFT"); map1.put("price", 26.36); publisher.publish("AllStockQuotes", map1);
o Twitter Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("company", "MSFT"); map1.put("wordCount", 8); publisher.publish("TwitterFeed", map1);
Queries
Queries from allStockQuotes[win.time(60000)] select symbol,price, avg(price) as averagePrice
group by symbol having ((price > averagePrice*1.02) or (averagePrice*0.98 > price )) insert into fastMovingStockQuotes
from twitterFeed[win.time(60000)] select company as company, sum(wordCount) as words group by company having (words > 10)
insert into highFrequentTweets
from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes join highFrequentTweets[win.time(60000)] as highFrequentTweets on fastMovingStockQuotes.symbol==highFrequentTweets.company
select fastMovingStockQuotes.symbol as company, fastMovingStockQuotes.averagePrice as amount, highFrequentTweets.words as words
insert into predictedStockQuotes
Alert ! As a Email Hi
Within last minute, people being twitting about {company} {words} times, and the last traded price of {company} has changed by 2% and now being trading at ${amount}.
From CEP
! As a SMS
Useful links ! WSO2 CEP 3.0.0
http://ec2-54-224-94-128.compute-1.amazonaws.com/chunk-02/N-25_09_2013/wso2cep-3.0.0.zip
! WSO2 CEP http://wso2.com/products/complex-event-processor/
! CEP Performance Info http://srinathsview.blogspot.com/2013/08/cep-performance-processing-100k-to.html
! Distributed Processing Sample With Siddhi CEP and ActiveMQ JMS Broker. http://suhothayan.blogspot.com/2012/08/distributed-processing-sample-for-wso2.html
! Creating Custom Data Publishers to BAM/CEP
http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep
! WSO2 BAM 2.3.0 http://wso2.com/products/business-activity-monitor/
Engage with WSO2
• Helping you get the most out of your deployments • From project evaluation and inception to development
and going into production, WSO2 is your partner in ensuring 100% project success
Questions?
Thank you.