1 Event Processing: An Academic Perspective Hans-Arno Jacobsen Bell University Laboratory Chair Middleware Systems Research Group University of Toronto

1

Event Processing:An Academic Perspective

Hans-Arno Jacobsen

Bell University Laboratory Chair

Middleware Systems Research Group

University of Toronto

MIDDLEWARE SYSTEMSRESEARCH GROUP

http://www.padres.msrg.utoronto.ca

EPTS Symposium, Orlando, 2007 2

MIDDLEWARE SYSTEMSRESEARCH GROUPQuerying the Future


MIDDLEWARE SYSTEMSRESEARCH GROUPAmazon to Chapters to

You ....Monday, October 10th in Cyberspace

Your book “...”is available

at .... $10 off

Thursday, November 15th,in Toronto


MIDDLEWARE SYSTEMSRESEARCH GROUPLocation Constraint

Matching

d

P1

P2

P3P4

P5

A

P4P3P2P

1 P5

dA


MIDDLEWARE SYSTEMSRESEARCH GROUPBPM & SLA Monitoring

N

Y

Far?Get

destinationValidaterequest

Find flight

Find train

cost < $0.02

service time < 3s

Only trusted partners



The Common Denominator These applications are driven by asynchronous state

transitions Information becomes available Change in state in the environment Something happens

These state transitions are events Events are disseminated and filtered against queries

events queries


MIDDLEWARE SYSTEMSRESEARCH GROUPAgenda

What abstractions to use? Our approach Overview of paradigm The PADRES project



What Abstractions are Best Suited to Support this Modus Operandi? Databases

Great for managing historic data But what about future data

Data streams Great for managing structured streams of tuples But what about un-structured, multi-typed, sporadic events

from many sources Rule-based expert systems

Great for inference and reasoning But what about managing large numbers of fined-grained

filters in distributed envrionments

Take this cum gran salis


MIDDLEWARE SYSTEMSRESEARCH GROUPOur Approach:

Publish/SubscribeMore specifically Content-based publish/subscribe

For fine-grained filtering Combined with content-based message

routing For selective data dissemination

Extended with persistence et al. For historic data access



Content-based Publish/Subscribe

Publisher Publisher

Subscriber Subscriber

Subscriptions

Publications

NotificationNotification

IBM=84

MSFT=27 INTC=19 JNJ=58ORCL=12

HON=24

AMGN=58

Stock marketsNYSE

NASDAQTSX

Subscriptions:IBM > 85ORCL < 10JNJ > 60

Broker(s)



data tuples subscriptions

query publication

Query and subscription are very similar.Data tuples and publication are very similar.

However, the two problem statements are inverse.

That’s Like Data Base Querying !!

sets of tuples

Abo

ut p

ast

Abo

ut f

utur

e

sets of tuples


MIDDLEWARE SYSTEMSRESEARCH GROUP The Content-based

Model Language and data model

Conjunctive Boolean functions over predicates Predicates are attribute-operator-value triples

[class,=,trigger]

Subscriptions are conjunctions of predicates [class,=,trigger],[appl,=,payroll],[gid,=,g001]

Publications are sets of attribute-value pairs [class,trigger],[appl,printer],[gid,g007]

Matching semantic A subscription matches if all its predicates are matched

P/S

events

notifications



Publish/Subscribe Matching Problem

Given a set of subscriptions, S, and a publication, e, return all s in S matched by e.

e is referred to as event or publication Splitting hairs

Event is a state transition of interest in the environment

Publication is the information about e submitted to the publish/subscribe system

Simple problem statement, widely applicable, and lots of open questions


Content-based Message Routing

Publisher

Subscriber

1. Advertise

2. Subscribe

3. Publish

Event-BasedEvent-Based

DecoupledDecoupled

FlexibleFlexible

ResponsiveResponsive

Content RoutingContent Routing

DeclarativeDeclarative

A: [class, =, stock], [name, =, HP], [price, >, 50]S: [class, =, stock], [name, =, *], [price, >, 50]P: [class, stock], [name, HP], [price, 55]



ToPSS - The Toronto Publish/Subscribe System Family [2000 – present]

Matching algorithms Language expressiveness

vs. efficient matching

Routing protocols Network architectures &

scalability

Higher level abstractions Workflow execution Monitoring

S-ToPSS(semantic)

X-ToPSS(XML matching)

A-ToPSS(approximate)

persistent-ToPSS(subject spaces)

L-ToPSS(location-based)

ToPSS(matching)

M-ToPSS(mobile)

Ad hoc-ToPSS(ad hoc networking)

Federated-ToPSS(federation of ToPSS brokers)

Rb-ToPSS(rule-based)

P2P-ToPSS(peer-to-peer)

LB-ToPSS(load balancing)

FT-ToPSS(fault tolerance)

Historic-ToPSS(historic data)

CS-ToPSS(composite subs)

BPEL-ToPSS(BPEL execution)

JS-ToPSS(job scheduling)



PADRES Publish/Subscribe System Project [2003-present]


•Peng Alex David aRno Eli Serge,

•PAdres is Distributed REsource Scheduling

•Publish/subscribe Applied to Distributed Resource Scheduling



The PADRES ESB Stack

Server Farm

Computers

ComputersDatabase

Laptops

Computers

Workstation

Laptops

DatabaseServer

ServerSwitch

Server

Switch

Computing, Storage, and Networking Resources

Content-based Routing

Content-based RouterClients (publisher/subscriber)

Scientific Workflow Execution

Deploy Control UpdateVisualize

Monitor ...

6

43

7start halt

Workflow Management and Activity Monitoring

Redirectresume

addremove

Event ManagementFramework

ScientificApplication

Events

Scientific WorkflowEvents

Scientific WorkflowExecution Events

Network andSystem Events

Business Process Execution Layer



Subscription Language & Publication Data Model

Subscriptions are conjuncts of predicates [class,=,job_info], [workflow,=, foo_example], [instanceID,=,

$x], [job,=,A], [status,=,succ]

isPresent operator, $X (variable binding), composite subscriptions, string operators etc.

Publications are sets of attribute value pairs [class, trigger], [workflow, foo_example], [instanceID, 54]



Padres Broker Architecture

QueueHandler

QueueHandlerQueueHandler

BrokerCore

Matching Engine

Controller

Lifecycle Manager Overlay Manager

Publication / Subscription Routing Table

JESS

InputQueue

…

QueueHandler

OutputQueues

Broker_Control Message

QueueHandler

…

RMITransportHandler

JMS

BrokerRMI

ClientRMI

DB


MIDDLEWARE SYSTEMSRESEARCH GROUPPADRES Features

A

B C

D

E

F

CompositeEvents

HistoricAccess

Management

Robustness

LoadBalancing

Security


MIDDLEWARE SYSTEMSRESEARCH GROUPLimitations of Acyclic

Overlays

Broker

Publisher

Subscriber

P

Sensitive to Congestion Imbalanced workloads Broker failures Overlay changes


MIDDLEWARE SYSTEMSRESEARCH GROUPRobustness Through

Cycles

PPPP

RobustRobust

Self-healingSelf-healing Adaptive routingAdaptive routing

Flexible overlayFlexible overlay

Publisher

Subscriber

Congested Link



End to End Delay(preliminary results)

•30 brokers on 15 machines (on LAN)•20 publishers (2400msg/min)•30 subscribers (2000 total)• toplogy with average connection degree of 4

Fixed routing does not exploit cycles

Dynamic routing does exploit cycles



Sub

Sub Sub Sub

Sub

Load Balancing Framework

Local Load BalancingGlobal Load Balancing

Pub

offloading broker

load-accepting broker


MIDDLEWARE SYSTEMSRESEARCH GROUPLoad Balancing Components

DETECTOR

PIEMEDIATOR

PIE messages

OFFLOAD ALGORITHMS

Input Match Output

LOAD ESTIMATION

PRESS

Mediation and migration protocols

Subscribers to offload

Target broker

Load Balancer

Establish and teardown of load balancing sessions between two brokers/clusters

Coordinate transparent subscriber migration

Calculates the set of subscribers to offload for balancing the performance metric in question based on load information about the subscriptions and load-accepting broker

Estimates load of subscriptions

Triggers load balancing if overload or uneven load distribution is detected by examining 3 performance metrics



Output Utilization Ratio (no LB)

Overload!

Edge broker dies



End to End Delivery Delay

Edge broker dies

Increasing delay due to output overload

at BOTH edge broker and cluster-

head broker


MIDDLEWARE SYSTEMSRESEARCH GROUPOutput Utilization Ratio (LB)

Overload!

Load balancing converges


MIDDLEWARE SYSTEMSRESEARCH GROUPComposite Subscription

AND

OR

S1 S2

OR

S3 S4

AND

S5

CS={{S1 OR S2} AND {S3 OR S4} AND S5}

A composite event is the constellation ofevents being detected by the compositesubscription.

S are atomic subscriptions. I.e., they are satisfied by a single, multi-attribute event.

Composite subscriptions (CS) are used for event correlation, in network filtering, and thedetection of complex events,

Applications: BMP, BAM (see Thursday’s talk)


MIDDLEWARE SYSTEMSRESEARCH GROUPTopology-based CS

Routing

Distributed Overlay Broker

NetworkB3

&&

&&

S1 S2

hS3

S

P2

Pk

CS={{S1 AND S2} AND hS3}

B2

B1

B6

B5

P1

CS’={S1 AND S2}P Publishers

S Subscribers

S2

S1

B4CS’

hS3

CS

hS3DB


MIDDLEWARE SYSTEMSRESEARCH GROUPComposite Event

Detection


NetworkB3

&&

&&

S1 S2

hS3

S

P2

Pk


B2

B1

B6

B5

P1

CS’={S1 AND S2}P Publishers

S Subscribers

CS’

hS3S2

S1

P1

P2

B4

CS

CS

P123

P12

P3

hS3

DB


MIDDLEWARE SYSTEMSRESEARCH GROUPOptimizations

Dynamic composite subscription routing

Runtime adjustments of CS joint points


MIDDLEWARE SYSTEMSRESEARCH GROUPCost Model I

Definition 1. Data set size of atomic subscription S

Ri is the message rate at the source

Apj Spjp/sP1

Pn

Ai Aj

Ak

SAi

Ak

S

S = p1 and … and pn


MIDDLEWARE SYSTEMSRESEARCH GROUPCost Model II

Definition 2. Data set size of composite subscription CS



CS Routing Optimization

Objectives Minimize network traffic Minimize notification delay

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}

(a) (b)


MIDDLEWARE SYSTEMSRESEARCH GROUPEstimate CE Detection

Costs Traffic-based model Traffic Cost (S1) =

Delay Cost (S1) =

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}



Optimized Subscription Routing Determine joint points based on cost model


NetworkB3

S

P2

Pk

B2

B1

B6

B5

P1

S2

S1

B7

B4

CS’

hS3

CS


hS3DB



Optimized Subscription Routing Dynamically maintain joint points


NetworkB3

S

P2

Pk

B2

B1

B6

B5

P1

S2

S1

B7

B4CS’

hS3

CS


hS3DB

39

MIDDLEWARE SYSTEMSRESEARCH GROUPSummary

Publish/Subscribe solves a problem inverse to database query processing

Publish/Subscribe is well suited as abstraction for many event processing based applications

Publish/Subscribe is a paradigm for data-centric networking & selective information dissemination

PADRES realizes a distributed publish/subscribe system offering many novel features

PADRES can serve as ESB and ISB (Internet Service Bus); more on this Thursday at 8AM

There are plenty of open research questions


MIDDLEWARE SYSTEMSRESEARCH GROUPAsking EPTS and

Industry for Data sets

Subscription data sets for various domains Workload characteristics For active research we are looking for

SLA examples, business process examples

Interesting and relevant problems Challenges of tomorrow Specific problems


MIDDLEWARE SYSTEMSRESEARCH GROUPAcknowledgements

Graduate students, visitors, and PDFs currentlyworking on PADRES. Alex Cheung Guoli Li Jian Li Vinod Muthusamy Alex Wun Songlin Hu Reza Sherafat

Partner from CA directly involved in project Serge Mankovskii

And many alumni

visit us @


42

“A forum dedicated to the dissemination of original research, the discussion of practical insights, and the reporting on relevant experience relating to event-based computing previously scattered across several communities.”

In cooperation with (approval pending):

July 2nd – 4th, 2008Roma, Italy

http://debs08.dis.uniroma1.it

Submission deadline: March 15th, 2008

Location: Dipartimento di Informatica e Sistemistica “A. Ruberti”Sapienza Università di Roma

www.debs.org


MIDDLEWARE SYSTEMSRESEARCH GROUPQuestions?

A D R E SP





Routing in Cyclic Networks


MIDDLEWARE SYSTEMSRESEARCH GROUPLimitations of Acyclic

Overlays

Broker

Publisher

Subscriber

P

Sensitive to Congestion Imbalanced workloads Broker failures Overlay changes


MIDDLEWARE SYSTEMSRESEARCH GROUPGeneral Overlay

PPPP

RobustRobust

Self-healingSelf-healing Adaptive routingAdaptive routing

Flexible overlayFlexible overlay

Publisher

Subscriber

Congested Link



1

Routing in General Overlays I

2

Advertisement Tree 1

Duplicate Messages

SSS

SS

Advertisement Tree 2


MIDDLEWARE SYSTEMSRESEARCH GROUPRouting in General

Overlays II Scenario 1

Subscriptions are routing in loops

Brokers receive duplicated subscriptions

2

3 4

5

1

6

Adv 1

Adv 2

S X

S

S



Overlays III Scenario 2

Subscriptions are multicasted to several destinations

Brokers receive duplicated subscriptions

Scenario 1 is exacerbated by duplicated subscriptions

2

3 4

5

1

6

Adv 1

Adv 2

S

Y

S

S

S

S


MIDDLEWARE SYSTEMSRESEARCH GROUPRedundant Messages

Problem

0

2000

4000

6000

8000

10000

12000

14000

16000

50 100 500

Number of Nodes

Num

ber o

f Dro

pped

Mes

sage

s

Average Connect Degree = 5

Average Connect Degree = 10



Overlays Extends standard content-based routing

protocol Maintain the same interface to pub/sub

clients Requires changes to

Advertisement routing Subscription routing

Atomic subscriptions Composite subscriptions

Publication routing


MIDDLEWARE SYSTEMSRESEARCH GROUPAdvertisement Routing

Each advertisement forms a spanning advertisement tree

Duplicated advertisements are discarded by brokers

Each advertisement is assigned a unique tree identifier (TID) e.g. a [class, eq, stock]……[TID, eq,

adv_msg_id] SRT

A set of [advertisement, last hop]


MIDDLEWARE SYSTEMSRESEARCH GROUPSubscription Routing

Each subscription has a TID predicate with a variable. e.g. s [class,eq,stock]……[TID,eq,$X]

The variable is bound to the TID of matching advertisement upon subscribing

PRT A set of [subscription, {TID, last hop of subscription }]



Scenario 1

2

3 4

5

1

6

Adv 1

Adv 2

S X

S

S: [class,eq,stock][name,eq,*][price,>,50] [TID,eq,$Z]

At Broker 1:

Adv1:

[class,eq,stock][name,eq,IBM][price,>,60][TID,eq,Adv1]

Adv2:

[class,eq,stock][name,eq,HP][price,>,50][TID,eq,Adv2]

S matching Adv1:

[class,eq,stock][name,eq,*][price,>,50] [TID,eq,Adv1]

S matching Adv2:

[class,eq,stock][name,eq,*][price,>,50] [TID,eq,Adv2]



Scenario 2

SRT(B4) Last hop

Adv1 2

Adv2 6 2

3 4

5

1

6

Adv 1

Adv 2

S

Y

S

S

PRT(B4)

TID Last hop

S Adv1 Y

Adv2 Y


MIDDLEWARE SYSTEMSRESEARCH GROUPPublication Routing

Each publication is assigned the TID of its matching advertisement

Publications are routed: Fixed TID routing Dynamic publication routing

Lemma 1: No broker receives duplicated publication messages

No subscriber receives duplicated publications according to Lemma 1


MIDDLEWARE SYSTEMSRESEARCH GROUPFixed TID Routing

2

3 4

5

1

6

Adv 1

Adv 2

XP

Sub

P


MIDDLEWARE SYSTEMSRESEARCH GROUPDynamic Publication

Routing Publication’s TID is

changeable Best path algorithms Lemma 2: Changing

a publication p’s TID while in transit will not change the set of subscribers, N, notified of p.

2

3 4

5

1

6

Adv 1

Adv 2

X

Sub

P



Advantages of TID-based Approach Retains the publish/subscribe client interface Speeds up subscription and publication

propagation Generates duplicated messages only at

advertisement level Builds multiple subscription routing paths for

publications Routes publications dynamically Delivers publications around failed brokers



Preliminary Experimental Results Setup

Five dual core servers CPU: 2.4GHz ~ 3.4GHz Memory: 1GB ~2GB

Cyclic broker overlays Number of nodes: 23 brokers Average connection degree: D = 5; D=3

Delay = D_queue + D_transmission + D_ propagation Metrics

Notification delay CPU and memory usage Bandwidth consumption


MIDDLEWARE SYSTEMSRESEARCH GROUPNotification Delay

0

20

40

60

80

100

120

140

160

(a) Notification Number (D=5)

Not

ifica

tion

Del

ay (

ms)

Fix Publication Routing

Dynamic Publication Routing



0

150

300

450

600

020

040

060

080

010

0012

0014

0016

0018

0020

00

(b) Notification Number (D=3)

Not

ifica

tion

Del

ay (m

s)

Fixed Publication Routing




0

10

20

30

40

50

60

70

80

10 50 100 200 250 350 500 1000

(a) Publication Rate (msg/s) (D=5)

Avg

. Not

ifica

tion

Del

ay (m

s)





0

80

160

240

320

400

480

10 50 100 200 250 350 500 1000

(b) Publication Rate (msg/s) (D=3)

Avg

. Not

ifica

tion

Del

ay (m

s)


Dynamic PublicationRouting



Overhead in terms of CPU Usage

0

10

20

30

40

50

0 2 4 6 8 10 12 14 16 18

Time (min)

CP

U u

sage

(%)



• Average over all brokers



Overhead in terms of Memory Use

01

234

56

0 2 4 6 8 10 12 14 16 18

Time (min)

Mem

ory

usag

e (%

)

Fixed Publication RoutingDynamic Publication Routing

• Average over all brokers



Benefits of Content-based Publish/Subscribe

Simplifies IT development and maintenance by decoupling enterprise components

Supports sophisticated interactions among components using expressive subscription languages – going beyond the limits of topics

Allows fine-grained queries and event management

Achieves scalability with in-network filtering and processing


MIDDLEWARE SYSTEMSRESEARCH GROUPAdaptive CS Routing

Setup 1 subscriber with composite subscriptions 4 publishers with unbalanced publication rates Focus on behaviours of four particular brokers:

The subscriber connects to Broker A Broker B is the joint point broker in Topology-based routing Broker C is the new joint point broker in adaptive CS routing The slower publishers connect to Broker D

Compare Simple routing Topology-based routing QoS-based adaptive routing



0

1000

2000

3000

4000

0 100 200 300 400 500 600

Time (s)

Byt

e pe

r S

econ

d

Broker ABroker BBroker CBroker D

Traffic of CS Routing

Simple Routing

A

B

C

D



0

500

1000

1500

2000

2500

3000

3500

4000

0 100 200 300 400 500 600

Time (s)

Byt

e pe

r S

econ

d



Topology-based Routing

AB

C

D



0500

1000150020002500300035004000

0 100 200 300 400 500 600

Time (s)

Byt

e pe

r S

econ

d



QoS-based Adaptive Routing

A

B

C

D


MIDDLEWARE SYSTEMSRESEARCH GROUPNotification Delay of CS

0

100

200

300

400

500

600

700

0 50 100 150 200 250

Number of Notifications

Not

ifica

tion

Rou

ting

Tim

e (s

)

Simple Routing

Topology-Based Routing

Adaptive CS Routing



CSs may be split according to potential publication traffic, bandwidth, latency, etc.

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}

(a) (b)


MIDDLEWARE SYSTEMSRESEARCH GROUPEstimate Data Set Size

Definition 1. Data set size of atomic subscription

Definition 2. Data set size of composite subscription


MIDDLEWARE SYSTEMSRESEARCH GROUPEstimate CE Detection

Costs Traffic-based model Traffic Cost (S1) =

Delay Cost (S1) =

QoS-based model

Delay Cost (S1) =

2

1 3Adv 1

Adv 2

CS={{S1 AND S2}



Algorithm Evaluate the cost model at each broker for

composite subscriptions Choose the broker with minimum detection

cost as the joint point Joint points are maintained dynamically

according to the traffic and network condition



Example: Topology-based CS Routing

2

3 4

5

1

6Adv 1

Adv 2

7

8

9

CS

S2S3

S1

CS’

Adv 3

CS’={S1 AND S2}AND

AND

S1 S2

S3

CS={{S1 AND S2} ANDS3}


MIDDLEWARE SYSTEMSRESEARCH GROUPExample: Adaptive CS

Routing

2

3 4

5

1

6Adv 1

Adv 2

7

8

9

CS={{S1 AND S2} ANDS3}CS

S2

S3

S1

CS’

Adv 3


MIDDLEWARE SYSTEMSRESEARCH GROUPEffect of Parallelism

0

20

40

60

80

100

10 50 100 300 1000

Request Rate (/min)

Thr

ough

put

(/m

in)

-20

-10

0

10

20

30

Impr

ovem

ent

(%)

CentralizedDistributedImprovement



3-body Constraint Matching

d1<d

satisfied

d2>d

unsatisfied

d3<d

satisfied

d

enclosing circle

|green set| < d satisfied|red set| < d unsatisfied


MIDDLEWARE SYSTEMSRESEARCH GROUPComposite Subscription

Routing


NetworkB4B3

AND

AND

S1 S2

S3

S

P2

P3


B2

B1

B6

B5

P1CS

CS’={S1 AND S2}

P Publishers

S Subscribers

CS’S3S2

S1


MIDDLEWARE SYSTEMSRESEARCH GROUPComposite Event

Detection


NetworkB4B3

AND

AND

S1 S2

S3

S

P2

P3


B2

B1

B6

B5

P1

CS

CS’={S1 AND S2}

P Publishers

S Subscribers

CS’

S3S2

S1 CS

P1

P2

P12

P3

P123


MIDDLEWARE SYSTEMSRESEARCH GROUPOptimized Routing

Algorithm Evaluate the cost model at each broker for

composite subscriptions Choose the broker with minimum detection

cost as the joint point Joint points are maintained dynamically

according to the traffic and network condition


MIDDLEWARE SYSTEMSRESEARCH GROUPPublish/Subscribe in

Industry Standards

CORBA Event Service CORBA Notification

Service OMG Data Dissemination

Service Java Messaging Service WS Eventing WS Notification WS Brokered

Notifications INFO-D (Grid Forum) AMQP

Emerging technologies (not complete) RSS aggregators

PubSub.com, FeedTree Real-time data

dissemination TIBCO, RTI Inc.,

Mantara Software Application integration

Softwired Hardware-based brokers

Sarvega (Intel), Solace Systems, DataPower (IBM)



P

S

= publisher / sensor

= subscriber / doctor

S

S

B

B

B

B

B

P

P

medication = NoNameDrugX

temperature = 40°C

illness = diabetes

B

B

heart rate = 150 bpm

blood pressure = 60 mmHg

P

P[medication = NoNameDrugX] & [blood pressure > 100] & [heart rate > 130]

[heart rate > 140 bpm] & [temperature > 39] & [illness != cold]

S[temperature > 42] & [medication = Advil]

Ptemperature = 38°C

Historic Query with Event Correlation

Monitoring patients who still have high blood pressure after taking NoNam

eDrugX

Monitoring patients having a fever but not

having a cold

Monitoring patients who still have fever after

taking Advil

Medical records database

[medication = NoNameDrugX]

[blood pressure > 100]

[heart rate > 130]

Documents

1 Event Processing: An Academic Perspective Hans-Arno Jacobsen Bell University Laboratory Chair Middleware Systems Research Group University of Toronto