SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS Blending Complex Event Processing with the RETE Algorithm iCEP@FIS 2008 Kay-Uwe Schmidt, SAP Research

SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS

Blending Complex Event Processing with theRETE AlgorithmiCEP@FIS 2008

Kay-Uwe Schmidt, SAP Research

28.09.2008

Addl. Authors: Roland Stühmer (SAP), Ljiljana Stojanovic (FZI)

© SAP 2008 / Kay-Uwe Schmidt / Page 2

1. The Evolution of Rule-based Web Applications

2. State of the Art in Combining CEP and Rete

3. Enhancing RETE with SnoopIB

4. Evaluation and Conclusions

Agenda


From Server- to Client-sideAdaptive Hypermedia Systems (AHSs)

Server

Web Application Server

BusinessRulesEngine

Presentation Logic

Business Logic

Business Rules

Browser

Plain HTML Rendering

Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

Presentation Logic

Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

ECARule Engine

Presentation Logic

Presentation Rules

Web Page Paradigm Rich Internet Applications Rule-enabled Rich Internet Applications

Traditional adaptation strategies have been intensively studied

Web Page Paradigm: Every Web page in a series of pages is downloaded separately.

Conventional architecture: tracking of user clicks, the user modeling, as well as the adaptation take place on the server.

Limited user tracking possibilities User requests seen by the server Subset of user clicks

Adaptation only on behalf of an explicit user request

On-the-fly adaptation not obtainable



Server


BusinessRulesEngine

Presentation Logic

Business Logic

Business Rules

Browser


Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

Presentation Logic

Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

ECARule Engine

Presentation Logic

Presentation Rules


RIAs / AJAX New user tracking

possibilities New user interface

adaptation possibilities

Look and feel of desktop applications

Highly responsive user interfaces

Adaptation rules still on the server

On-the-fly adaptation not obtainable



Server


BusinessRulesEngine

Presentation Logic

Business Logic

Business Rules

Browser


Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

Presentation Logic

Server


BusinessRulesEngine

Business Logic

Business Rules

Browser

Rich Client Engine

ECARule Engine

Presentation Logic

Presentation Rules


Client-side rule processing Reduction of client-server-communication to a

minimum Enhanced user tracking capabilities In-time response to user actions

On-the-fly adaptation is obtainable


Run-time Architecture

Internal BrowserRepresentation

JavaScript/JSONPresentation

Event Handler

DOM

UIAdaptation

UserTracking

Manipulate

User Interactions

RuleEngine

User Model

Adaptation Rules

Update

Call

Call

Trigger

Read

Read






Agenda


State of the Art: Production Rule Systems

Pattern Recognition Algorithms RETE algorithm [1, 2]

Others TREAT [3], LEAPS [4] or GATOR [5]

CA-Rules OPS5 [6]

Scope Long-living Business Objects

1. Charles L. Forgy. On the efficient implementation of production systems. PhD thesis, Carnegie Mellon University, Pittsburgh, PA, USA, 1979.

2. Charles L. Forgy. Rete: a fast algorithm for the many pattern/many object pattern match problem. Artificial Intelligence, 19:17-37, 1982.

3. Daniel P. Miranker. Treat: A better match algorithm for ai production system matching. In AAAI, pages 42-47, 1987.

4. Don Batory. The leaps algorithms. Technical report, University of Texas at Austin, Austin, TX, USA, 1994.

5. E. N. Hanson and M. S. Hasan. Gator: An optimized discrimination network for active database rule condition testing, technical report tr-93-036. Technical report, CIS Department, University of Florida, 1993.

6. Charles L. Forgy. The ops5 user's manual. technical report cmu-cs-81-135, 1981.


Event Condition Action Rules

First mention of the ECA paradigm by U. Dayal, A. P. Buchmann, and D. R. McCarthy in 1988 “Rules are objects too: A knowledge model for an active, object-oriented databasesystem”. In

Lecture notes in computer science on Advances in object-oriented database systems, pages 129-143, New York, NY, USA, 1988. Springer-Verlag New York, Inc.

This paper describes work in progress on the knowledge model (an extended data model that includes constructs for representing rules) of HiPAC, an active, object-oriented DBMS.

“Central to our knowledge model is the concept of event-condition-action (ECA) rules, which generalizes the many different mechanisms introduced previously in the literature to support active DBMS functions. The event part of an ECA rule specifies database operations, temporal events, or signals from

arbitrary processes; the condition part specifies database queries; and the action part specifies a program. When the event occurs (is signaled, the condition is evaluated; if the condition is satisfied, the

action isexecuted.” ON event IF condition DO action


Event Processing

The two kinds of event processing: Complex Event Processing (CEP) Event Stream Processing (ESP)

Dealing with different problems in event processing using different approaches

ESP – extraction of simple events from a stream Events are totally ordered by time Emphasis of ESP on efficiency for high throughput and low latency Algorithmic stock trading

CEP – extraction of complex event patterns from a cloud Only a partial temporal order of events Other partial order of interest for CEP is for instance causality More time and memory needed Business Process Monitoring

CEP is a superset of ESP

CEP and ESP nowadays adopt each others approaches


Complex Event Detection Algorithms

Finite State Automata Ode 1992 Transformation of complex event expressions into deterministic finite automata Convenient model to define the semantics of complex event operators Downside no acceptance of overlapping occurrences of the same complex event

Colored Petri Nets SAMOS 1994 Convenient model to define the semantics of complex event operators Also the detection of overlapping occurrences is possible

Graph-based Approaches



Sentinel based on Snoop, SnoopIB in the mid 90’s

Construction of the graph from the event expressions

Example: (E1,E2);E3

The graph is a directed acyclic graph and generally does not form a tree Nodes may have several parents, when their represented expression is part of more than one

complex events There is no single root node, when there is no overarching, single most complex event.

;

,

E3E1 E2


Overview of the State of the Art inEvent Processing


Finite State Automata

Colored Petri Nets

Pattern Matching in Streams then DB Tables, Sliding Windows

StreamSQL

EventProcessing

Complex EventProcessing

Event StreamProcessing


Adding Complex Event Processing to Rete

Research Question

How can both pattern recognition algorithms be integrated in an efficient manner?

Different approaches Merging of event processing into rule processing Merging of rule processing into event processing Loose coupling of event with rule processing


Conceptual Differences between Facts and Events

7. Opher Etzion. On events and data. Online Article. http://epthinking.blogspot.com/2008/07/on-events-and-data.html, July 2008.

8. Francois Bry and Michael Eckert. Twelve theses on reactive rules for the web. In Torsten Grust, Hagen Höpfner, Arantza Illarramendi, Stefan Jablonski, Marco Mesiti, Sascha Müuller, Paula-Lavinia Patranjan, Kai-Uwe Sattler, Myra Spiliopoulou, and Jef Wijsen, editors, EDBT Workshops, volume 4254 of Lecture Notes in Computer Science, pages 842{854. Springer, 2006.

[7]

[8]


Maloof ’93

Maloof et al. [9] Reasoning about relative time representations Temporal operators before, during and after Interval-based semantics Details of implementation unclear

9. M.A. Maloof and K.J. Kochut. Modifying rete to reason temporally. Tools with Artificial Intelligence, 1993. TAI '93. Proceedings., Fifth International Conference on, pages 472-473, Nov 1993.


Berstel ’02

Berstel [10] Temporal operators before and after Time-point-based semantics (no correct detection of sequences) Garbage-collection mechanism Events are special facts which are asserted through a special API

10. Bruno Berstel. Extending the rete algorithm for event management. In Proc. Ninth International Symposium on Temporal Representation and Reasoning TIME 2002, pages 49-51, Washington, DC, USA, 7{9 July 2002. IEEE Computer Society.


Walzer ’08

Walzer [11, 12] 13 temporal relations from Allen [13] – all relations intervals can have Temporal operators for each of the 13 relations are introduced, e.g. equal, before, after,

during, overlaps, etc. Interval-based semantics Sliding window garbage collector

11. Karen Walzer, Alexander Schill, and Alexander Löser. Temporal constraints for rule-based event processing. In Aparna S. Varde and Jian Pei, editors, PIKM, pages 93-100. ACM, 2007.

12. Karen Walzer, Tino Breddin, and Matthias Groch. Relative temporal constraints in the rete algorithm for complex event detection. In DEBS '08: Proceedings of the second international conference on Distributed event-based systems, pages 147-155, New York, NY, USA, 2008. ACM.

13. James F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM, 26(11):832-843, 1983.


Comparison

Time-point vs. interval-based semantics

Number of temporal operators

Working memory holds all events Garbage collection No real publish / subscribe

New rules can match events from the past Retained by an already existing rule The match of the new rule might depend on older ones Unclear semantics

Events are not facts Immutable Modify and Retract operation seem not applicable

No notion of event selection and consumption


Implications of Holding Events in Working Memory

Garbage collection

No real publish / subscribe New rules can match events from the past Retained by an already existing rule The match of the new rule might depend on older ones Unclear semantics

Events are immutable Modify and Retract operation seem not applicable

No notion of event selection and consumption Contexts: recent, chronicle, continuous, cumulative [18]

18. Sharma Chakravarthy, V. Krishnaprasad, Eman Anwar, and S. K. Kim. Composite events for active databases: Semantics, contexts and detection. In Jorge B. Bocca, Matthias Jarke, and Carlo Zaniolo, editors, 20th International Conference on Very Large Data Bases, September 12-15, 1994, Santiago, Chile proceedings, pages 606-617, Los Altos, CA 94022, USA, 1994. Morgan Kaufmann Publishers.






Agenda


Coupling RETE with CEP

new top nodes

Event Graph

Ret

e

CArule

ECArule

ECArule

The RETE network, depicted at the top, is responsible for rule conditions and finds matching facts fulfilling rule conditions.

The event-detection graph, at bottom left, is responsible for event specifications and finds events matching an event pattern.

Both RETE and the event graph propagate their results to newly introduced top nodes.

These nodes trigger rule actions.

Figure:

(E1,E2);E3

x.y=1 & x.z=2

Action

Example:


Advantages

The event graph can have nodes with more than two input edges This more closely resembles higher-level event operators from Snoop which we are using.

Also garbage-collection comes more naturally Events are stored and discarded in the queues of event nodes where they are still needed.

There is no global repository for unused or partially unused events like the working memory from RETE.

Consumption modes are selectable (on a per-node basis, if necessary) This allows for the controlled selection of events from a stream, if more than one events have

arrived and might be fitting for a match.

For an ECA rule action to fire, an event must be detected, and for its complete interval, the condition must be fulfilled Vice versa this means that no events need to be correlated as long as RETE supplies no

matched tokens.






Agenda


Conclusions

Rete seems not a natural fit for doing event processing Reasoning over persistent and mutable vs. transient and immutable facts Many changes towards temporal capabilities as well as event expiry must be added

Our approach: Blending CEP with RETE Solution to avoid the indicated mismatch of requirements Combination of dedicated event processing nodes with RETE nodes

Pending evaluation


Questions?

Thank you!


Contexts

Documents

SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS Blending Complex Event Processing with the RETE Algorithm iCEP@FIS 2008 Kay-Uwe Schmidt, SAP Research