Getting Started with OpenSplice and Esper

Getting Started with

OpenSplice and Esper

Angelo Corsaro, Ph.D.Chief Technology Officer OMG DDS Sig [email protected]

Thomas BernhardtChief Technology Officer / [email protected]

mailto:[email protected]




!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Agenda

! Welcome & Intro

! DDS Overview

! CEP Overview

! DDS + CEP in Action

! Q&A

DDS Overview...from a Stream Processing Perspective

Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig [email protected]

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Stream Processing ! Stream Processing is an architectural style for building

systems that operate over continuous (theoretically infinite) streams of data

! Stream Processing is often reified under one of its many declinations, such as:! Reactive Systems! Signal Processing Systems! Functional Stream Programming! Data Flow Systems

[1/3]

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Stream Processing! The Stream Processing Architecture very

naturally models systems reacting to streams of data and events produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.

! Stream Processing Systems usually operate in real-time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.

[2/3]

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Stream Processing! Stream Processing Systems are typically

modeled as collection of modules communicating via typed data channels called usually streams

! Modules usually play one of the following roles:! Sources: Injecting data into the System! Filters/Actors: Performing some

computation over sources! Sinks: Consuming the data produced by

the system

[3/3]

Filter

Filter

Filter

source

Sink

Stream

More on Streams

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Defining Streams! In abstract terms, a stream is an infinite sequence of

data samples of a given type T

! Streams can be further classified in continuous and discrete streams. Sometimes referred as Behaviors/Signals and Events

! In this presentation we’ll refer to Continuous Streams as Data Streams and to Discrete Streams as Event Streams

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Data StreamsData Streams! The value of a Data Stream

is always defined, i.e. continuous.

! Good examples of a Data Stream are the value assumed by a real-world entity, such as temperature, pressure, a price, etc.

time

Temp

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Event StreamsEvent Streams! The value of the stream is

defined at precise point in time, i.e. it is discrete

! Good examples of Event Streams are events in the real world, such a violation of a regulatory compliance, the temperature higher than a given value, etc.

OverheatAlarm

time

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

What is DDS?! DDS is a high-performance, real-time, highly-

available, fully-distributed, messaging technology that allows you to define data/event streams and make them dynamically discoverable

! DDS is equipped with a rich set of QoS providing control on the key temporal and availability properties of data

[A Stream Perspective]

Defining Streams in DDS

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

DDS Topics! A Topic defines a stream class/

category! A Topic has associated a user

defined type and QoS! The Topic name, type and QoS

defines the key functional and non-functional invariants

! Topics can be discovered or locally defined DURABILITY,

DEADLINE,PRIORITY,…

[1/2]“org.opensplice.demo.TTempSensor”

struct TempSensor { long Id; float temp; float hum;}#pragma keylist TempSensor id

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

DDS Topics

! DDS Topic types can have associated keys

! Each unique key-value identifies a unique sub-stream of values -- called Topic Instance

[2/2]“org.opensplice.demo.TTempSensor”

DURABILITY,DEADLINE,PRIORITY,…

struct TempSensor { long id; float temp; float hum;}#pragma keylist TempSensor id

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

“Seeing” Streams

Topic

InstancesInstances

struct TempSensor { @key long id; float temp; float hum;};

id =701

id =809

id =977

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

QoS Policies

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Anatomy of a DDS Application

Domain

Reader/WritersUser Defined for Types

Session

val dp = DomainParticipant(0)

// Create a Publisher / SubscriberPublisher p = dp.create_publisher();Subscriber s = dp.create_subscriber();// Create a TopicTopic<TempSensor> t = dp.create_topic<TempSensor>(“com.myco.TSTopic”)

DomainParticipant

Publisher

DataWrter

Topic Subscriber

DataReader

// Create a DataWriter/DataWriterDataWriter<TempSensor> dw = pub.create_datawriter(t);DataReader<TempSensor> dr = sub.create_datareader(t);

Gives access to a DDS Domain

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;


Domain

Reader/WritersUser Defined for Types

Session


// Create a Publisher / Subscriberval pub = Publisher(dp)val sub = Subscriber(dp)// Create a Topicval topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”)

DomainParticipant

Publisher

DataWrter

Topic Subscriber

DataReader

// Create a DataWriter/DataWriterDataWriter<TempSensor> dw = pub.create_datawriter(t);DataReader<TempSensor> dr = sub.create_datareader(t);

Pub/Sub Abstractions

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;


Domain

Reader/Writers for User Defined for Types

Session

DomainParticipant

Publisher

DataWrter

Topic Subscriber

DataReader

// Create a DataWriter/DataWriterval writer = DataWriter[TempSensor](pub, topic)val reader = DataReader[TempSensor](sub, topic)

// Write dataval t = new TempSensor ts(101, 25, 40)writer write ts;

Reader/Writer for application defined Topic Types

// Create a Publisher / Subscriberval pub = Publisher(dp)val sub = Subscriber(dp)// Create a Topicval topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”)


!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Data & Event Streams! DDS does not provide different types for Data/Event Streams.

The difference between the two can be made through the DataReader API by using properly using read/take operations

! DataReader::read! Reads the value of the stream w/o removing it from the stream. As a result

multiple read can see the last known value of the stream

! DataReader::take! Takes the value available on the stream (if any yet) and removes it from the

stream

Stream Processing

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Content Filtered Topics! Content Filtered Topics

provide a way of defining a filters over an incoming stream associated with a given topic

! Filters are expressed as the “WHERE” clause of an SQL statement

! Filters can operate on any attribute of the type associated with the topic

Example:

// Create a Topic (on default domain)val topic = Topic[TempSensor](“TTempSensor”)val ftopic = ContentFilteredTopic[TempSensor](“CFTempSensor”, topic, filter, params)

// - filter is a WHERE-like clause, such as: // “temp > 20 AND hum > 50”// “temp > %0”// “temp > hum”// “temp BETWEEN (%0 AND %1)//// - params is the list of parameters to pass to the // filter expression – if any

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Filter Expression Syntax

! DDS Filters are condition over a topic type attributes

! Temporal properties or causality cannot be captured via DDS filter expression

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

History

! DDS provides a way of controlling data windows through the History QoS

The window keeps the last n data samples

Data older than “n samples ago” get’s out the window

pastfuture

now

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

[Putting it All Together]

TempSensor Moving Averageobject MovingAverageFilter { def main(args: Array[String]) { if (args.length < 2) { println("USAGE:\n\tMovingAverageFilter <window> <filter-expression>") }

val topic = Topic[TempSensor]("TTempSensor") val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))

val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt) val reader = DataReader[TempSensor](ftopic, rqos)

reader.reactions += { case e: DataAvailable[_] => { var average: Float = 0 val window = e[TempSensor].reader.history window foreach (average += _.temp) average = average / window.length println("+--------------------------------------------------------") println("Moving Average: " + average) } } }}

Product Organization

!"#$%&'

()*+,--.*/%&01234(*5**6

77*8&'()0*8303%93:

Product OrganizationCommercial Edition! No Cost Runtime Licenses! Your choice of licensing

! LGPL or Commercial! Subscription or Perpetual

! Complete DDS Implementation! Comprehensive Developer and

Deployment Support Options with a range of Service Level Agreements

Commercial Add-Ons! Individually licensable technologies! Rich ecosystem covering tools,

integration, testing, etc.

CommercialEdition

Commercial Add-ons

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Key Points So Far

! DDS key abstractions for building stream processing architectures

! DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters

! What else can you use for “Stream Processing” in combination with DDS? ! Let’s have Tom introduce us to the world of CEP and Esper!

��

��

��

�

�� !

�� !� ��

"�!��#�$��

%!��#��&!��!��'��&(

)

��!� ��

��

��*!+!�!��#��

&��,��!� ��-� !-#��!��+!�!��#��

.��/�&#��

0��#��!�� 12��#�!�� +�� 2��,!��1��

3��3��

��455�� !��

��

��#��2�!��6��' ��#��!27$82�1$9(

��!##�� -�� /�&

��!##��!��

:��!�#!-�#��2��!�!��!-�#��

0��#��#��#�2��!#�;!��

http://esper.codehaus.org/

1

��4��#�$��

��!��!�� !!

<�#��,�� !��!��/��#��=��#!�'��() ��

��!��!��#!�'��(��

0��!��!��

��/��#�<"��!�� !�8>,��

��##�,� -�?!��@<"�� !�� !�� >��A

��8�� !� B��8�#!��!�-�8��#��A

��#�$��

8

��!�� 0CD%

�0CD%

%�� !!'!#�(

:!� #�E��

%F&'��!-��!�� (

��E��5��

��

��

��

%��#��

:!� #� !!

G ��&4��&!��!��

%�-��-�53��

��6�!��!#��

H IJ��E��JK

��

��

��

��

��

�

��<!#��

��#��!��!#��4��

�� #�$��2�� #!��2!� ��-��L

��

��!#��#!��

��!#��#!�� !��!��5��!��

.��!#��-�� !� �$�#��#�$��

��#��!#��4��!��!� ��#��!+��5

0C5C��',��!��#��-!�=��(

�� !#��;��%F&��!��!� +!�!��!�� #!�!��

��#��'��-��!�� (

��!��-�� =+��0C!� C��!��

-��#��

&�,�� M!�#��#��5:��!�#!-�#��,��!��#�

��-�� !#!-#��#!��!��#� ��-N��!��

9

��&%�#��#!��

%�#��#!��

%!��#�

[insert into insert_into_def]select select_listfrom stream_def [as name] [, stream_def [as name]] [,...][where search_conditions][group by grouping_expression_list][having grouping_search_conditions][output output_specification][order by order_by_expression_list]

select acctId, sum(amount) from Withdrawal.win:time(1 minute)group by acctIdhaving sum(amount) > 1000order by acctId asc

7

�!��$��O��P.��!��

�!��$��!��!��!� �!��

��!��

��4C!��-��# ��-#��=�

M�#��$��

.-��-!��

��#��-��$��!#��2��!�.%��,��=

��

.��!��

&��!#.��!��4��

.��!��!��##��#�4� ��

/�!� �4��

��#��!� ��!��$��-!�� $��!#

��QM�##�,�R�!��!��4�!��S��!$�

��!#��!��4��

T

&��!#.��!��

.��!��$!��#��

M�##�,� -�

�!��$��

&��!#��

��##��,��

select * from pattern [w=Withdrawal -> Deposit(acctId = w.acctId)]

w=Withdrawal -> (pin=PinChange(acctId = w.acctId) and pwd=PasswordChange(acctId = w.acctId) )

every ( (w=Withdrawal -> Deposit(acctId = w.acctId)) or (d=Deposit -> Withdrawal(acctId = d.acctId)) )

�

&��!#.��!��

.��!��$!��#��

��!��!��!#��

.-�� !��!��!#

/�!� �� -�$��!��!��!#

��!��

��!��

every login=LoginEvent -> (timer:interval(20 sec) and not LogoutEvent(userId=login.userId))

every [2] login=LoginEvent

every [2..] login=LoginEvent until logout=LogoutEvent

every login=LoginEvent -> logout=LogOutEvent where timer:within(2 sec)

��

�� .��

��

H.��!��!��2��#��#�� =2��#��!#��!��

H�#��!� ��-!�=��!�!-�#��!� ��!��!�

H0��43�� !��Q%!��#��#F�%��!��

H%��#��!��!#��E��Q�$��-#�2��

�� !��"��#$��%

H��,��$��&!��!��

H��E��2!��!��2N��2�!��!#��!� ��2N��!# !!2��

�#�,��#2��

H�!�� #��!� ��#��

H:��2#�,#!��

H%!� !#��-� � 2��2+!�!�!� !� ��

�� # � �� & � � ��

H��4��!�!��2��!#��!��!#�;!��!� ! ��E��

H��4D�#��#!�� !!��!��Q��!� ��-��!�!��

H��"�'�4+0C��#�!��#��!� �� !-�#��

��

��

��#��!��2#��!#�!��2

��!��!��2�#��!� ��-!�=��

�� !#��

��%��!�

��6

! !��

:��!# !!

!��#!��

��F��6�!��!#��!��&!��!��

��

��

%!��

�.+.

.��

! !��

�.+.

%�-��-��

��!��

:�� #��

��!#�� !!��!��

:��!# !!

3!��

"�� ,�

��

��

��"�'�

'%�� (

��"�'�

'�#�� (

��

(�� )*��+��

��,�#��*��-".��/

0 �� 1�� )2��

�)

F6�

F��U

��!��

��V��


��

��!�=��

��455,,,��

��V��

http://www.espertech.com/

DDS + CEP in Action

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

OpenSplice + Esper

! Esper Provides an EsperIO framework for plugging-in new stream transports

! Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework

! Let’s have a look…

Demo Application

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

iShapes Application! To explore play with OpenSplice and

Esper, we’ll use the simd-cxx ishapes application

! Three Topics! Circle, Square, Triangle

! One Type:

struct ShapeType { string color; long x; long y; long shapesize;};#pragma keylist Shapetype color

Spotted shapes represent subscriptions

Pierced shapes represent publications

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Esper Setup

Step 1: Register Topic Typesval config = new Configurationval ddsConf = new ConfigurationEventTypeLegacy

ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)

config.addEventType("ShapeType", classOf[org.opensplice.demo.ShapeType].getName, ddsConf)

val cep: EPServiceProvider = EPServiceProviderManager.getDefaultProvider(config)

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Esper Setup

Step 2: Register a Listener for receiving Esper Events

val listener = new UpdateListener { def update(ne: Array[EventBean], oe: Array[EventBean]) { ne foreach(e => {! // Handle the event }) } }

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

Esper Setup

Step 3: Hook-up DDS to Esper

reader.reactions += { case e: DataAvailable[ShapeType] => { (e.reader read) foreach(runtime sendEvent _)

}}

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

iShapes FrameRate

! Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color

! In Esper this can be achieved with the following expression:

insert into ShapesxSec select color, count(*) as cnt from ShapeType.win:time_batch(1 second) group by color

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

iShapes Center of Mass

! Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second

! The Esper expression for this would be:

select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg(y),int), shapesize) as NewShape from ShapeType.win:time(10 sec)

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

" Fastest growing JVM Language" Open Source" www.scala-lang.org

References

" #1 OMG DDS Implementation" Open Source" www.opensplice.org

" Scala API for OpenSplice DDS" Open Source" code.google.com/p/escalier

" #1 Java-Based CEP Engine" Open Source" www.espertech.com

http://www.scala-lang.org

http://www.scala-lang.org

http://www.opensplice.org


http://code.google.com/p/escalier

http://code.google.com/p/escalier

http://www.espertech.com

http://www.espertech.com

!"#$%&'

()*+,-,.*/%&01234(*5**6

77*8&'()0*8303%93:;

:: Connect with Us ::

! @prismtech

! @acorsaro

! youtube.com/opensplicetube ! slideshare.net/angelo.corsaro

! [email protected]! [email protected]

! opensplice.com ! forums.opensplice.org

! opensplice.org ! [email protected]

http://www.slideshare.net/angelo.corsaro

http://www.slideshare.net/angelo.corsaro





http://www.opensplice.com




http://forums.opensplice.org/

http://forums.opensplice.org/







