Laboratory for Advanced Collaboration

L  A  C  

Towards  Stream-­‐based  Reasoning  and  Machine  Learning  for  IoT  Applica<ons

Markus Endler1, Jean-Pierre Briot2, Francisco Silva e Silva3, Vitor P. Almeida1 , Edward H. Haeusler1

1 PUC-Rio, Rio de Janeiro, Brazil

2 LIP6, Paris, France 3 UFMA, São Luiz, Brazil

2

Towards  Real-­‐<me  Intelligence  for  IoT  

General goals: •  Get a better understanding of what is happening in the physical world; •  Correlate apparently unrelated sensed events; •  Predict events that have not been sensed yet; •  Identify consequences of sensed events. •  Send out correct and timely actuation commands or notifications to humans.

www.iotglobalnetwork.com/iotdir/2017/05/08/

4

Challenges of IoT Intelligence

§  Sensors are multi-modal, distributed and heterogeneous §  Their data is noisy and incomplete §  Associated with time and space (Context) §  Availability and reliability are dynamic §  Must be processed fast – several IoT applications require

“soft” real-time analysis §  Privacy and security are important issues

But  data  analysis  alone  is  not  sufficient!  IoT  demands  ac7onable  informa7on  based  on  domain  knowledge.  

IntelliSys 2017

6

Outline  

§  The  ESMOCYP  Approach  §  Example  of  Scenario  §  Implemented  Prototype  §  Machine  Learning  Prospects  §  Conclusion  

IntelliSys 2017

7

Project ESMOCYP(1): Merging three disciplines

& Data Stream Processing

(Complex Event Processing)

Cyber-Physical Systems (IoT)

Semantic Web & Reasoning

IntelliSys 2017 (1) Efficient Semantic MOdels and Fault-tolerant Middleware for CYber-Physical Systems

10

The  ESMOCYP  Approach  

Goal: achieve online reasoning as a combination of CEP and Deduction Logic reasoning Main steps: 1.  Semantic annotation: Identify the type of entities (smart

objects) and the action happening with it 2.  Generate RDF streams (Semantic Event stream) from

annotated sensor data 3.  Use these Semantic Events to generate facts (Fact

stream) using the Knowledge Base (KB), and also input new facts into the KB

4.  In offline mode, process the KB and derive new Event Processing rules for online analysis

ESMOCYP relies on the ContextNet middleware [Endler et al, Middleware 2011], that has cloud and mobile components.

IntelliSys 2017

11

M-EPA: Mobile Event Processing Agent

ContextNet services in the Cloud

§  Mobile Hub [Rios/Endler:2015] is a middleware component for the Internet of Mobile Things (IoMT)

§  M-EPA [Rios/Endler,2016] allows the Mobile Hub to pre-process “local” sensor data streams and send to the cloud just relevant information

§  Performs Complex Event Processing (CEP) using ESPER engine •  Can identify relevant event patterns and generate complex events •  Can receive and deploy new CEP/EPL rules on the fly

12

Semantic Annotation of Simple Sensor Events/Data §  The M-EPA (Mobile Event Processing Agent) identifies the entity/

object and the action performed with the smart object §  The entity is identified by a mapping of UUID (Universal Unique

Identifier) to ontology concepts §  The action is identified by the analysis of sensor data signal (e.g.

value)

(pr,sub)

UUID Mapping

Sensor data events

UUID is cell phoneY or

Action on entity: e.g. kicking

Predicate: moved Subject: cell phone

UUID value

UUID value

UUID value

Mobile EPA

IntelliSys 2017

UUID is temp. by sensorW

13

From Data Event Stream to Semantic Event Stream

(pr,sub) (pr,obj) (pr,obj) (pr,sub)

Context mapper A

(sub,pr,obj) (sub,pr,obj) (sub,pr,obj)

Subject +Action Object +Action

Context-specific RDF triples Semantic Event Stream

Annotated Sensor Event Stream

Annotated Sensor Data Context

mapper B Spatio-

temporal correlations

IntelliSys 2017

Main Idea: An action (predicate) of a subject (sub) on an object (obj) occur at a unique instant of time and in a same place.

14

From Data Event Stream to Semantic Event Stream

(pr,sub) (pr,obj) (pr,obj) (pr,sub)

Annotated Data Event Stream

Context mapper CEP rules link subjects and objects that have same/related predicates (actions), same timestamp and same location. This correlation is done by the M-EPA (close to the sensors)

< , t , location > < , t, context>

RDF +Timestamp + Conetxt Attributes

(pr,sub)

< , t , location > (pr,obj)

Time window

≅ ≅ (sub, pr, obj)

IntelliSys 2017

15

(sub,pr,obj) (sub,pr,obj) (sub,pr,obj)

Semantic Event Rule

A

FACT A FACT C FACT B

(sub,pr,obj) (sub,pr,obj) (sub,pr,obj)

(sub,pr,obj) (sub,pr,obj) (sub,pr,obj) Sub

stream 1

Sub stream 2

Sub stream 3

Knowledge facts

…

Semantic Event Stream (RDF)

Fact Stream

Semantic Event Rule

B Knowledge

Base

From Semantic Event Stream to Deduced Fact Stream

IntelliSys 2017

16

Example of Scenario §  Mr. Silva carries a smartphone running MobileHub and

travels by bus with air conditioning and smart sensors §  MobileHub automatically discovers and connects to

nearby smart sensors (with temperature and accelerometers)

§  Analyzed data (from sensor location, accelerometer...): ambient condition (temperature), movement pattern...

Deduced information: §  Silva is riding on Bus Line #435 §  Silva is in an air-conditioned ambient

IntelliSys 2017

17

Example  of  Scenario  RDF  events  in  the  seman<c  stream:    §  (cell-­‐phoneY,  connectedTo,  sensorX),    §  (sensorX,  reads,  Abs(Accelerometer)  >  10),    §  (sensorX,  reads,  temp=20)    Knowledge  Base  facts:  §  (Silva,  carries,  cell-­‐phoneY)  §  (sensorX,locatedIn,  BusKKZ8674),    §  (BusKKZ8674,  serves,  BusLine  435).  DL  Rules:  §  (cell-­‐phone,  connectedTo,  sensor)  ∧  (sensor,  in,  ambient)  =>  (cell-­‐phone,  

in,  ambient)  §  (person,  has,  cell-­‐phone)  ∧  (cell-­‐phone,  in,  ambient)  =>  (person,  in,  

ambient)  §  (person,  in,  ambient)  ∧(ambient,  isA,  bus)  ∧(bus,  servers,  line)  =>  

(person,  riding,  line)  §  (person,  in,  ambient)  ∧  (sensor,  in,  ambient)  ∧  (sensor,  reads,  data)  =>  

(person,  experiences,  data)  

IntelliSys 2017

18

Prototype  

Level 1 uses Esper: receives the UUID of the Bluetooth sensor/beacon and the temperature values. By matching a sequence of data it will generate RDF triples with the entities’ type information.

Level 2: runs C-SPARQL (a continuous stream reasoning engine) that uses the KB. i.e., new facts can be deduced doing reasoning on their type hierarchy and their relationship to other objects in the application domain

KB: Ontholgy has relations between entities (people, roles, devices, rooms, reachability, and time)

IntelliSys 2017

19

Technologies used in the Prototype

§  ESPER •  ESPER Event Processing Language (EPL) •  EPL: declarative (SQL-like) language with pattern language and sliding

window •  Queries on Java objects and their attributes

§  C-SPARQL •  Continuous/Stream Extension of SPARQL (SPARQL Protocol and RDF

Query Language) •  Queries on stream of RDF triples

§  Mobile Hub •  Android-based middleware for opportunisctically detecting, connecting and

retrieving sensor data from Bluetooth Smart SensorTags

§  M-EPA •  Android-based ESPER CEP engine where EPL rules can be downloaded

and managed remotely (from the cloud)

IntelliSys 2017

20

Machine Learning Prospects

Machine  Learing  is  more  appropriate  for  sta7s7cal  analysis,  and  therefore  should  be  employed  in  the  early  phases  of  the  reasoning,  e.g.  to  detect  which  phenomenon  is  occuring  with  the  subjects/objects.  Some  ideas  are:  

1.  Extract  Features  from  Sensors  Data  §  Autoencoders,  RBMs  

2.  Extract  Paaerns  from  Features  §  k-­‐Means  

3.  Learn  Temporal  Series  of  Paaerns  §  Recurrent  Networks,  Markov  Chains…  

Ex:  SensorSax  [Ganz  et  al.  2016]:  1;  2;  3  4.  Deduc7on  of  abstract  Facts  and  Rules  

§  Induc7ve  Logic  Programming  

IntelliSys 2017

22

Conclusion  

Project  ESMOCYP  is  exploring  real-­‐7me  reasoning  for  IoT  systems,  by  combining  Complex  Event  Processing  and  knowledge-­‐based  reasoning.  §  So  far,  we  are  just  exploring  the  technological  feasibility    §  Our  prototype  implementa7on  has  shown  promising  results  

But…  many  problems  remain  to  be  solved:  §  How  to  handle  facts  that  are  only  transient?  §  What  happens  if  the  deduc7on  of  new  facts  is  made  on  stale  

knowledge  §  Knowledge  base  should  have  a  no7on  of  7me  (for  rela7ons  and  

rules)  §  How  to  deal  with  large  ontologies?  §  It  is  possible  to  “clip/cut  off”  the  relevant  part  of  the  ontology?  §  How  to  merge  the  modified  clipped  part  into  the  whole  

ontology?    

23

Thank you!

Questions? Acknowledgement: CAPES-DAAD PROBRAL Cooperation Program For more information please concact:

endler@inf.puc-rio.br http://www.lac.inf.puc-rio.br

IntelliSys 2017