Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Private Public Partnership Project (PPP)
Large-scale Integrated Project (IP)
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT)
Project acronym: FI-Core Project full title: Future Internet - Core Contract No.: 632893 Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform Project Document Number: ICT-2013-FI-632893-WP14-D.14.1.1 Project Document Date: 2015-06-16 Deliverable Type and Security: PU Author: Carlos Ralli Ucendo (TID) Contributors: All partners in IoT Chapter.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 2
Preface
Executive summary
This document describes the FIWARE Internet of Things enablement chapter Reference Architecture,
comprising a set of Generic Enablers that provide the various FIWARE IoT capabilities.
Structure of this Document
The document is generated out of resources publicly provided in the FIWARE wiki. For the current
version of the documents, please visit the public wiki at
https://wiki.fiware.org/Internet_of_Things_(IoT)_Services_Enablement_Architecture
Change History
Release Major changes description Date Editor
v0.1 Initial contributions 2015.04.12 Carlos Ralli Ucendo, All
V0.2 Reviewed version 2015.05.20 All partners
V0.3 Updated based on contributions by partners 2015.05.21 Carlos Ralli Ucendo
V0.4 Review by Security Chapter’s architect 2015.05.29 Cyril Dangerville
Table of Contents
Preface .......................................................................................................................................................... 2
Executive summary ................................................................................................................................... 2
Structure of this Document ...................................................................................................................... 2
Change History .......................................................................................................................................... 2
Table of Contents ...................................................................................................................................... 2
1 Internet of Things enablement Architecture ........................................................................................ 7
1.1 Introduction .................................................................................................................................. 7
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 3
1.2 Architecture Overview .................................................................................................................. 7
1.2.1 Complete Architecture .......................................................................................................... 8
1.2.2 Typical IoT use-case Scenarios (I): Common Simple scenario ............................................... 9
1.2.3 Typical IoT use-case Scenarios (II): NGSI Native Scenario enabling Devices composition .. 11
1.2.4 Typical IoT use-case Scenarios (III): Full Scenario enabling Devices composition .............. 12
1.3 Basic Concepts ............................................................................................................................ 13
1.3.1 Device, IoT-end node, IoT Resource, Thing and IoT NGSI Context Entity ........................... 13
1.3.2 Interface Abstraction Levels ............................................................................................... 13
1.3.3 OMA NGSI – Context Management .................................................................................... 16
1.4 Architecture description of GEs .................................................................................................. 16
1.4.1 Backend ............................................................................................................................... 16
1.4.2 IoT Edge ............................................................................................................................... 17
1.5 Interfaces to other Chapters ....................................................................................................... 17
1.5.1 Data ..................................................................................................................................... 17
1.5.2 Security ............................................................................................................................... 17
1.5.3 I2ND .................................................................................................................................... 17
2 IoT Backend Device Management ...................................................................................................... 19
2.1 Copyright ..................................................................................................................................... 19
2.2 Overview ..................................................................................................................................... 19
2.2.1 IoT SDKs ............................................................................................................................... 21
2.2.2 IoT Agent Manager.............................................................................................................. 22
2.2.3 IoT Edge Management ........................................................................................................ 22
2.3 Basic Concepts ............................................................................................................................ 22
2.3.1 Ultralight2.0 ........................................................................................................................ 22
2.3.2 OMA LWM2M ..................................................................................................................... 22
2.3.3 MQTT ................................................................................................................................... 22
2.3.4 IETF CoAP ............................................................................................................................ 23
2.3.5 FIWARE NGSI ....................................................................................................................... 23
2.4 Main Interactions ........................................................................................................................ 23
2.4.1 IoT Agent: Ultralight2.0/http .............................................................................................. 23
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 4
2.4.2 IoT Agent: Ultralight2.0/MQTT ........................................................................................... 26
2.4.3 IoT Agent: LWM2M/CoAP ................................................................................................... 27
2.5 Basic Design Principles ................................................................................................................ 31
3 IoT Backend IoT Broker ....................................................................................................................... 33
3.1 Copyright ..................................................................................................................................... 33
3.2 Overview ..................................................................................................................................... 33
3.2.1 Role of IoT Broker GE .......................................................................................................... 33
3.2.2 Data model and interface outline ....................................................................................... 33
3.2.3 Functionality outline ........................................................................................................... 35
3.3 Main Concepts ............................................................................................................................ 36
3.3.1 Basic Concepts .................................................................................................................... 36
3.3.2 Additional Concepts ............................................................................................................ 37
3.4 Main Interactions ........................................................................................................................ 37
3.4.1 Query Handling ................................................................................................................... 38
3.4.2 Subscription Handling ......................................................................................................... 39
3.4.3 Update ................................................................................................................................. 40
3.4.4 Notification ......................................................................................................................... 41
3.4.5 Availability Notification ....................................................................................................... 42
3.5 Basic Design Principles ................................................................................................................ 43
3.6 References .................................................................................................................................. 44
4 IoT Backend - IoT Discovery ................................................................................................................ 45
4.1 Copyright ..................................................................................................................................... 45
4.2 Overview ..................................................................................................................................... 45
4.2.1 Data model outline ............................................................................................................. 45
4.2.2 Functionality outline ........................................................................................................... 45
4.3 Basic Concepts ............................................................................................................................ 46
4.3.1 FIWARE NGSI ....................................................................................................................... 46
4.3.2 GE Internal Architecture ..................................................................................................... 47
4.4 Additional Concepts .................................................................................................................... 49
4.4.1 Associations in FIWARE NGSI-9 ........................................................................................... 49
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 5
4.4.2 Optional Features for Discovery ......................................................................................... 49
4.5 Main Interactions ........................................................................................................................ 50
4.5.1 Registration ......................................................................................................................... 50
4.5.2 Discovery ............................................................................................................................. 52
4.5.3 Subscription ........................................................................................................................ 53
4.5.4 Notification ......................................................................................................................... 53
4.6 Added-value (Optional) Features ................................................................................................ 54
4.6.1 Geographical Discovery ....................................................................................................... 54
4.6.2 Probabilistic Discovery ........................................................................................................ 54
4.6.3 Semantic Discovery ............................................................................................................. 60
5 FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling ............................................................ 65
5.1 Copyright ..................................................................................................................................... 65
5.2 Overview ..................................................................................................................................... 65
5.2.1 Internal architecture components diagram ........................................................................ 66
5.2.2 Main interfaces ................................................................................................................... 68
5.2.3 Main components ............................................................................................................... 69
5.3 Basic Concepts ............................................................................................................................ 72
5.3.1 Event concepts .................................................................................................................... 72
5.3.2 Complex Event Processing concepts ................................................................................... 73
5.3.3 Choosing the right implementation for you ....................................................................... 73
5.3.4 EspR4FastData implementation .......................................................................................... 74
5.3.5 SOL/CEP implementation .................................................................................................... 75
5.3.6 NGSI ..................................................................................................................................... 79
5.3.7 ETSI M2M ............................................................................................................................ 79
5.4 Main Interactions ........................................................................................................................ 79
5.5 Basic Design Principles ................................................................................................................ 81
5.6 Re-utilised Technologies/Specifications ..................................................................................... 81
6 IoT Gateway – Protocol Adapter ........................................................................................................ 82
6.1 Copyright ..................................................................................................................................... 82
6.2 Overview ..................................................................................................................................... 82
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 6
6.3 Basic Concepts ............................................................................................................................ 82
6.3.1 Base Driver .......................................................................................................................... 86
6.3.2 Protocol Adaptation ............................................................................................................ 86
6.3.3 Generic Device Access API .................................................................................................. 87
6.4 Main Interactions ........................................................................................................................ 87
6.5 Basic Design Principles ................................................................................................................ 89
7 IoT Gateway Device Management ...................................................................................................... 90
8 Glossary ............................................................................................................................................... 91
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 7
1 Internet of Things enablement Architecture
1.1 Introduction
Internet of Things chapter provides the Generic Enablers that allow Things to become available,
searchable, accessible, and usable context resources, thus fostering interactions between FIWARE-based
Applications and real-life objects.
In this context, Things mean any physical object, living organism, person or concept interesting from the
perspective of an application and whose parameters are totally or partially tied to sensors, actuators or
combinations of them.
The successful innovative idea behind FIWARE IoT is that all Things or IoT resources are exposed to
FIWARE Application developers just as other NGSI Context Entities. Therefore, developers will not have
to deal at all with today's complexity and high fragmentation of IoT technologies and deployment
scenarios. On the contrary, Application developers will just need to learn and use the same NGSI Context
Broker API used in FIWARE to represent all Context information.
Developers will just read Entity attributes to gather sensors information and will just modify (update)
specific attributes to trigger commands that will be delivered to actuators.
Normally, during FIWARE Releases (R1 to R6), the Backend will refer to the IoT Backend enablers
instances running in the FIWARE Lab, as described in the FIWARE Catalogue.
1.2 Architecture Overview
IoT chapter Architecture deployment varies from simple scenarios – connecting few devices using
standard IoT communication protocols to the Data chapter’s Context Broker – to more complex
scenarios distributed across a large number of IoT networks connecting IoT Gateways and IoT nodes and
providing advanced composition and discovery functions.
IoT GEs are spread over two different domains:
IoT Backend. It comprises the set of functions, logical resources and services hosted in a Cloud
data enter. Up north, it is connected to the Data chapter’s Context Broker, so IoT resources are
translated into NGSI Context Entities. South-wise the IoT backend is connected to the IoT edge
elements that make up the physical IoT infrastructure.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 8
IoT Edge. It is made of all on-field IoT infrastructure elements needed to connect physical
devices to FIWARE Apps. Typically, it comprises: IoT end-nodes, IoT gateways and IoT networks
(connectivity). The IoT Edge and its related APIs will facilitate the integration of new types of
gateways and devices, which are under definition in many innovative research projects, and
warranty the openness of FIWARE IoT architecture.
1.2.1 Complete Architecture
This section depicts a complete scenario with all FIWARE IoT GEs (Backend and IoT Edge) and their
relationship in terms of APIs and communication protocols.
Please, to avoid unnecessary complexity, check first our current and planned typical scenarios. Check
them at the next subsections "Typical IoT use-case Scenarios".
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 9
Figure 1: IoT Architecture Overview
1.2.2 Typical IoT use-case Scenarios (I): Common Simple scenario
This scenario stands for a simple integration of IoT devices into the Data chapter Context Broker so that
FIWARE Application developers can interact with them. This use-case scenario is currently working and
available to FIWARE users at the FIWARE Lab.
The mandatory GEs for this scenario are:
Backend Device Management GE. That will translate the Device or Gateway specific
communication protocol into NGSI. It will always handle sensor notifications from the device
towards the Context Broker and for some specific protocols actuation/command messages from
the Context Broker to the device. Please refer to this GE architecture and documentation for
more information.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 10
Data Chapter's Context Broker GE. It will handle all Context Entities representing the IoT
devices. It is the natural interface for FIWARE Application Developers for reading IoT sensing
information (Entity attributes) and, if implemented for the Device communication protocol, also
to trigger commands (Entity specific attributes).
Optional GEs and functions are:
Gateway Logic GE. It will handle the IoT Edge management API and functions plus the gateway-
to-gateway API and functions. For the first (Edge Management) it needs the correspondent
function/module to be activated and configured in the BE Device Management GE.
Figure 2: IoT Common Simple Scenario
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 11
1.2.3 Typical IoT use-case Scenarios (II): NGSI Native Scenario enabling Devices
composition
This scenario describes a use case where IoT end nodes or at least intermediate Gateways implement
the NGSI protocol. This use-case scenario is currently under experimentation and will be available soon
to FIWARE users at the FIWARE Lab.
If devices are NGSI native they might be directly connected to the Data Context Broker, but also they
may be connected to the Gateway Data Handling GE to perform events classification and/or
composition (Complex Event Processing) at the Gateway level.
Data Handling GE may push NGSI notifications directly to the Data Context Broker or might be pushing
them to the IoT Broker GE that combined with the IoT Discovery GE will be able to provide Things
composition and discovery functions.
Backend Device Management GE is optional, as it is only needed in this scenario if IoT Edge
Management functions are offered to IoT integrators.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 12
Figure 3: IoT NGSI Native Scenario
1.2.4 Typical IoT use-case Scenarios (III): Full Scenario enabling Devices composition
This scenario includes all IoT enablers and it is depicted by the complete diagram shown at the
beginning of this section (figure 3). It is able to handle and combine (i.e. build Things from devices)
native NGSI devices/Gateways and also any other kind of IoT devices/gateways (via the BE Device
Management IoT Agents).
This use-case scenario is currently under experimentation and will be available at the end of Release 5 to
FIWARE users at the FIWARE Lab.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 13
1.3 Basic Concepts
1.3.1 Device, IoT-end node, IoT Resource, Thing and IoT NGSI Context Entity
A device or IoT end-node is a hardware entity, component or system that either measures or influences
the properties of a thing/group of things or performs both activities concurrently. Sensors and actuators
are devices. Normally, we will use the term IoT-end node for complex physical devices with several
sensors and actuators (e.g. Arduino-based complex system. This means several boards -Arduino or not-
connected to an Arduino board for communications). Devices might use standard or proprietary
communication protocols that are natively pushed to the Backend GEs or might be translated into any
other standard or proprietary communication protocol at IoT Gateways. A particular case of
communication protocol is OMA NGSI. However, we do not expect devices and gateways to implement
it and therefore FIWARE GEs provide a way to translate these protocols into NGSI.
An IoT resource is a computational element providing access to sensor/actuator devices. An information
model for the description of IoT resources can include context data like location, accuracy, status
information, etc. IoT resource-level data consists not only of the measured data, but also context
information like the data type, a time stamp, accuracy of measurement, and the sensor by which the
measurement has been performed. IoT resources can be addressed using a uniform addressing scheme.
The resource is usually hosted on the device but it has a logical representation in the backend as well.
A thing can be any object, person, or place in the real world. Things are represented as virtual things
having an entity ID, a type and several attributes. Sensors can be modelled as virtual things, but other
real-world things like rooms, persons, etc. can be modelled as virtual things as well. So thing-level data
consists of descriptions of things and their attributes, while information on how the data has been
obtained might be contained as meta data, but is in general out of scope.
In FIWARE, all Things, IoT resources or IoT end-nodes are represented in the end as IoT NGSI Context
Entities in the Data chapter’s Context Broker GE.
1.3.2 Interface Abstraction Levels
This section explains in detail the functionalities and interfaces provided at each one of the IoT domains
or levels.
1.3.2.1 Backend Level
From a functionality point of view, IoT backend GEs perform the following functions:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 14
Provision of Things NGSI Context entities. This means to create an NGSI Context Entity per each
one of the IoT resources. It is typically carried out by the Backend Device Management GE,
although in some advanced scenarios it might be alternatively implemented by the IoT Broker
GE. The GE in charge of this function plays the role of Context Producer for all Entity attributes
related to IoT resources sensing/observations. On the other hand, it registers itself as Context
Provider for those Entity Attributes related to actuation capabilities. This way, whenever a
FIWARE developer modifies (updates) one attribute related to actuation, the Context Broker will
push back the notification to this enabler that will in turn trigger a command to the device.
IoT Southbound protocol adaptation. The variety (fragmentation) of communication protocols
for IoT devices and/or gateways is extremely high today. FIWARE implements the most reputed
open standards in the market today to ease the connection of any kind of IoT devices.
Moreover, it provides an easy way to extend the number of supported communication protocols
regardless they are based on open or proprietary specifications. This function (in the backend) is
carried out by the Backend Device Manager GE so it handles the translation of IoT southbound
protocols (sensing and actuation) into/from the OMA NGSI protocol. For some specific
communication protocols, some tools (SDK) for specific or generic hardware platforms are also
provided for IoT integrators convenience.
IoT Edge management. Some configuration, operation and monitoring functions regarding IoT
Edge elements (Connectivity/Networks, Gateways, End-nodes) might be controlled from the
Backend Device Management GE and thus exposing a convenient API to IoT integrators in
addition to the NGSI API for Context Broker interconnection.
IoT Devices composition and discovery. IoT resources might be combined into new IoT
resources representing a more abstract concept. For instance, all temperature sensors in a room
might be combined (e.g. average of all observations) to create an improved "Room-
A_Temperature" thing with its related Context entity. Discovery facilities are provided for
atomic and composed IoT resources. These functions are provided by the IoT Broker GE in
combination with the IoT Discovery GE.
1.3.2.2 IoT Edge level
On the other hand, IoT Edge GEs provide the following functionalities:
IoT Southbound protocol adaptation. This functionality is similar but pretty much reduced
compared to the one provided at the backend, it just means to translate specific protocols into
NGSI within a gateway device so that native NGSI entities might be pushed directly to the
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 15
backend enablers (typically the IoT Broker or directly the Data chapter Context Broker). This
function is provided by the Protocol Adaptor GE. Current supported technologies are ZigBee and
RFID tags.
Complex (NGSI) Event Processing. Although there is one specific GE in the Cloud for this
functionality (Data Chapter’s CEP GE), for some scenarios it might be useful to reduce the
network traffic exchanged between the IoT edge elements and the Cloud infrastructure. It is
provided by the Data Handling GE within a Gateway device. Data Handling GE consumes,
processes and delivers NGSI events so it is typically used together with the Protocol Adaptor in
the same gateway device.
Gateway Logic. This function handles a gateway-to-gateway API (currently under definition) and
the IoT Edge configuration API at the gateway level (currently under definition too). For the IoT
Edge configuration of gateways existing protocols such as OMA-LWM2M management
interfaces, OMA-DM or BBF TR.69 are to be considered. It will be provided by the Device
Management GE.
IoT end-nodes Configuration. Some IP-capable nodes will not traverse gateways and therefore
they might be operated and monitored directly. This means they will implement themselves an
IoT Edge configuration API at the IoT-end node level. In this specific case mainly the
management interfaces of OMA-LWM2M are expected to be considered due to the constrained
nature of IoT-end nodes.
IoT Networks Configuration. In complex scenarios such as smart cities, IoT-end nodes are not
expected to be connected over a unique connectivity network. On the other hand, several IP
alternatives are expected to co-exist, namely: cellular (2G, 3G, 4G and soon 5G), meshed-radio
networks (6LowPAN/IEEE.802.15.4), IP/BLE, LPWA, etc. It is important to note that this control
plane is not the IoT devices or end-nodes control plane, but a different one that may include
also interaction with network APIs. This function will be provided at the IoT Edge module in the
Backend Management GE, the Gateway Logic module at Gateways and in a similar module at
IoT-end nodes.
A key design statement is that, whenever present, IoT Gateways are not expected to be permanently
connected to the Backend as per communications design or because of failures. Another relevant
remark is that IoT Gateways are expected to be constrained devices in some scenarios. Therefore, light-
weight implementations of the same GEs as in the IoT Backend, plus additional GEs interfaces helping to
save unnecessary features/GEs might be considered in the Gateway domain.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 16
1.3.3 OMA NGSI – Context Management
The OMA NGSI Context Management standard provides the NGSI-9 and NGSI-10 interfaces to manage
and exchange Context Information about Context Entities. A Context Entity is any entity which has a
state. Values of attributes of defined entities become the Context Information that applications have to
be aware of in order to support context-awareness. Context Information is any volatile or persistent
information, which describes a state of a Context Entity. Of course, Context Entities could be users,
devices, places, buildings, therefore “things” as defined in the FIWARE IoT chapter. Context Information
related to things can be measured by sensors, and combined with other context information manually
set by humans, derived from interactions with the user, operations on handsets or terminals, inferred
from other information, or requested from databases. Adoption of OMA NGSI in the FIWARE IoT chapter
enables to manage the configuration of, and the data associated to, arbitrary physical objects in the real
world, (Devices and Things). Interestingly, it enables this at a level of abstraction that allows getting rid
of the complexity of managing connections with gateways and devices. Actually, updates on the state
and configuration of those physical objects will come as updates on the Context Information (i.e.,
updates on attributes of Context Entities representing those physical objects) and Configuration (i.e.,
updates on information about available Context Entities). The purpose of the NGSI-9 interface is to
exchange information about the availability of Context Information and Context Entities, while NGSI-10
is designed for exchanging the Context Information itself.
References:
NGSI Context Management - Open Mobile Alliance V1.0 May 29, 2012
FIWARE NGSI 9/10 Open API Specification
1.4 Architecture description of GEs
1.4.1 Backend
IoT Edge Generic Enablers (GEs) are:
FIWARE.ArchitectureDescription.IoT.Backend.DeviceManagement
FIWARE.ArchitectureDescription.IoT.Backend.IoTBroker
FIWARE.ArchitectureDescription.IoT.Backend.IoTDiscovery
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 17
1.4.2 IoT Edge
FIWARE.ArchitectureDescription.IoT.Gateway.DataHandling
FIWARE.ArchitectureDescription.IoT.Gateway.ProtocolAdapter
FIWARE.ArchitectureDescription.IoT.Gateway.DeviceManagement. This component will be
renamed to Gateway Logic in forthcoming Releases.
1.5 Interfaces to other Chapters
1.5.1 Data
IoT chapter mainly interacts with the Data chapter's Context Broker GE as long as all IoT resources
(Things) are translated into NGSI Context Entities that are provisioned in a specific Context Broker. It
uses both NGSI10 (for sensing and actuation attributes of Thing entities) and NGSI9 (just for actuation
attributes, if implemented). Please refer to Context Broker GE documentation for more information.
1.5.2 Security
IoT chapter GEs are secured by hiding themselves behind a Security Chapter’s PEP proxy GE that
provides OAuth2.0-token-based access control (including users permissions and roles plus the concept
of FIWARE service-path and subservice-path) using the Security Chapter’s Identity Management GE as
Authentication server, aka OAuth Provider in the context of OAuth. Please refer to FIWARE deliverable
D.17.1.1 for more information.
Security at the Backend domain APIs will be provided by the overall security access mechanism based on
Oauth2.0, developed in the FIWARE Security chapter and used as a reference in the FIWARE testbed.
1.5.3 I2ND
IoT chapter will interact with Robots included in the I2ND chapter. The typical scenario will be Robots
based on ROS architecture interacting with IoT resources represented as NGSI Entities in a Context
Broker. However, straight interactions with IoT elements might be envisaged. Please refer to I2ND
Robotics section for more information.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 19
2 IoT Backend Device Management
2.1 Copyright
Copyright © 2015 by Telefonica I+D
2.2 Overview
The Backend Device Management GE is normally the central enabler at the IoT backend for most
common scenarios.
Basically, it provides the following features:
Connect physical devices to a FIWARE platform. Devices and/or gateways may use different
standard or proprietary communication protocols and API. The following sections provide
details on currently/planned supported protocols, how to configure and scale up the translators
(IoT-Agents) plus information on extensibility.
Manages IoT-related NGSI Context Entities. It handles the connection (northbound) to a Data
chapter Context Broker to create one Context Entity per physical connected device. For most
cases, FIWARE Application developers will only interact with those NGSI Entities.
IoT Edge Management. This functionality is provided by the "IoT Edge Manager" module, which
is planned for FIWARE Release 6. It will provide IoT Integrators with the ability of configure,
operate and monitor IoT end-nodes, IoT Gateways and IoT networks.
The following diagram depicts its overall planned architecture.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 20
Figure 4: Southbound IoT protocols
3.1 Main Components
3.1.1 IoT Agents
The IoT Agents are the software modules handling South IoT Specific protocols and North OMA NGSI
interaction. The minimum configuration of a Backend Device Management GE in a FIWARE ecosystem
includes at least one IoT Agent.
IoT Agents must:
Handle the creation of an NGSI Context Entity in a Context Broker (at its northbound) per each
one of the connected IoT Devices.
Act as Context Producer for those attributes related to sensing capabilities or observations.
Provide an Administration/Configuration API.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 21
IoT Agents may:
Perform Commands triggering whenever a developer updates specific attributes in a
Northbound Context Broker. In this case, the IoT Agent will register itself and play the role of
Context Provider of those specific actuation attributes. If this functionality is not provided, the
IoT Agent must provide an API to receive commands and deliver them to devices.
Currently, the following IoT Agents are specified in the roadmap:
UL2.0/HTTP. Ultralight2.0 is an open specification for devices semantics (capabilities,
observations, commands). This IoT Agent handles devices using this representation over
HTTP/IPv4 connections.
UL2.0/MQTT. This IoT Agent handles devices using this representation over MQTT topics.
LWM2M/CoAP. This IoT Agent connects devices compliant with the ETSIM2M and OneM2M
southbound proposition: OMA Lightweight M2M specification over the REST protocol for
constraint devices/networks (IETF CoAP).
Generic IoT Agent. A skeleton of a generic IoT agent is provided to help IoT integrators to cope
with new standards or proprietary protocols.
2.2.1 IoT SDKs
For some specific hardware Devices/IoT-end nodes and protocols/APIs, SDKs (client/device side tools)
might be optionally provided for certain Hardware platforms in order to facilitate testing and integration
activities.
Those SDKs are not installed in the Backend but at the devices or gateways to be connected. However,
SDKs depend on specific communication protocols and thus specific IoT Agents, so they are provided
together with them and not as IoT Edge GEs.
For FIWARE Release 4, an SDK related to Ultralight2.0/HTTP Agent is provided.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 22
2.2.2 IoT Agent Manager
The IoT Agent Manager is an optional module that will interface with all the IoT Agents installed in a
data enter throughout their Administration/Configuration API. This will enable a single point to launch,
configure, operate and monitor all IoT-Agents in a FIWARE Ecosystem.
This module will be specified in FIWARE Release 5.
2.2.3 IoT Edge Management
The IoT Edge Manager is an optional module that will interface with IoT end-nodes, IoT Gateways and
IoT network APIs throughout their IoT Edge API in order to operate and monitor the IoT Edge
infrastructure, i.e. connectivity, gateways and devices.
This module will be specified in FIWARE Release 6.
2.3 Basic Concepts
The Backend Device Management GE is based on the usage of the following protocols: FIWARE NGSI,
Ultralight2.0 and OMA-LWM2M.
2.3.1 Ultralight2.0
Ultralight2.0 Specs. (HTTP & MQTT Transport)
2.3.2 OMA LWM2M
OMA LightweightM2M v1.1
2.3.3 MQTT
MQTT 3.1 Specification
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 23
2.3.4 IETF CoAP
IETF Constraint Application Protocol: RFC 7252
2.3.5 FIWARE NGSI
NGSI Context Management - Open Mobile Alliance V1.0 Aug 3, 2010
FIWARE NGSI Open API Specification
Context Information interface OMA_NGSI_10
2.4 Main Interactions
2.4.1 IoT Agent: Ultralight2.0/http
This IoT Agent connects devices (or Gateways grouping devices) described with the open description
Ultralight2.0 over the HTTP protocol.
From FIWARE Release 4 onwards, commands will be triggered from specific attributes in the related
NGSI Context Entity.
The following subsections detail the main interactions needed to work with this specific IoT Agent.
2.4.1.1 Service Creation
This operation is thought for private instances of the component or Admin users of the FIWARE Lab
public instance.
Each tenant (customer or group of users) is expected to use a different service. This is the way this GE
enables multitenancy.
A Service is tied to a shared secret, called API-KEY that all devices (or gateways) need to know to be able
to send measurements or receive commands to/from the IoT Agent. This security mechanism is
complementary to the overall IDM architecture for all FIWARE GEs.
Devices sending observations with a FIWARE-Service "S1" and FIWARE-Service-Path "PATH" headers will
be represented up north in the Context Broker as NGSI Entities with the same FIWARE-Service and
FIWARE-Service-Path.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 24
In order to create a Service, an admin will run a specific HTTP POST request. HTTP GET operation at the
same URL will return the existing services.
Figure 5: IoT Agent API – Service Creation
2.4.1.2 Device Registration/Creation
Before a device sends observations it needs to be registered. This registry will include the Device ID,
Entity ID, Entity type, attributes, static attributes and command parameters related to the device.
Depending on the specific service agreements, device registration might be carried out by an Admin,
regular users or both.
To register a device, an HTTP POST request will be run.
Figure 6: IoT Agent API - Registration
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 25
2.4.1.3 Device Observation
Once a device is registered, observations might be sent with a simple HTTP POST request (executed from
the Device or a gateway) that includes the Device ID and the API-KEY in the URL.
Figure 7: IoT Agent API - Observation
2.4.1.4 Device Commands
There are two ways to send UL2.0 commands: Push and Pooling modes.
Push commands are sent to devices in a synchronous way. Therefore it assumes always-on
reachable devices (or gateways) implementing an HTTP server.
Pooling commands are queued in the IoT Agent until a Device asks for them. The Administration
API provides a method to check out the status of commands addressed to a specific device.
For the case of Pooling commands, Devices (or gateways) will run an HTTP request periodically and get
the full queue of pending commands. Push devices will run an HTTP server to synchronously receive the
command once delivered to the IoT-Agent.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 26
Figure 8: IoT Agent API - Commands
2.4.1.5 Administration API
The current Administration API of this IoT Agent comprises the above-described Functions "Service
Creation" and "Device Register".
2.4.2 IoT Agent: Ultralight2.0/MQTT
To be provided in the next Architecture release.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 27
2.4.3 IoT Agent: LWM2M/CoAP
This IoT Agent connects Lightweight M2M Clients (devices), communicating over COAP, to the NGSI
Context Broker in the Data Chapter. In this release, the IOT Agent accepts writes to device attributes
through writes in the Context Entities, but it doesn’t support full commands.
The following sections detail the main interactions with the agent (without entering in low-level details).
2.4.3.1 Device Provisioning
In order for the system to recognize the device and do the appropriate mapping to NGSI context
elements, one of two things must happen: either the device is provisioned in advance into the system,
or a service is provisioned in the system and the device assigned to it. In the former case, the DeviceID
will be used to identify the device, using it as the Endpoint name in LWM2M. The following pieces of
data can be specified in provisioning: device ID, security information, NGSI entity mappings and types of
attributes provided.
Figure 9: IoT Agent API - Provisioning
2.4.3.2 Service creation
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 28
For those cases where the devices are not specified individually, services can be provisioned as a whole.
A service is identified by a pair of (resource, API Key) attributes, and can contain roughly the same kind
of information as the device provisioning requests. The specified resource corresponds to a LWM2M
server endpoint where the clients will send their requests (the API Key will be used in southbound
security interactions, not implemented in the first version of the Agent GEri).
Each time a device arrives to the specified endpoint, it will be assigned to the proper service based on
the endpoint.
Figure 10: IoT Agent API – Service Creation
2.4.3.3 Device Registration
Every device in LWM2M must register to the LWM2M server before starting any interaction. In this
registration the LWM2M client has to indicate the following information to the server: Endpoint name
(DeviceID) and supported objects (a list of links for every OMA LWM2M object that can be accessed by
the server). The LWM2M server uses this information to assign the device to a service (or to retrieve the
device information in case it has been individually provisioned).
Based on the service or device information, the server will decide what attributes of the objects
provisioned by the device are active, lazy or commands, and it will register itself in the Context Broker as
the Context Provider of those in the two last categories (lazy and commands).
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 29
Figure 11: IoT Agent API - Registration
For those attributes defined as active, the server will emit an Observe request (Information Reporting
interface in LWM2M), indicating the desired timeout between measures to be received (using the
WriteAttributes operation).
2.4.3.4 Device lazy observation
For those attributes of the devices that marked as lazy, the updates and reads of Context Entities in the
Context Broker GE will be mapped to Read and Write operations from the Device Management Interface
in LWM2M.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 30
Figure 12: IoT Agent API – Lazy Observation
2.4.3.5 Device active observation
For those attributes of the devices marked as active, an Observe operation is issued from the server
upon registration, expecting subsequent measures to be issued as Notify responses to the Observe
message (as stated in the Information Reporting Interface of the LWM2M Specification).
Figure 13: IoT Agent API – Active Observation
2.4.3.6 Device command
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 31
Commands are issued as changes in a particular value in the Context Entity, and are mapped to Execute
operations from the server to the client. The status of the execution is maintained in a special attribute
in the Context Entity that is updated with any upcoming information to the server.
Figure 14: IoT Agent API – Commands
2.5 Basic Design Principles
Face IoT fragmentation: Today's IoT framework considers a number of vertical silos exploiting
different M2M technologies and protocols. The Backend Device Management GE aims to offer a
simple and flexible solution to connect any kind of IoT devices to a FIWARE ecosystem.
Modular design: Functionalities (including supporting different IoT southbound protocols) are
provided as modules. The minimum configuration of a Backend Device Management GE in a
Data enter comprises one single IoT-Agent. Other configurations may add other IoT Agents, the
IoT Agent Manager or the IoT Edge Manager. All those modules might be installed in the same
or different virtual machines for the sake of scalability and redundancy.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 32
Easy extensibility: An skeleton of an IoT-Agent should be provided to allow IoT integrators to
create new IoT Agent modules for non supported standards or proprietary device/gateway
protocols.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 33
3 IoT Backend IoT Broker
3.1 Copyright
Copyright © 2015 by NEC
3.2 Overview
3.2.1 Role of IoT Broker GE
The IoT Broker GE is an IoT Backend enabler. It is foreseen to run on a machine in a data center, where it
serves as a middleware which enables fast and easy access to Internet-of-Things data. Instead of having
to deal with the technical details of existing FIWARE IoT installations, application developers only need
to set up their application to communicate with the IoT Broker in order to retrieve the data they need.
The IoT Broker takes care of the task of communicating with the different IoT Devices and Gateways to
retrieve the needed information on behalf of the applications.
3.2.2 Data model and interface outline
The main interface exposed by the IoT Broker GE is FIWARE NGSI. This API has been developed by the
FIWARE community as a binding of the OMA NGSI (Next Generation Service Interface) Context
Management standard (OMA NGSI 9/10). This interface is exposed both southbound towards IoT
Gateways and Devices and northbound towards IoT Applications. In other words, the IoT Broker GE
retrieves information from IoT Gateways and Devices via the FIWARE NGSI protocol, while the same
protocol is used by applications to retrieve information from the IoT Broker GE. The IoT Broker
aggregates, filters, translates, and enriches the FIWARE NGSI based information before passing it to the
applications.
FIWARE NGSI consists of an information model and two distinct interfaces for information exchange.
The information model is centered on the concept of so-called context entities. Context entities
represent arbitrary objects of the real world, and the state of such objects is described in terms of the
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 34
values of attributes. In addition, metadata can be used to describe properties of attribute values. In the
context of the Internet-of-Things, context entities are used for representing devices like sensors and the
values they measure. Additionally, and even more importantly, arbitrary physical objects (Things) like
rooms, vehicles, or persons and their attributes like temperature or location, are as well expressed by
FIWARE NGSI context entities.
The two FIWARE NGSI context management interfaces distinguish between two types of information.
The first type is called context information and consists of attribute values and associated metadata as
described above. In other words, context information is information about the state of context entities.
This kind of information is exchanged using the operations that have been defined in OMA NGSI 10 and
adopted by FIWARE NGSI. The second type of information is context availability information, i.e.,
information on where context information can be retrieved by OMA NGSI 10 operations. This kind of
information is exchanged using the operations defined in OMA NGSI 9. More details and references on
these interfaces can be found in the next sections.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 35
Figure 15: IoT Broker interfaces
3.2.3 Functionality outline
The IoT Broker GE is the point of contact for accessing information about things and their attributes (see
data model above). Applications can access this information using the NGSI 10 interface of the IoT
Broker. In the overall FIWARE reference architecture [7], the Context Broker GE from the Data & Context
Chapter enables up-to-date access to Context Information (not only things) by any application. Such
Context Information can be of any nature and comprises, but is not limited to, the Context Information
about things that is made available by the IoT Broker GE. However, the IoT Broker GE can as well be
accessed by any number of applications directly via its NGSI 10 interface. This enables a simpler setup
when applications only use IoT information.
For basic usage, the IoT Broker is a stateless component in the sense that it neither stores context
information (i.e. attribute values) nor context availability information. Instead, it interacts with the
whole IoT deployment in order to satisfy the requests it receives from the Context Broker GE or final
applications: when receiving a query on Context Information, e.g. attributes of Context entities, the IoT
Broker GE first contacts the IoT Discovery GE to perform a discovery. Using the resulting availability
information, the IoT Broker then contacts the information providers using NGSI 10 to receive the actual
attribute values. Note that the number of information providers to be contacted can be arbitrary. The
information is then aggregated and returned to the application.
The NGSI context management interface describes three types of operations for exchanging context
information. The first and most basic kind of operation is a simple query. When an application invokes
the query, it expects to receive context information as the response. The second kind of operation is a
subscription. When an application subscribes to certain context information, it just receives a
subscription ID as the response. Context information is then sent to the application in the form of
notifications. Depending on the kind of subscription, context information can be sent whenever
attribute values change, whenever attribute values exceed or are inside some range, or simply at fixed
time intervals.
Finally, there is a mode of information exchange defined in the NGSI context interface where
information updates are sent without the need to ask for them. When the IoT Broker receives such
updates, it forwards the information to some default application that is responsible for further
processing and/or storing such updates. In the FIWARE reference architecture, this application is the
Context Broker GE.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 36
In case of advanced usage IoT Broker keeps certain kinds of states as represented by the two
repositories in the architecture picture. Firstly, it needs to keep track of existing subscriptions using the
Subscription Storage. This includes both subscriptions received from applications and subscription
issued to IoT Agents (i.e. Gateway Data Handling GE) by the IoT Broker. Secondly, the IoT Broker
includes a repository of context registrations, that is, information about where certain context
information can be retrieved. The latter is a new feature since version 4.2, and its purpose is to enable
usage of the IoT Broker without an external IoT Discovery GE in simple scenarios.
3.3 Main Concepts
3.3.1 Basic Concepts
The IoT Broker GE is based on the OMA NGSI context data model [1].
3.3.1.1 FIWARE NGSI
The IoT Broker GE provides FIWARE's RESTful binding specification of the OMA NGSI context
management interface (FIWARE NGSI specifications for short). More specifically, the GE provides the full
set of NGSI-10 operations, where it acts both as a client and as a server. As for NGSI-9, the IoT Broker GE
acts as a client and provides only the server function required for receiving context availability
notifications.
FIWARE NGSI Context Management specifications are based on the NGSI Context Management
specifications defined by OMA (Open Mobile Alliance) [2]. They take the form of a RESTful binding
specification of the two context management interfaces defined in the OMA NGSI Context Management
specifications, namely NGSI-9 [3] and NGSI-10 [4]. They also solve some ambiguities in the OMA specs
and extend them when necessary to implement the FIWARE Vision.
You can visit the FIWARE NGSI Open RESTful API Specification [5] to learn the main concepts underlying
FIWARE NGSI Context Management specifications. Please also refer to the e-learning course [9] on the
IoT Broker Generic Enabler.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 37
3.3.2 Additional Concepts
3.3.2.1 Associations in FIWARE NGSI
One of the central features of the FIWARE IoT Backend is its ability to derive information about arbitrary
things (cars, people, houses, etc.) from device-level information. The link between these two levels of
abstraction is provided by the association concept defined for FIWARE NGSI. A definition of the concept
and its representation in NGSI-9 messages can be found on the NGSI association definition page [6]. An
association is an ordered pair consisting of a sensor-level entity (possibly with an attribute name) and a
thing-level entity (possibly enhanced with an attribute name). The interpretation of such a pair is that
the sensor provides information about the thing.
Associations potentially play a role for every operation supported by the IoT Broker.
The IoT Broker GE queries the IoT Discovery GE not only for context providers, but also for
associations.
For resolving a queryContext operation, the IoT Broker uses the association concept to find out
which lower-level entities, typically device-level entities, it has to retrieve from the context
providers. For example, for finding out the speed of a certain car, the IoT Broker might ask for
the measurement of a speedometer.
For the translation of thing-level subscriptions into device-level subscriptions the association
concept is used in the same way as in the query case.
When context providers call the update operation of the IoT Broker GE, the latter analyzes
associations in order to find out if updates of attribute values of Thing-level entities have to be
triggered.
3.4 Main Interactions
The IoT Broker Generic Enabler is a middleware used for setting up and maintaining the data flows in
IoT deployments. It is designed to interact with large numbers of IoT data providers and data
consumers. On behalf of the consumers, the IoT Broker retrieves, assembles, and processes information
from the providers, offering the consumers a simple interface and masking the complexity and
heterogeneity of the Internet of Things. For this reason, the IoT Broker GE interacts potentially with a
large number of Gateways, other Backend instances, Devices, and of course data consumers.
Furthermore, the IoT Broker GE typically interacts with at least one instance of the IoT Discovery GE (but
there is also a standalone mode since release 4.2). The latter is where the IoT Broker GE retrieves
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 38
information about where information is available in the IoT installation. Please note that due to a recent
architectural change the IoT Discovery GE is still referred to as "ConfMan GE" in the Flow Charts below.
In the FIWARE architecture, the role of the data consumer is played by a Generic Enabler in the Data &
Context Chapter: the Context Broker. The IoT Broker GE communicates with the Context Broker GE via
the Northbound Interface. The Southbound interface is used to communicate with the so-called IoT
Agents, which are providing data. The role of IoT Agent can be played by either the Backend Device
Management GE, or by the Gateway Data Handling GE.
3.4.1 Query Handling
Queries are one-time requests for information. They are realized by the queryContext operation of OMA
NGSI-10. With NGSI-10, GEs or applications can query for Thing-level information. In reaction to a query,
the IoT Broker determines the set of IoT Agents that can provide the requested information. This is done
by sending a discoverContextAvailability request to the IoT Discovery GE, which returns relevant
associations between Thing-level entities/attributes and Device-level entities/attributes, and provides
addresses of the IoT Agents who can provide the required information. After this step, the IoT Broker GE
queries the identified IoT agents, it aggregates the results, it maps Device-level entities to Thing-level
entities and it forwards them to the GE or application that issued the query.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 39
Figure 16: IoT Broker – Query handling
3.4.2 Subscription Handling
Subscriptions are requests for information updates the issuer wishes to receive, under certain
conditions to be specified in the request message. The picture 17 shows the interaction diagram for the
OMA NGSI-10 subscribeContext operation, but the same interaction pattern also applies to the
updateContextSubscription and to the unsubscribeContext operations. Also in these interactions, like in
Query Handling, the role of the IoT Discovery GE is to provide the relevant associations between the
Device-level and Thing-level entities and the addresses of the relevant information sources (i.e. the IoT
agents).
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 40
Figure 17: IoT Broker – Subscription handling
3.4.3 Update
In general, updates received by the IoT Broker GE are forwarded to the IoT Discovery GE. Before
forwarding, the IoT Broker discovers the Thing-level entities associated to the Device-level entities about
which it has received an update. For that reason the IoT Broker contacts the IoT Discovery GE using a
discoverContextAvailability request. Note that a single received update may translate into several
updates because attributes of several things may need to be updated as a result of an update of a
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 41
device-level entity (e.g., the temperature measured by a given thermometer may lead to updates on a
thing representing a room close to the thermometer, as well as updates on things representing the floor
and the building where the room is located).
Figure 18: IoT Broker – Update handling
3.4.4 Notification
Notifications are the counterpart of subscriptions. A notification is sent whenever the condition that has
been specified in the subscription is satisfied. Like in the update operation, also here the IoT Broker
needs to resolve the relevant associations.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 42
Figure 19: IoT Broker – Notification handling
3.4.5 Availability Notification
When a new IoT Agent having information that is relevant for an existing subscription becomes
available, the IoT Broker is notified about this in an availability notification, so that it can then make a
subscription to the new IoT Agent.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 43
Figure 20: IoT Broker – Availability Notification handling
3.5 Basic Design Principles
Stateless Design: The IoT Broker GE is stateless insofar as no context information is stored by it.
Its role in the Internet of Things is not to serve as a central data repository, but as a central point
which retrieves and aggregates data from various sources on behalf of applications. The absence
of state enables easy replication of IoT Broker GE instances, or federated deployments where
multiple instances of the IoT Broker GE are responsible for different parts or domains of the
Internet of Things. In such cases each IoT Broker GE instance serves as an IoT Agent for the other
instances, and each IoT Broker GE can in principle retrieve any piece of information available in
the IoT deployment.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 44
Silent System: The IoT Broker enables IoT systems whose level of activity is proportional to the
number and complexity of the requests by applications. This is due to the fact that the IoT
Broker only queries IoT Agents when there are requests from applications. The principle of silent
systems is particularly helpful when working with battery constrained devices. However, also
systems that continuously produce data can be built using the IoT Broker GE. In this case, IoT
Agents have to be configured to use the NGSI 10 update operation in order to deliver data
independently of application requests.
3.6 References
[1] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_9/10_information_model
[2] http://technical.openmobilealliance.org/Technical/release_program/docs/NGSI/V1_0-20120529-A/OMA-
TS-NGSI_Context_Management-V1_0-20120529-A.pdf
[3] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI-
9_Open_RESTful_API_Specification_%28PRELIMINARY%29
[4] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI-
10_Open_RESTful_API_Specification_%28PRELIMINARY%29
[5] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_NGSI_Open_RESTful_API_Spe
cification
[6] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/NGSI_association
[7] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Architecture
[8] https://forge.FIWARE.eu/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Open_Specifications_Legal_N
otice
[9] http://edu.FIWARE.eu/course/view.php?id=33
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 45
4 IoT Backend - IoT Discovery
4.1 Copyright
Copyright © 2014-2015 by University of Surrey
4.2 Overview
4.2.1 Data model outline
The IoT-Discovery GE is the part of the Backend tier of the IoT Architecture which is responsible for
context source availability management. The underlying data model of this GE is based on the OMA
NGSI-9 Context Management Information Model. This model relies on the concept of context entities,
which are generic entities whose state is described by the means of values of attributes and associated
metadata. In the context of IoT, context entities and context entity attributes can be used to model IoT
resources and the variables they measure, respectively. Additionally - and more importantly -arbitrary
physical objects (Things) like rooms, people, cars, etc. and their attributes like temperature, geo-
location, etc. can be used as well.
4.2.2 Functionality outline
The IoT-Discovery GE is responsible for the context availability registrations from IoT Agents, thus
making it the access point for information about entities and their attributes. In particular, the context
availability information provided by the IoT-Discovery GE is either forwarded from IoT Agents exposing
the FIWARE NGSI-9/10 interfaces. The role of IoT Agents can be played by either the Data Handling GE in
IoT Gateways, or the Backend Device Management GE. It is also possible that the context information is
provided by other IoT Backend systems.
Information about context source availability is typically forwarded to the FIWARE Context Broker GE so
that context information about Things becomes accessible to applications. However, it can be the case
that the Context Broker GE may manage context availability information that is not necessarily provided
by the IoT-Discovery GE, therefore linked to the Internet of Things, but gathered from other parts of the
application.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 46
More precisely, using the FIWARE NGSI-9 interface that the IoT-Discovery GE provides, applications and
services will be able to register, discover and subscribe to updates on context availability information
such as:
Information about IoT resources and the variables they measure
Information about Things and their attributes
Information about Associations between entities, whereby the attributes of a Thing can be
derived from attributes of other Things, or from attributes measured by IoT resources.
Optionally, the IoT-Discovery GE may provide different flavors of discovery mechanisms, along with the
main mode of discovery. These optional features include (but are not limited to):
Geographic: where entities can be discovered based on their location.
Probabilistic: where entities are discovered by similarity in syntax.
Semantic: where entities can be semantically-annotated and linked to well-known ontology
models that can provide more detailed information about a particular attribute, and increase
their interoperability with other information on the Web.
4.3 Basic Concepts
The IoT-Discovery GE is based on the OMA NGSI context data model.
4.3.1 FIWARE NGSI
The IoT-Discovery GE implements the RESTful FIWARE NGSI-9 binding specification. FIWARE NGSI
Context Management specifications are based on the NGSI Context Management specifications defined
by OMA (Open Mobile Alliance). They include two RESTful context management interfaces NGSI-9 and
NGSI-10 (see Open RESTful API Specification), which also solve some ambiguities in the OMA
specification for NGSI and extends them where necessary to realize the FIWARE Vision.
You can visit the FIWARE NGSI Context Management tutorial to learn the main concepts underlying
FIWARE NGSI Context Management specifications.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 47
4.3.2 GE Internal Architecture
The internal architecture of IoT-Discovery consists of two main components, the Configuration Manager
and the Configuration Repository.
Figure 21: IoT Discovery – Architecture
4.3.2.1 Configuration Manager
The Configuration Manager component consists of a context information registry in which context
provider applications can be registered. In addition, components interacting with the Configuration
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 48
Manager can perform discovery operations on that context registration information or subscribe to
changes on it.
It detail, the component provides the following functionality:
Allow NGSI-9 clients, such as the IoT Broker to discover and subscribe to context information,
through the NGSI-9 interface exposed by Iot Discovery.
Announce the availability of context information towards the Context Broker GE (or any other
component supporting the NGSI-9 interface) through the northbound NGSI-9 interface. The
Configuration Management component stores the context information in a Configuration
Repository described in next subsection. This repository is used to answer NGSI-9 request in the
exposed interfaces described above.
Receive registrations from IoT Gateways and the Thing-level Adapter through the NGSI-9
southbound interface and store this information in the Configuration Repository, thus updating
the context information registry.
4.3.2.2 Configuration Repository
The Configuration Repository stores information on the availability of context information and can be
accessed through the Configuration Management. When a NGSI-9 client sends a
discoverContextAvailabilityRequest operation in order to find out where the desired context
information can be found, the Configuration Manager returns zero or more ContextRegistration
instances that match the client's request.
Results can also include associated context sources if specified in the operation scope field in the
request. With this approach, IoT-Discovery can maintain higher-level context information that is not
available in the IoT Gateways or Devices. For example, a Gateway might not know the concept of cars,
but only maintains a list of sensors and their measurements. The information about which specific
sensors provide information about same car could be maintained only by the IoT-Discovery GE.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 49
4.4 Additional Concepts
4.4.1 Associations in FIWARE NGSI-9
Figure 21: IoT Discovery – Associations
IoT-Discovery enables enhanced associations. See section 5.7.3.3 to have more information on this.
4.4.2 Optional Features for Discovery
When discovering and subscribing to notification for specific entities, the Operation Scope field in the
respective operation i.e. discoverContextAvailability and subscribeContextAvailability can be used to
specify a particular type of discovery, which can be:
By Association: using the type includeAssociations in the scope field
Geographical: using the type FIWARE::Location.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 50
Probabilistic: using the type search.
There is also an option for discovering context sources via semantic query methods such as SPARQL. This
requires registrations to be available in semantic form. An NGSI-9 message can link to a corresponding
semantic description. This can be done by the client by specifying the link to the semantic description
the in the registration metadata using the type linked-data.
4.5 Main Interactions
IoT-Discovery has a number of interfaces for interacting with applications, users, and other GEs within
the FIWARE architecture. For example, the GE communicates with the Context Broker GE via the
Northbound Interface and with the IoT Agents on the Southbound Interface. With respect to GEs, the
role of IoT Agent can be played by either the Backend Device Management GE, or by the Gateway Data
Handling GE. There is also an interface between the IoT Broker GE and the IoT-Discovery GE. In the
context of NGSI, IoT Discovery plays the role of the server, and any other actors interacting with it play
the role of the client. To illustrate this, in the following diagrams all IoT Agents, GEs, and applications are
abstracted using "NGSI-9 Client".
4.5.1 Registration
In order for their information on Things and/or Resources to be available at the Backend, IoT Agents
(which play the role of "NGSI9 Client") need to register their information to the IoT-Discovery GE. This is
done via the RegisterContext operation. Note that the registration can be performed at various levels of
granularity: registering specific Entity/Attribute combinations, registering that information about certain
entity type is available, or even in general registering that some entity information is available (without
getting more specific) are all possible registrations. Also note that in absence of a Context Broker GE
instance the registration is still stored in the IoT-Discovery GE.
This sequence is used for both new registrations and registrations updates.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 51
Figure 22: IoT Discovery – Registration
4.5.1.1 Registration Update
A context provider can update a registration by using the registration ID that is returned from the
previous registration response.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 52
Figure 23: IoT Discovery – Registration Update
4.5.2 Discovery
The entities that typically play the "NGSI-9 Client" role in this case are the IoT Broker GE (when
requesting context availability information to process NGSI-10 query/updates) or any other NGSI
application that wants to know the context availability associated with a given entity/attribute.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 53
Figure 23: IoT Discovery – Discovery
4.5.3 Subscription
The entities that potentially could play the "NGSI-9 Client" role are any application that needs to be
aware of changes in context availability information. The "Subscribed App" could be the "NGSI-9 Client
itself".
Figure 24: IoT Discovery – Subscription
4.5.4 Notification
The entities that potentially could play the "NGSI-9 Client" role are IoT Agents, while "Subscribed App"
are applications subscribed to context availability information changed by the registration issued by the
"NGSI-10 Client" (we assume that such Subscribed Application are previously subscribed, as shown in
the sequence diagram in previous section).
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 54
Figure 25: IoT Discovery – Notification
4.6 Added-value (Optional) Features
The following sections describe additional features that IoT-Discovery may implement. However, they
are not mandatory for a basic implementation of this GE.
4.6.1 Geographical Discovery
As an optional feature, the IoT-Discovery GE supports the usage of geographic scopes in the operations
discoverContextAvailability and subscribeContextAvailability. In particular, the geographic scopes
defined under keyword SimpleGeoLocation in Appendix D of the OMA NGSI 9/10 specification [1] will be
supported.
For specifying the geographic location of entities and/or context providers, a new type of registration
metadata will be defined and implemented. The exact syntax and semantics of this metadata type is
under discussion.
4.6.2 Probabilistic Discovery
The Probabilistic Search Engine is a component that uses machine learning (ML) techniques for resolving
IoT Descriptions (whether plain XML or RDF). It uses unsupervised probabilistic ML techniques to group
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 55
similar IoT Descriptions. Its purpose is to allow automated discovery, ranking and recommendation of
IoT Descriptions. It provides an efficient mechanism for the search and discovery of registered IoT
Descriptions by automatically clustering all the IoT Descriptions and hence allowing the IoT Search
Engine to be scalable to large object repositories. It also automates the ranking of results in order of
relevance.
Figure 26: IoT Discovery – Probabilistic Search
4.6.2.1 Training the Engine
For the Probabilistic Search Engine to effectively refine the searching process, it goes through several
stages. First it involves a training process whereby all the Descriptions for each model is retrieved from
the repository. It will then extract "textual concepts" from the Descriptions using a parsing mechanism.
The concepts are essentially the elements, attributes and values of a Description. The extracted textual
concepts will then be used to create a "Document-term matrix", which is a mathematical matrix that is
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 56
used to reflect the frequency of textual concepts in a particular IoT Description, whereby the rows
represent the number of IoT Descriptions and the columns represent the numbers of concepts.
Figure 26: IoT Discovery – Concept Extraction
The next stage is another training process for clustering the IoT descriptions. The training involves a ML
technique known as the Latent Dirichlet Allocation. The aim of this technique is to discover a hidden
dimension behind the vector of concepts. This dimension essentially reflects a "topic" or "theme", which
is shared by certain IoT Descriptions. The topic itself is modelled as Latent Factors, which allows any type
of Descriptions to be represented in vector form, and hence making the search much more efficient
compared to keeping them in text format. The model then associates the Latent Factors with the
distribution of Concepts.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 57
Figure 27: IoT Discovery – Latent Factor Extraction
Once the second training process is done, the clusters are then created. The number of clusters created
is based on the number of Latent Factors generated. The vector of Latent Factors that corresponds to an
IoT Description is used to determine which Latent Factor best describes it.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 58
Figure 28: IoT Discovery – Folding in New Descriptions
After the training process, the Probabilistic Search Engine becomes online and is ready to receive
requests. During runtime, the repository is expected to receive more registrations of IoT Descriptions.
When this occurs the IoT Search Engine will read the new IoT Description and "fold" it in to one of the
clusters based in its probability of similarity with one of the Latent Factors in the vector.
4.6.2.2 Querying the Engine
For discovery, a user can query the IoT Search Engine by providing a description template or a SPARQL
query as the input to the search engine. The template should include concepts that are relevant to what
the user is searching for. The search engine will then convert the query into Latent Factor vector- as
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 59
done previously in the second training stage - and then match the query vector to the vector of Latent
Factors that represents the IoT Descriptions stored in the repository. A query example is shown below.
Figure 29: IoT Discovery – Query Match
A query about an Advanced Description is given in the form:
...whereby the query is asking for an Entity - in this case a room (38) - that has Humidity information
about it.
The Search Engine does as follows:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 60
1. The query is converted to a vector of latent factors: {0.038, 0.916, 0.006, 0.006, 0.0, 0.006, 0.01,
0.008, 0.003, 0.0}
2. After analysing the vector, the query belongs to latent factor #2 with the highest probability.
3. Query is forwarded to the registry responsible for cluster #2 and query is matched within that
cluster.
4. The first five results obtained are:
Rank #1: 38_BA_01.owl with a score of 0.998479530076855
Rank #2: 44_BA_01.owl with a score of 0.998479530076855
Rank #3: 01_BA_02.owl with a score of 0.998479530076855
Rank #4: 37_BA_01.owl with a score of 0.998479530076855
Rank #5: 18_BA_02.owl with a score of 0.998479530076855
Once this is done, the respective IoT Descriptions are retrieved from the repository and returned to the
user.
4.6.3 Semantic Discovery
4.6.3.1 Overview
NGSI provides a simple, generic and effective means of exchanging information about context entities.
However, it does not specify a global, interoperable naming convention for them. Hence an entity type,
attribute, or attribute metadata can be syntactically declared differently from different context
producers, even though they might point to the same phenomenon. Or the case could be the other way
round, whereby the same syntax has different semantics among different context producers. Therefore
there is a need to enable the semantic interoperability of context information among context producers
and consumers, so that they are "talking" about the same thing. Here Linked Open Data can play a role
in increasing interoperability, whereby fields within an NGSI description can be linked to other well-
known descriptions that provide more information about it. Since this GE is designed for IoT entities,
then an IoT reference information model is needed to define the linking of an IoT related attribute
declared in NGSI to one that is.
This optional functionality within the IoT Discovery GE is targeted for:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 61
Context Producers wanting to register their IoT entities using semantic IoT descriptions.
Context Consumers using SPARQL for discovering IoT Context Entities.
Applications requiring Semantic Discovery of IoT Context Entities.
Semantically-enabled GEs;
as a reference, semantic IoT Descriptions in this specification are based on the IoT-A ontology
models.
4.6.3.2 Linking NGSI Entities to Semantic Descriptions
Within the FIWARE architecture, NGSI-9 is the main protocol for the registration and discovery of
sources of context information. The NGSI-9 interface for discovering IoT entities is efficient due to its
simplicity and generality. But it is also regarded as a loose binding insofar as it does not provide a global
naming convention for entities and their attributes, which will essentially affect interoperability
between descriptions that might match in terms of syntax, and not in semantics. Therefore, to increase
interoperability, descriptions annotated using NGSI Context Information model should be exposed as
linked open data, so that its attributes can be compared with other entities on the web, whether it be
spatial, temporal or thematic. In order to do this, a mapping must be done between the information
annotated in NGSI to an semantic information model that reflects IoT entities and their properties. As an
initial reference, the ontology models defined by the EU FP7 IoT-A project can be used.
A Context Producer can link an NGSI Entity Description to an Advanced Description by adding a Context
Attribute that provides the URI of the corresponding Semantic Description, when registering the NGSI
Entity via the registerContextRequest operation. This requires the Semantic Description to be available
beforehand.
4.6.3.3 Associations
The Association Engine (AE) is a component that establishes and maintains associations between IoT
Entities and Services that expose Resources, which acts as the representative for interacting with a
Resource. The associations are made based on spatial, temporal and thematic relativity:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 62
Thematic (feature) match - using domain ontologies that capture an Entity's attributes and IoT
Service's input/output parameter.
Spatial match - using either co-ordinate comparison between IoT Entities and Services for
determining close proximity, or location ontologies that model logical locations with properties
such as 'contains'
Temporal match - utilising temporal aspects of entities which have a temporal aspect, such as
meeting rooms with the IoT Service's observation_schedule.
Figure 30: IoT Discovery – Association Process Overview
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 63
Creating the Associations
Formulation of associations utilises the IoT-A Entity and Service models (i.e. Advanced Descriptions).
Dynamic association inference is maintained through rules that incrementally reason on theme, spatial
aspects and time. At start-up, the AE will retrieve from the database all the different types of Advanced
Descriptions - Entities, and Services (N.B. Services represent Resources) - and processes them through
an Association Engine for matching. The matching process first looks for similar themes between Entity
Attributes and Service observation types, e.g. temperature, light intensity, acoustic level. It will then
look for similar locations. Lastly in the matching process it will look for temporal availability of a Service.
Once the matching process is done, the associations generated will then be stored, which can then be
queried via SPARQL.
Figure 31: IoT Discovery – Association Mechanism
Querying for Associations
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 64
The association can then be queried by users via the SPARQL interface. In the query, the predicate used
to retrieve the association is "isAssociatedToService". This will return the URI of the IoT-A service
description, which can then be retrieved by querying the IoT-A Service repository. It is this description
that will provide access to contextual information (e.g. temperature reading) produced by the IoT-A
Resource (e.g. temperature sensor).
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 65
5 FIWARE.ArchitectureDescription.IoT.Gateway.DataHandli
ng
5.1 Copyright
Copyright © 2012-2015 by Orange, Atos
5.2 Overview
The IoT world mainly consists of a huge amount of resources, creating a lot of events to be processed.
The Data Handling GE addresses the need of filtering, aggregating and merging real-time data from
different sources. A lot of applications expect to receive value-added data that are relevant to their
needs, and this can be easily achieved in a decoupled manner thanks to the Complex Event Processing
technology (CEP). This is also referred to as event stream analysis, or real time event correlation. Events
are published by Event Producers and subscribed to by Event Consumers. One can only subscribe to
events that are allowed by its granted access rights.
Typical applications that require Data Handling GE are sensor network applications, RFID readings,
supply chains, scheduling and control of fabrication lines, air traffic, smart buildings, home automation,
and so on. These applications have in common the requirement to process events (or messages) in real
time, or near real time. Key considerations for these types of applications are throughput, latency and
the complexity of the logic required. The CEP Engine component of the Data Handling GE typically
processes input data, in order to generate a smaller set of output data, which avoids upper-level
software overloading and network flooding.
The Data Handling GE handles data streams from IoT devices that cannot continuously be online, for
various reasons. Among them are possible network disruptions and power resource constraints. The
latter typically requires optimizing and limiting the communications with the IoT resources. For this
purpose, local storage is desirable, in order to cache the last available processed data and contexts.
The natural surrounding GEs for Data Handling are the IoT Broker, theIoT Discovery GE, and the Protocol
Adapter GE.
Although it can be operated standalone, the Data Handling GE typically receives events from the
Gateway Protocol Adapter GE, and propagates the processed data towards the IoT Broker GE. Prior to
propagating any data, the Data Handling GE must be registered to the IoT Discovery GE, in order to be
identified as a context producer.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 66
It should be noted that Publish-Subscribe and CEP features are also available in other GEs. This should
not be seen as useless redundancy: some features are deliberately distributed at different levels of
granularity, in order to provide reliability to the whole architecture, and also to appropriately dispatch
the load at different levels.
The Data Handling GE comes in two configurations: EspR4FastData and SOL/CEP. EspR4FastData targets
both mobile and fixed IoT gateways: it supports both local storage and data transformation. Despite its
gateway orientation, EspR4FastData could also be used server-side, whereas SOL/CEP is aimed at more
constrained environments, which lack both the CPU resources and memory size for storage, or data
transformation mechanisms. It targets small gateways, various devices and sensors that still have
enough CPU power to run it.
Both configurations are equipped with a CEP Engine component, but they operate in distinct ways. The
differences reflect the requirements for the underlying host hardware, on which the Data Handling GE is
deployed and executed.
The next sections discuss the overall architecture reflecting the different configuration options. Then
each component is described, if necessary in subsections according to the differences.
5.2.1 Internal architecture components diagram
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 67
Figure 32: IoT gateway Data Handling – Architecture
The interfaces for this GE are compliant with the FIWARE implementation of the Next Generation
Services Interface (NGSI), and more precisely, NGSI-9 and NGSI-10 subsections. It should be noted that
the official OMA NGSI Context Management standard neither specifies nor imposes any particular
implementation regarding the exchange format, but simply describes NGSI data types as belonging to
the "XML Schema Part 2" W3C recommendation. For the time being, FIWARE has chosen to support an
XML implementation of the NGSI standard: more information can be found here.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 68
5.2.2 Main interfaces
5.2.2.1 NGSI-9/10 API
The Data Handling API is NGSI-9/10 compliant. It accepts events from any NGSI compliant Event
Producer. In the IoT Service Enablement architecture, the Gateway Device Management GE publishes
device events and data towards the Data Handling GE.
In this setup, the Gateway Protocol Adapter GE registers as an NGSI context provider, by calling the
NGSI-9 registerContext method exposed by the Data Handling GE.
After context registration, the Gateway Protocol Adapter GE can send events by calling the NGSI-10
updateContext method of the Data Handling GE.
The Data Handling GE also registers as an NGSI context provider, by calling the NGSI-9 registerContext
method of the upper-level GE that acts as an NGSI registrar. In the context of the IoT Services
Enablement architecture, this GE is the IoT Backend Configuration Management. After registration, Data
Handling GE calls the NGSI-10 updateContext method of the IoT Broker GE, for each propagated event.
The IoT broker acts as an event consumer.
Third party software and other external GEs can subscribe to Data Handling GE output events by calling
its NGSI-10 subscribeContext methods. For every subscriber, events are propagated by calling their NGSI-
10 notifyContext method, which means subscribers must implement this method, in order for them to
receive events.
5.2.2.2 CEP Management API
This API allows the configuration and operation of the Complex Event Processor: it provides features
such as setting rules, defining event types, and configuring output event recipients.
In the standard configuration, it takes place as follows.
For more hardware-constrained configurations, it takes place as follows.
However, the meaning behind "standard" and "constrained" is left up to common sense and to the one
who is in charge of implementing a solution. There are no strong guidelines regarding how much a
system is embedded-oriented, hardware constrained, or server-oriented. For example, a PC-style
gateway might run EspR4FastData, whereas a high-end sensor might better run SOL/CEP.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 69
It should be noted that the CEP management API methods should ideally be consistent, whether the
implementation concerns a constrained device, a mobile phone, or a powerful gateway.
5.2.2.3 Local Storage API
This API provides features to manage the local database at a low-level, and to query historical data.
This is not available in the SOL/CEP constrained configuration.
5.2.2.4 Data Transformation Management API
This API allows configuring transformers on both CEP input and output. Transformers define how to
update event properties and structure on the fly, before and after event processing.
5.2.3 Main components
5.2.3.1 NGSI-9/10 Publish-Subscribe component
This component is the core of the Data Handling GE NGSI functionality. It allows the Generic Enabler to
communicate with the outside world and the other GEs, in an ingoing and outgoing way, and also
features publish-subscribe functions. It implements the necessary subset of NGSI-9 and NGSI-10
methods for the purpose. It also allows the GE to register itself to the Context Broker GE, which is a
prerequisite to the propagation of events towards the IoT Broker GE.
This organization enables the creation of a Publish/Subscribe network of NGSI compliant GEs that
expose their services to each others.
It should be noted that NGSI compliance does not imply the support of the full set of NGSI methods.
5.2.3.2 CEP Engine component
This is the core functionality of the Data Handling GE.
The role of the CEP Engine component is to process events in real time, which mainly involves filtering,
aggregating and merging data. This processing is based on a set of rules that can be defined by the CEP
administrator.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 70
The CEP Engine component can process XML data structures on its input. It is fed by the Data
Transformation component, which is in charge of XSLT event transformations before processing. Events
do not directly reach the CEP Engine component from outside.
Once processed, events pass through Data Transformation for optional output XSLT transformation
purpose, before being finally sent to their recipients, through the Publish-Subscribe component.
5.2.3.3 Data Transformation component
This component performs XSLT transformations on both input and output XML events.
The Data Handling GE supports XML events. It might also support other lower-level formats, when
configured for constrained devices with SOL/CEP.
When a NGSI-compliant event arrives, it first flows through the NGSI 9-10 Publish-Subscribe component
before being pushed towards Data Transformation and thus transformed or not, depending on
configured transformations. Other XML non-NGSI events come directly to the input of Data
Transformation.
In all cases, NGSI input events are finally translated into a simplified CEP-friendly XML format, that is
easily understood by the CEP Engine component, and easier to address with CEP rules scripting
language. Events are then pushed towards CEP Engine.
Data Transformation also receives processed events from CEP Engine output, for optional XSLT
transformation purpose, before the processed and transformed events are finally propagated towards
external configured recipients.
It should be noted that in the case of the SOL/CEP implementation for constrained environments, the
Data Transformation component is not available.
5.2.3.4 Local Storage component
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 71
This component stores output events from the Publish-Subscribe component and other XML output
events. For example, it could retain historical records of sensor readings. Compulsory metadata are
locally added to the actual stored data, such as source identification and local time stamps.
The amount of stored events can be configured, but of course depends on the available memory, which
may vary greatly among devices and gateways.
The Local Storage component provides methods to access and manage historical event data.
SOL/CEP implementation does not feature a Local Storage component.
5.2.3.5 Graphical rules editor
This optional component allows to manage CEP rules graphically, through the CEP Management API.
It allows graphical CEP rules building, by placing and biding blocks together. The resulting diagram is
then converted into EPL syntax, which is understood by EspR4FastData.
The Graphical rules editor is useful to manage CEP rules in a user-friendly environment. It is not available
for the SOL/CEP configuration.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 72
Figure 33: IoT gateway Data Handling – Graphical Rules Editor
5.3 Basic Concepts
5.3.1 Event concepts
Similar to many topics in science and engineering, the term "event" has different meanings based on
who is observing the event and the context of the observation. Generally speaking, an event is
something notable that happens in the real world or in a system. But in the computer world, an event is
understood as a computer representation of such an occurrence, usually for the purpose of computer
processing. Events can contain data, are immutable, but more than one event may record the same
activity.
Before the actual event processing occurs, optional event pre-processing can be implemented: it is often
referred to as data normalization, validation, and pre-filtering and basic feature extraction. When
required, event pre-processing is often the first step in a distributed, heterogeneous CEP solution.
Heterogeneous, distributed event processing applications normally require some type of event pre-
processing for data normalization, validation and transformation.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 73
An "event object" is a computer representation of an actual event in the real world or in any system. An
event object carries data or properties. But the term “event” is actually more often used than “event
object”, as a simpler, confusing, but yet improper replacement. However, we can consider the terms
“event” and “event object” to be somewhat equivalent when talking about CEP topics.
5.3.2 Complex Event Processing concepts
Complex Event Processing (CEP) is computing that performs operations on events. Common event
processing operations include reading, creating, transforming, and deleting events (ETZION & NIBLETT,
2011). It can also be considered as a method of analyzing real-time streams of data about things that
have happened and deriving a conclusion from them (LUCKHAM, 2011). It typically processes input
events order to produce a smaller set of output events.
It is also a distributed technology that creates actionable, situational knowledge from distributed
message-based systems, databases and applications in real time or near real time. It can provide an
organization with the capability to define, manage and sometimes predict events, situations,
exceptional conditions, opportunities and threats in complex, heterogeneous networks. (BASS, 2007)
CEP processes many events happening across all the layers of an organization or system, identifying the
most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in
real time.
CEP also refers to state processing, which is the change of state occurring when a defined threshold is
exceeded. It requires event monitoring, event reporting, event recording, and event filtering. Any event
may be observed as a change of state with any physical or logical, or otherwise discriminated condition,
of and in a technical or economical system. Each state change usually comes with an attached time
stamp defining the order of occurrence, and a topology mark that can define the location of the
occurrence.
5.3.3 Choosing the right implementation for you
Regarding the Data Handling GE, the choice among the two implementations of the Complex Event
Processor highly depends on the constraint level of the hardware that hosts and runs the enabler.
For very constrained environment, SOL/CEP is the right choice, whereas for mid-end to high-end
gateways, EspR4FastData is appropriate.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 74
5.3.4 EspR4FastData implementation
EspR4FastData highly relies on the Esper (by EsperTech) java library. The latter is available for both Java
and .Net. It features a dedicated CEP rule (EPL) language. The Java version has been chosen for the
FIWARE project.
It is an open-source software under the GNU General Public License GPL.
Esper (by EsperTech) CEP library is mainly used for:
Complex computations: applications that detect event correlation, filter events, merge event
streams, trigger specific actions when particular conditions are satisfied, etc...
High throughput: applications that process large volumes of messages (between 1,000 to 100k
messages per second)
Low latency: applications that react in real-time to conditions that occur
Native supported event formats are plain Java objects, XML, and map objects. XML is the one
implemented in FIWARE.
Esper (by EsperTech) provides an dedicated Event Processing Language (EPL) which allows rules
definition. EPL is an SQL-like language, but is specifically optimized for dealing with high frequency event
data.
The EPL language allows grouping, aggregating, sorting, filtering, merging, splitting and duplicating event
streams, defining time and length sliding windows, matching event pattern, and so on...
Some important concepts drive the CEP Engine component structure:
The event types describe the internal structure of incoming events
The rules define the way event are processed. Rule triggering occurs when event data match the
rule criteria.
Sliding windows store event data in a First-In First-Out (FIFO) manner. They can be sized in
length or duration.
A listener is executed when a rule is triggered. A listener is always associated to a rule.
5.3.4.1 EPL statement structure
A typical statement looks like the following. The brackets indicate optional features.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 75
[INSERT into insert_into_def] SELECT select_list FROM stream_def [AS name] [, stream_def [AS name]]
[,...] [WHERE search_conditions] [GROUP BY grouping_expression_list] [HAVING
grouping_search_conditions] [OUTPUT output_specification] [ORDER BY order_by_expression_list]
5.3.4.2 Examples of rules statement:
Simple fields "select" from a single event source:
Merge event sources:
5.3.5 SOL/CEP implementation
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 76
SOL/CEP is a Complex Event Processor characterised by scalability and performance. It accepts complex
events that are described in the domain specific Dolce Language
With Dolce it is possible to specify what individual events need to be detected, and how and which of
these events are composed into Complex Events.
Figure 34: IoT gateway Data Handling – High-level Architecture
The CEP Engine reads events from the Event Collector, analyses them in the Complex Event Detector and
finally emits them through the Complex Event Generator. Events are accepted from a variety of sources
and protocols (Database, file system, TCP/IP), translated to an internal format by means of adapters and
processed. Likewise, Complex Events can be published towards different channels in a variety of
formats.
For the FIWARE Gateway Data Handling GE it will interact using NGSI.
5.3.5.1 Characteristics
SOL/CEP is designed for the following characteristics:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 77
Sliding Time Window
This applies a time window over the evaluation of the complex event.
Example: all transactions during the last 10 hours. The “last 10 hours” is called the sliding time window
and always refers to the events that happened in this time frame.
Correlation between Event attributes and Complex Events
This means that the values of the attributes that are specified in the Event can be reused and evaluated
in the Complex Event.
Example: the customer name from an incoming event can be copied to the Complex Event.
Complex Event Functions
Aggregation allows totalizing the values of the incoming events, Averaging calculates the average of all
values of a time windows.
Example: the sum of all the money that was withdrawn in the last 5 days.
Example: The average temperature during the last 20 minutes.
Temporal Awareness
A complex event is a combination of simple, individual events. The temporal order in which these events
should happen for the complex event to be raised can be specified using the relative “after” or “during”
keywords.
Example: WarningLightsOff before LandingGearOut LandingGearUp during Approach
Evaluation of attributes
Attributes in the Complex Event can be evaluated using a free formula.
Example: Average(temperature) >= 20
Event Filtering
Incoming Events can also be filtered based on a formula.
Example: TransactionAmount > 1000
5.3.5.2 Technical aspects
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 78
The CEP runs as a process and is language agnostic. Accepts input and generates output through Format
Adapters (explained before)
Application-level language binding is C++.
Low footprint: designed for performance and small resource usage, but scalable to high end
servers.
Operating system support: BSD, Linux (Debian).
5.3.5.3 Dolce Language
In Dolce, a user can specify Events that need to be detected as follows:
This tells the CEP to detect RainfallMeasurement events, with the attributes amount and sensorId. The
event is only accepted for Sensor number 5. Other sensors are ignored. The amount represents the mm
of rain that fell since the last measurement.
NOTE: the Format Adapter needs to map the event type and assign a number to the different incoming
events. This is a convention that is decided at design time by the developer – in this case, the number 5
was chosen to identify RainfallMeasurement events.
A Complex Event could be specified as follows:
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 79
This tells the CEP to raise a complex event when the total amount of rain exceeds 100 mm in 10
minutes. The event will be of type 1600 and contains a field called “Total” which is an integer and
contains the amount of rain that fell.
5.3.6 NGSI
[1] Ref. NGSI Context Management - Open Mobile Alliance V1.0 May 29, 2012
NGSI 9/10 information model for FIWARE
FIWARE NGSI Open RESTful API Specification (PRELIMINARY)
5.3.7 ETSI M2M
[2] ETSI M2M communications
5.4 Main Interactions
The NGSI 9-10 Publish-Subscribe component registers the Data Handling GE to the Configuration
Management GE by calling its registerContext method.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 80
The NGSI 9-10 Publish-Subscribe component sends events to the IoT Broker by calling its
updateContext NGSI method.
When needed, the IoT broker can call the queryContext method to synchronously retrieve
context data from the Data Handling NGSI Publish-Subscribe component.
The Protocol Adapter GE registers itself to the Data Handling GE by calling its registerContext
method.
The Protocol Adapter GE sends events towards the Data Handling GE by calling its
updateContext method.
Figure 35: IoT gateway Data Handling – Main Interactions
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 81
5.5 Basic Design Principles
The Complex Event Processing API was designed with the main CEP concepts in mind: CEP
component engine state, CEP rules management, event type management, and configuration of
the actions that are executed on rule triggering.
The API was designed in a resource-oriented manner.
NGSI interface is also part of the API and it was designed in a restful manner. The Data Handling
GE has to keep technical consistency on its whole API.
5.6 Re-utilised Technologies/Specifications
The Gateway Data Handling GE is based on RESTful Design Principles. The technologies and
specifications used in this GE are:
RESTful web services
HTTP/1.1 (RFC2616)
XML data serialization format.
Other technology which is re-used
Esper (By EsperTech)
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 82
6 IoT Gateway – Protocol Adapter
6.1 Copyright
Copyright © 2014-2015 by Orange, Ericsson, Telecom Italia
6.2 Overview
The Protocol Adapter GE deals with the incoming and outgoing traffic and messages between the IoT
Gateway and registered devices, to be served by either the Gateway Device Management GE or the Data
Handling GE. There may be multiple instances of Protocol Adapter GEs capable of serving partially IoT-
compliant devices, i.e. devices that do not support ETSI M2M (the specifications may be found at the
following link ETSI M2M Latest Drafts). These devices can be IP-based devices, that communicates using
the IP stack (IPv4 or IPv6), or "legacy devices", meaning devices communicating using non-IP based
protocols, for instance ZigBee, or Z-Wave.
The Protocol Adapter GE receives these device-specific protocols and translates them to a uniform
internal API. The exposed API handles capabilities to read and write to the resources, as well as IoT
specific management and configuration services such as resource discovery consisting of both look-up
and publication.
In particular, the ZigBee Protocol Adapter provides a communication conduit into a ZigBee PAN(s)
(Personal Area Network). It supports a mechanism whereby a gateway can interact with individual
ZigBee nodes to exert control over or to obtain data from those nodes, or conversely a mechanism
whereby the nodes can communicate some information to the gateway.
6.3 Basic Concepts
An overview of the Protocol Adapter GE is provided below, and is followed by an identification of the
interfaces.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 83
Figure 36: IoT Gateway Protocol Adapter – Architecture
There are three interfaces to the Protocol Adapter GE. The southbound APIs provide the gateway
external interface to non-ETSI M2M devices hosting sensor and actuator resources. The currently
supported interfaces are IETF CoRE (provided by Ericsson and no more supported) and ZigBee (the
specifications may be found at the following link ZigBee Network Devices Standard Overview).
On the northbound side there are two interfaces:
the first interface is the Protocol Adapter Interface that is a communications protocol to the
Gateway Device Management GE, based on the Generic Device Access API. This interface can be
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 84
used for initiating subscriptions to resources and receiving notifications from resources that
have been tasked with subscriptions, read or write resources that are determined to be online,
publishing resource capabilities in the Resource Directory, or querying devices for their
resources.
the second one is the NGSI compliant Gateway Data Handling API that allows to interact with
the Gateway Data Handling GE. Through this interface, device events and data are published
towards the Gateway Data Handling GE.
Figure 2 introduces the architecture of the FIWARE Protocol Adapter GE.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 85
Figure 37: IoT Gateway Protocol Adapter – Internal Architecture
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 86
6.3.1 Base Driver
The Base Driver is the low-level API for legacy devices (i.e. an implementation of the device specific
protocol stack). Base Drivers handle device discovery and access to sensor and actuator resources in a
protocol-specific way.
For instance, the ZigBee Base Driver is based on the Network Device Gateway Specification defined by
the ZigBee Alliance (ZigBee document 075468r35 may be found at this link ZigBee Network Devices
Standard Overview). The included operations in this specification are:
operations to read and write attributes, and configure and report events;
macro operations for network and service discovery;
endpoint management;
flexible start-up and network join operations;
bi-directional communication mechanisms between ZigBee Base Driver and gateway
6.3.2 Protocol Adaptation
A Protocol Adapter is the glue between a base driver and the Generic Device Access API or the Gateway
Data Handling API. Via the base driver, it discovers devices, tracks events occurred on them and
executes commands to actuate them. Therefore, a Protocol Adapter is necessary for each protocol that
the Base Drivers support. Whether the protocol is standardized or proprietary does not matter. Provided
that the Base Driver is available and the Protocol Adapter is implemented on top, the Generic Device
Access API or the Gateway Data Handling API is able to support the protocol and provide a unified way
to access devices with the protocol. The Protocol Adapter is able to support:
Device discovery. When a new device is discovered and gets available, it shall create and register
this device as a service within the Protocol Adapter framework. When a previously discovered
device gets unavailable, it shall unregister the corresponding service.
Device measurement update. When a service parameter update occurs on a device, it shall
update the corresponding variable on the device in the Protocol Adapter framework and trigger
update for the corresponding device service.
Device actuation. For each action in a service that a device supports, it should implement a
protocol-specific logic and put it as an action in the device service that is registered in the
Protocol Adapter framework.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 87
Device event and data. Its behavior is like an Event Producer and for each device connected it
shall send events and data
6.3.3 Generic Device Access API
The Generic Device Access API (GDA) exposes a high-level, protocol agnostic API towards the Gateway
Data Handling and Gateway Device Management. GDA uses service schemas which are XML-files that
describe the supported resources, i.e. the application profiles. This schema based approach makes it
possible to cover a wide range of applications spanning from home automation, to media, to health
care.
The GDA defines two main data structures:
Device: it represents the sensor/actuator,
Service: it represents a set of functionalities provided by the Device.
The GDA main methods, which have to be used by the Gateway Data Handling and Gateway Device
Management are:
device.getService(<name of service>) – used to get the services associated to a specific device,
service.getProperties() – used to get the list of properties (i.e. attributes) implemented by a
specific service implementation,
service.getAction(<name of action>) – used to get a single action (i.e. command) implemented
by a specific service implementation.
Gateway Data Handling and Gateway Device Management can get sensor data or configure a device by
reading/writing a Service property, and can command an actuator by calling a Service action.
6.4 Main Interactions
Figure 3 shows an example of device and resource discovery, and how to subscribe to resources using
the Protocol Adapter GE. The Device Management GE and Protocol Adapter register listeners for new
devices with the gateway framework (OSGi service bus). When the basedriver finds a new device in the
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 88
network it will register this device in the framework which in turn notifies the listening Protocol Adapter.
The Protocol Adapter can now query the device for resources which are then mapped to a service
schema. These resources are then made available to the Generic Device Management GE.
Subscriptions are also registered in the framework which then triggers the Protocol Adapter to start a
subscription to the requested resource. A new update (e.g. change in sensor value) will send an update
to the Generic Device Management GE.
Figure 38: IoT Gateway Protocol Adapter – Device discovery and subscribe
Moreover the Protocol Adapter GE is an NGSI compliant Event Producer towards the Data Handling GE.
In the IoT Service Enablement architecture, the Protocol Adapter GE publishes device events and data
towards the Data Handling GE. In this setup, the Protocol Adapter GE registers itself as an NGSI Context
Provider, by calling the NGSI-9 registerContext method exposed by the Data Handling GE. After this
context registration, the Protocol Adapter GE sends events, related to the connected devices, by calling
the NGSI-10 updateContext method of the Data Handling GE.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 89
6.5 Basic Design Principles
Projects deciding to support additional protocols should make sure that their implementations provide
the following functionality:
Device discovery
Device measurement update
Device actuation
Device events and data support
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 90
7 IoT Gateway Device Management
This section refers to the previous version of this GE and thus obsolete. It will be replaced soon with
the new functionalities including the Gateway Logic, currently under definition.
At that time it will be included as well in the IoT Architecture deliverable.
Future Internet Core Platform
D.14.1.1: Contribution to FIWARE Reference Architecture (IoT) 91
8 Glossary
GW Gateway. A hub device that intermediates IoT devices communications among
themselves and an IoT Cloud and/or IoT Application.
GW2GW Gateway to Gateway. Refers to Gateway to Gateway communications/APIs.
IoT Internet of the Things.
MQTT MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
http://mqtt.org
NGSI9/10 OMA Context Management Protocol.
http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/NGSI-9/NGSI-
10_information_model
UL2.0 Ultralight2.0. A lightweight version of the SensorML protocol.
https://forge.fiware.org/docman/admin/index.php?editdoc=1&docid=5084&group_id=
11