Project · PDF fileLOD Linked Open Data ... Figure 16: An example ElasticSearch mapping for sensor data storing ..... 33 Figure 17: MQTT message conversion

1

This project has received funding from the European Union’s Horizon 2020 research and

innovation programme under grant agreement 643275, and from the Japanese National

Institute of Information and Communications Technology

Project deliverable Project Number: Project Acronym: Project Title:

643275 FESTIVAL FEderated interoperable SmarT ICT services

deVelopment And testing pLatforms

Instrument: Thematic Priority

Research and Innovation action ICT

Title

D2.4 Integrated Reusable components for EaaS – First release

Contractual Delivery Date: Actual Delivery Date:

01.12.2015 26.12.2015

Start date of project: Duration:

October, 1st 2014 36 months

Organization name of lead contractor for this deliverable: Document version:

ENG V1.0

Dissemination level ( Project co-funded by the European Commission within the Seventh Framework Programme)

PU Public X

PP Restricted to other programme participants (including the Commission

RE Restricted to a group defined by the consortium (including the Commission)

CO Confidential, only for members of the consortium (including the Commission)

2




Authors (organizations) :

Jander Nascimento (CEA)

Juan Ramón Santana (UC)

Giuseppe Ciulla (ENG)

Martino Maggio (ENG)

Shuuichirou Murata (ACUTUS)

Teruaki Yokoyama (KSU)

Toyokazu Akiyama (KSU)

Reviewers (organizations) :



Abstract:

The deliverable "D2.4 Integrated Reusable components for EaaS – First release" consists in the release of

concrete components developed and integrated during the first 14 months of the FESTIVAL project. This

documents reports the technical details of these components that represent the reference implementation of

the FESTIVAL architecture logical modules. The main objective of this document is therefore to show

information related to the different technical activities that have been performed in order to achieve the

testbeds federation to support the “Experimentation as a Service” model of FESTIVAL platform. The

implementation of the FESTIVAL federation, that started with the components presented in this deliverable, will

be completed in the next months to support the experimentation phase to be started in second part of 2016.

Keywords :

Testbed federation, Experimentation as a Service, Resource federation

3




Disclaimer

THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY

OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY

OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Any liability, including liability for

infringement of any proprietary rights, relating to use of information in this document is disclaimed. No

license, express or implied, by estoppels or otherwise, to any intellectual property rights are granted herein.

The members of the project FESTIVAL do not accept any liability for actions or omissions of FESTIVAL

members or third parties and disclaims any obligation to enforce the use of this document. This document is

subject to change without notice.

4




Acronyms and Definitions

Acronym Defined as

CA Consortium Agreement

CKAN Comprehensive Knowledge Archive Network

CKP Cyber Kansai Project

CMS Content Management System

DCAT Data CATalog

DSL Domain Specific Language

DNS-SD DNS-based Service Discovery

FODP Federated Open Data Platform

GE Generic Enabler

GUI Graphical User Interface

HDFS Hadoop Distributed File System

IdM Identity Management

IoT Internet of Things

IT Information Technology

KVM Kernel-based Virtual Machine

LL Living Lab

LOD Linked Open Data

MQTT MQ Telemetry Transport

NICT National Institute of Information and Communications

Technology

NGSI Next Generation Service Interfaces

OD Open Data

ODMS Open Data Management Systems

5




OMA Open Mobile Alliance

P2P Peer-to-Peer

RAML RESTful API Modeling Language

REST REpresentational State Transfer

RIA Rich Internet Application

RSpec Resource Specification

SFA Slice based Federation Architecture

STH short term Statistics and data

VM Virtual Machine

6




Revision History

The following table describes the main changes done in the document since it was created.

Revision Date

Description Author (Organisation)

V0.1 16/11/2015 Table of contents – first version Martino Maggio (ENG)

V0.2 19/11/2015 Table of contents – second version Martino Maggio (ENG)

V0.3 29/11/2015 Initial contribution on: Introduction, SensiNact – FESTIVAL’s testbeds integration, FIWARE – JOSE integration, Open Data Federation and Connectors ACL for FIAPStorage, JOSE’s VMs




V0.4 7/12/2015 Updated contribution on: Introduction, SensiNact – FESTIVAL’s testbeds integration, FIWARE – JOSE integration, Open Data Federation and Connectors ACL for FIAPStorage, JOSE’s VMs



Teruaki Yokoyama (KSU)


V0.5 13/12/2015 Updated contribution on previous chapters.

Jander Botelho do Nascimento (CEA)





V0.6 21/12/2015 Updated contribution on previous chapters. Added, chapter 6 and Annex 1.

Final draft for internal review

Jander Botelho do Nascimento (CEA)




V1.0 23/12/2015 Final version Martino Maggio (ENG)



7




The FESTIVAL Project

The development of the Internet of Things is set to have a strong impact on many aspects of society. Testbeds

and experimental facilities, both of small scale and up to city scale, will be an essential enabler to facilitate

the development of this vision. Facilitating the access to these test-beds to a large community of

experimenters approach is a key asset to the development of a large and active community of application

developers, necessary to address the many challenges faced by European and Japanese societies. FESTIVAL

project’s vision is to provide IoT experimentation platforms providing interaction facility with physical

environments and end-users, where experimenters can validate their Smart ICT service developments in

various domains such as smart city, smart building, smart public services, smart shopping, participatory

sensing, etc. FESTIVAL testbeds will connect cyber world to the physical world, from large scale deployments

at a city scale, to small platforms in lab environments and dedicated physical spaces simulating real-life

settings. Those platforms will be connected and federated via homogeneous access APIs with an

“Experimentation as a Service” (EaaS) model for experimenters to test their added value services. There have

been long years of research work in Europe and Japan on federation of testbeds and more recently on IoT

testbeds. FESTIVAL will as much as possible make reuse of existing software and hardware available in Europe

and in Japan for building such testbeds. Mutually applying European enablers for Japanese testbeds and vice

versa in real-life trials that the project will organize, FESTIVAL will bring have a strong impact in bridging the

gap among the aforementioned component technologies. FESTIVAL, as a common testbeds infrastructure for

efficient communication and collaboration among stakeholders, will push European and Japanese IoT

testbeds and their practices one step beyond the current state of the art.

The FESTIVAL Consortium consists of:

Participant

no. Participant organisation name

Participant

short name Country

1 (CO) Commissariat à l'énergie atomique et aux énergies alternatives –

Laboratoire d’électronique et des technologies de l’information CEA France

2 Universidad de Cantabria UC Spain

3 Engineering Ingegneria Informatica SpA ENG Italy

4 Easy Global Market EGM France

5 Inno TSD Inno France

6 Ayuntamiento de Santander SAN Spain

7 Sopra SOP France

8 (CO) Osaka University OSK Japan

9 Japan Research Institute for Social Systems JRISS Japan

10 Kyoto Sangyo University KSU Japan

11 Knowledge Capital Management Office KC Japan

12 West Japan Communications JCOMM Japan

13 Ritsumeikan University RU Japan

14 ACUTUS ACUTUS Japan

8




Table of contents

EXECUTIVE SUMMARY 13

1. INTRODUCTION 14

2. IOT GATEWAY: INTEGRATION BETWEEN SENSINACT AND FESTIVAL TESTBEDS 21

2.1. SensiNact integrations .......................................................................................................... 21

2.1.1. OpenHab communication bridge ........................................................................................................ 21

2.1.2. EchoNet communication bridge .......................................................................................................... 23

2.1.3. SmartSantander communication bridge ............................................................................................. 24

2.1.4. Ad-Hoc SensiNact ................................................................................................................................ 27

2.2. Integration between Sensors, MQTT and Time Series Database ............................................. 30

2.2.1. Integration target description ............................................................................................................. 30

2.2.2. Fluent plugins for flexible platform federation ................................................................................... 30

2.2.3. Next steps ............................................................................................................................................ 35

3. IT RESOURCE TESTBEDS: INTEGRATION OF FIWARE GE AND JOSE PLATFORM 36

3.1. FIWARE Lab and Generic Enablers ......................................................................................... 36

3.2. First deployment of FIWARE GE on JOSE platform ................................................................. 40

3.2.1. General descriptions ............................................................................................................................ 40

3.2.2. Deploying GEs on JOSE ........................................................................................................................ 42

3.2.3. Connecting sensors .............................................................................................................................. 42

3.2.4. The testbed .......................................................................................................................................... 44

3.2.5. User access controls of FIWARE in JOSE .............................................................................................. 45

3.3. Next steps ............................................................................................................................ 46

4. OPEN DATA FEDERATION: INTEGRATION WITH FESTIVAL OPEN DATA TESTBEDS 47

4.1. ODMS Connectors ................................................................................................................. 48

9




4.1.1. CKANConnector ................................................................................................................................... 51

4.1.2. SocrataConnector ................................................................................................................................ 55

4.1.3. OpenDataFederationNativeConnector ................................................................................................ 59

4.2. Federation API ...................................................................................................................... 62

4.3. Federated ODMS .................................................................................................................. 66

4.3.1. Santander Datos Abiertos .................................................................................................................... 66

4.3.2. Greater Lyon Data ................................................................................................................................ 66

4.3.3. FESTIVAL Japanese Open Data Platform ............................................................................................. 67

4.3.4. FIWARE lab data portal ........................................................................................................................ 67

5. TESTBEDS ACCESS INTEGRATION 68

5.1. Exposing JOSE VMs to the Internet ........................................................................................ 68

5.1.1. Next step .............................................................................................................................................. 69

5.2. ACL for FIAPStorage for JOSE................................................................................................. 70

5.2.1. General Descriptions ........................................................................................................................... 70

5.2.2. Implementation of ACL for FIAPStorage for JOSE ............................................................................... 70

5.2.3. Next steps ............................................................................................................................................ 72

6. FUTURE IMPLEMENTATION PLAN 74

7. CONCLUSION 78

8. REFERENCES 79

ANNEX 1: FEDERATION API – RAML DESCRIPTION 83

10




List of Figures

Figure 1: Festival Logical Architecture ............................................................................................................. 15

Figure 2: Resource based federation technological components ................................................................... 16

Figure 3: KNX devices at PTL ............................................................................................................................ 18

Figure 4: OpenHab device materialization in Sensinact .................................................................................. 22

Figure 5: Sensinact OpenHab bridge device materialization........................................................................... 22

Figure 6: EchoNet metadata device declaration ............................................................................................. 23

Figure 7: FIWARE GE's implementation in SmartSantander ........................................................................... 24

Figure 8 NGSI10 Reference model for ORION ................................................................................................. 25

Figure 9: NGSI Sensinact device bridge ........................................................................................................... 27

Figure 10: Peer to peer sensor data distribution ............................................................................................ 28

Figure 11: Ad-hoc SensiNact sensor data query .............................................................................................. 29

Figure 12: Libelium sensor data collection ...................................................................................................... 31

Figure 13: A sequence diagram of unbuffered plugin ..................................................................................... 31

Figure 14: A sequence diagram of buffered plugin ......................................................................................... 32

Figure 15: An example fluentd configuration for sensor data collection ........................................................ 32

Figure 16: An example ElasticSearch mapping for sensor data storing .......................................................... 33

Figure 17: MQTT message conversion ............................................................................................................ 33

Figure 18: An example fluentd configuration for in-network MQTT message conversion ............................. 34

Figure 19: Sensor data uploads from tiny computers, e.g. Raspberry Pi, Edison, etc ..................................... 34

Figure 20: WireCloud mashup editor .............................................................................................................. 39

Figure 21: Example of behaviours ................................................................................................................... 39

Figure 22: Pinpoints representing location of NGSI sources and graphs ........................................................ 40

Figure 23: Component diagram of the FIWARE in JOSE .................................................................................. 41

11




Figure 24: Libelium sensor ............................................................................................................................... 43

Figure 25: Wi-Fi person counter ...................................................................................................................... 43

Figure 26: The testbed constructed from FIWARE GEs and resources of JOSE and CKP ................................ 44

Figure 27: User access controls of FIWARE in JOSE ......................................................................................... 45

Figure 28: Integrate PEP Proxy and IDM into FIWARE in JOSE ........................................................................ 46

Figure 29: Federated Open Data Platform - General architecture .................................................................. 47

Figure 30: Federation Manager and ODMSConnectors .................................................................................. 49

Figure 31: CKANConnector - General schema ................................................................................................. 51

Figure 32: CKANConnector sequence diagram - registration of a CKAN ODMS ............................................. 52

Figure 33: CKANConnector sequence diagram - Synchronization of a CKAN ODMS ...................................... 53

Figure 34: CKANConnector sequence diagram – Live search on a CKAN ODMS ............................................. 54

Figure 35: SocrataConnector - General schema .............................................................................................. 55

Figure 36: SocrataConnector sequence diagram - Registration of a Socrata ODMS ...................................... 56

Figure 37: SocrataConnector sequence diagram - Synchronization of a Socrata ODMS ................................ 58

Figure 38: OpenDataFederationNativeConnector - General schema ............................................................. 59

Figure 39: ODFNConnector sequence diagram - Registration of a custom ODMS ......................................... 60

Figure 40: ODFNConnector sequence diagram - Synchronization of a custom ODMS ................................... 61

Figure 41: ODFNConnector sequence diagram – Live search on a custom ODMS ......................................... 62

Figure 42: Exposing JOSE VMs to the Internet ................................................................................................ 68

Figure 43: Exposing JOSE to the Internet (Web Service/REST API) ................................................................. 69

Figure 44: Exposing JOSE to the Internet (sensors/actuators) ........................................................................ 69

Figure 45: Original FIAPStorage for JOSE ......................................................................................................... 71

Figure 46: FIAPStorage for JOSE with ACL ....................................................................................................... 72

Figure 47: FIAPStorage for JOSE with OAuth2.0 .............................................................................................. 73

12




List of Tables

Table 1: status of the federation components and the technical activities .................................................... 19

Table 2: OMSConnector Java code .................................................................................................................. 49

Table 4: Example of JSON representation of a dataset ................................................................................... 64

Table 4: Example of JSON representation basic information of a dataset ...................................................... 65

13




Executive Summary

The deliverable "D2.4 Integrated Reusable components for EaaS – First release" consists in the release of

concrete components developed and integrated during the first 14 months of the FESTIVAL project. This

document reports the technical details of these components that represent the reference implementation

of the logical modules defined in the FESTIVAL architecture described in D1.3 "First Architecture". The main

objective of this document is therefore to show information related to the different technical activities that

have been performed in order to achieve the testbeds federation to support the “Experimentation as a

Service” model of FESTIVAL platform. The implementation of the FESTIVAL federation, which started with the

components presented in this deliverable, will be completed in the next months to support the

experimentation phase to be started in second part of 2016.

The deliverable is organised in four main chapters: the first one recalls the logical architecture in order to

move to the technical one, introducing the categories of the components that will be described in next

sections of the document. In this chapter are reported the relationship between the components and the

architecture and it is presented an overview of the status of the implementations achieved until month 14.

The chapter 2 describes all the different components related to the federation of IoT testbeds and the

integration into FESTIVAL’s platform of their resources: in particular are described the developments related

to the sensiNact’s bridges, the components needed to support specific IoT protocol in order to federate the

IoT FESTIVAL testbeds. Furthermore, the chapter provides details about experiments conducted to integrate

sensors and Time Series Databases. For each of these components technical details about design and

implementation are provided.

The third chapter is related to the IT resource testbeds: it describes the integration activities performed in

order to deploy and execute some FIWARE Generic Enablers on the Japanese JOSE platform. The different

deployment scenarios are described in terms of communication between components with a specific focus

on the issues related to the authentication and user access control.

The fourth chapter presents the technical components of the Open Data Federation: in particular, it describes

the general structure of open data connectors and their specific implementations and internal processes for

federating open data repositories based on different technologies. Moreover, technical details about

Federation API are provided (complete specification is included in Annex 1) and status of integration of the

Open Data testbeds (Santander Datos Abiertos, CKAN portal for FESTIVAL in JOSE testbed, FIWARE Lab Open

Data portal and Greater Lyon Data).

The fifth chapter is related to the testbed access: it provides technical information about the activity

performed in order to expose the JOSE virtual machines on the internet and about the implementation of

access control list for FIAPStorage in JOSE.

Finally, the sixth chapter resumes the maturity status of the FESTIVAL architectural components and presents

the next steps to be followed in order to complete the implementation of the technical artefacts needed to

enable the first experimentation phase.

14




1. Introduction

The Festival project aims to create a federation of heterogeneous testbeds in order to provide to

experimenters a uniform access to different type of resources and to execute and to monitor experiments

related to different application domains. Starting from the logical architecture described in D1.3 [1], this

document will provide details about the technical components developed during the first 14 months of the

project, which aimed to achieve the FESTIVAL testbeds federation. In order to have a clear understanding of

the role of the different software artefacts presented in this deliverable, it is necessary to recall some concept

related to the FESTIVAL architecture: Figure 1 depicts the logical architecture of FESTIVAL, based on two

levels of federation:

• Resource based federation

• Experiment based federation

The first level (resource based federation) federates resources of the same category, in order to

provide a unique access to them using a common data model and/or technological protocol. The four

categories of resources are:

• Open Data Resources

• IoT Resources

• IT Resources

• Living Lab Resources

The corresponding four logical components (Open Data Federation, IoT Gateway, IT Resource Manager

and the Living Lab Manager) aggregate and manage the specific type of resources and provide a

common model to represent every single resource and the interfaces to access it.

The logical architecture also defines the “Experiment based Federation” layer, responsible for

aggregating and exposing the resources coming from the underlying levels. This type of federation

aims to provide a unique access point and a common representation for the different four categories

of resources of FESTIVAL federation. The component responsible for achieving this result is “FESTIVAL

Uniform Access Layer”.

15




Figure 1: Festival Logical Architecture

During the first year, technical activities have been focused on the first level of the federation and

different technological solutions were proposed and implemented in order to achieve the “Resource

based federation”; the most of activities that are described in this document (including development

activities) are already completed, while others will be completed in the upcoming months.

Figure 2 depicts the resource based federation layer and provides details about the concrete technical

components that are implemented and integrated as reference implementation for the first version

of the FESTIVAL architecture. In the following a brief introduction to these components is reported,

while next sections of this document describe them in detail.

16




Figure 2: Resource based federation technological components

Open data federation components

17




The components developed to implement Open Data Federation functionalities are:

• Open Data Federation Platform: it provides the main functionalities for managing and implementing

Open Data Federation, as well as federating Open Data repositories (ODMS: Open Data Management

Systems); in order to interact with ODMSs, Open Data Federation Platform uses ODMS Connectors

that provide functionalities for communicating with specific ODMSs. Three types of ODMS

Connectors are provided: CKAN Connector, Socrata Connector and Open Data Federation Native

(ODFN) Connector. Technical details about Open Data Federation Platform are provided in

deliverable D2.3 [2] and in section 4.1.

• Federated open data catalogue: is the web interface that allows the access to the Open Data

Federation Platform functionalities to the end user. Technical details about Federated Open Data

Catalogue are reported in D2.3.

• CKAN Connector: CKAN Connector enables Open Data Federation Platform to interact with CKAN

based ODMS; it is based on a customised third party client for CKAN. More details about CKAN

Connector are provided in section 4.1.1.

• Socrata Connector: Socrata Connector enables Open Data Federation Platform to interact with

Socrata based ODMS; Socrata Connector directly calls REST API provided by Socrata based portals.

More details about CKAN Connector are provided in section 4.1.2.

• Open Data Federation Native Connector: Open Data Federation Native Connector (ODFN

Connector) enables Open Data Federation Platform to interact with custom ODMSs (ODMS that are

not based on CKAN or Socrata) that implements a specific set of API (Federation API) that are natively

supported by Open Data Federation Platform. More details about ODFN Connector and Federation

API are provided respectively in sections 4.1.3 and 4.2.

The involved testbeds that will be federated through Open Data Federation component are:

• Santander City – Santander Datos Abiertos: it supports CKAN Open Data Management System, so

it has been automatically federated through CKAN Connector, without any additional

implementation from the Santander side.

• Lyon City - Greater Lyon Data: it provides a proprietary technology ODMS, so in order to participate

in the Open Data Federation it will implement the Federation API that will be invoked by the ODFN

Connector.

• FESTIVAL Japanese Open Data Platform: it provides Open Data about sensors data collected from

devices available in testbeds.

• FIWARE Lab Open Data portal: it provides Open Data from a various cities and territories related to

FIWARE initiative.

For more details about involved testbeds refer to section 4.3.

18




IoT Gateway components

In this case, the SensiNact Platform is the gateway responsible for the aggregation of the IoT resources.

SensiNact supports different protocols and API layers such as MQTT, REST API, JSON-RPC and Bluetooth 4.0,

among others. In order to aggregate the IoT Festival testbeds, CEA has developed new bridges, which

communicate with the testbeds resources. These bridges are:

• KNX Bridge: this bridge connects SensiNact to PTL testbeds, using the KNX Worldwide Standard for

Home and Building Control [3]. This enables SensiNact to send messages on the KNX message

bus and consequently change the state of devices attached to the bus. PTL (Figure 3), one of

the experimentation facilities involved in FESTIVAL, is used to test the bridge development,

since it is equipped with KNX bus and controllers. This bridge is still under test and not yet

available in the platform.

• Orion Bridge: this bridge connects SensiNact to SmartSantander testbeds, using the Orion Context

Broker, a Generic Enabler provided by FIWARE Lab (see section 2.1.3).

• EchoNet Bridge: this bridge connects SensiNact to EchoNet Lite devices. EchoNet Lite is a

communication protocol developed by Japan focused in devices for SmartHome (see section 2.1.2).

• OpenHab Bridge/Discovery: this bridge connects SensiNact to OpenHab running instances in the

local network and provides access and discovery capabilities to items configured in OpenHab as

devices into SensiNact (see section 2.1.1).

Figure 3: KNX devices at PTL

IT Resource Manager

For the IT Festival Testbeds, the integration activities are still ongoing; the technological solution considered

more suitable for the objective of the Festival project is given by the project Fed4Fire [4], based on "Slice

Based Federation Architecture". In this context, the work to be done will cover both the development of a

centralized component that communicates with the various testbeds and the implementations of connectors

for various testbeds that will be based on the Federation Aggregate Manager API and that will use Resource

Specification (RSpec) language [5] to describe resources.

19




Some experiments have been executed in order to demonstrate the possible integration between the

components of European and Japanese testbeds, in particular, an experiment on the JOSE platform,

regarding an IoT scenario, involved some Generic Enablers from FIWARE-Lab. In this scenario some

experiment have been conducting involving the usage of sensors (i.e. temperature, humidity, noise and dust)

and some generic enablers that collect, provide and visualise this data.

Living Lab Manager

The development of the “Living Lab Manager” component, for creating the federation of Living Labs, is an

ongoing activity. During the first year, a study was conducted in order to identify a model suitable for

representing a Living Lab in an experiment and to identify the mechanisms that allow experimenters to

involve a Living Lab in their experiments.

The main challenge identified in this phase is a consequence of the nature of this particular type of testbeds,

which carries out its activities involving human resources: different Living Labs have different ways of

performing the same activities. Therefore, the management of resources of a Living Lab based testbeds is

very different if compared with testbeds relative to IoT or ICT resources, because they cannot be handled

fully automatically.

In this particular context, it has been chosen to represent a Living Lab through its assets and expertise,

identifying three fundamental aspects:

1. Services: represent the activities that can be performed in the Living Lab. Each Living Lab provides a list of its services that can be part of an experiment.

2. Methodologies: represent the methodologies used by a Living Lab for performing a particular service. These methodologies could be those commonly used (e.g.: brainstorming, mind mapping, etc.) or specific methodologies performed in the Living Lab following a custom approach. Each Living Lab provides, other than the list of services, also a list of methodologies adopted in its activities.

3. People: people that can be involved in an experiment, selecting by some parameters (e.g.: age, gender, education, etc.). Each Living Lab can provide an end-user database as asset that will be used in the experiment for selecting people involved in an experiment.

The works on the Living Lab Federation will regard the development of a centralized component and

connectors that, starting from the concepts described above, will provide a unique representation of Living

Labs to the higher level of the FESTIVAL Architecture.

provides an overview of the status of the different technical components developed and integrated in the

first 14 months of the project; the different colours show the status of maturity of the component/activity.

Table 1: status of the federation components and the technical activities

20




Testbeds Open data

federation IoT Gateway

IT Resource

manager

Living Lab

manager

City of Lyon ODFN Connector for

Greater Lyon Data

City of Santander

/ SmartSantander

ODF CKAN

Connector for

“Santander Datos

Abiertos”

SensiNact NGSi

Bridge

PTL SensiNact KNX

Bridge

ATR FIAP storage

iHouse SensiNact Echonet

Bridge

Jose

ODF CKAN

Connector for

FESTIVAL Japanese

Open Data Platform

Fluent Plugins

FIWARE GE - JOSE

integration

JOSE VMs Internet

access control

Ad-Hoc

SensiNact/PIAX

FIAP storage for

JOSE Fed4Fire

OpenStack AM

FIWARE Platform/

Engineering

FIWARE-LAB

ODF CKAN

Connector for

FIWARE Lab Data

Portal

FIWARE GE - JOSE

integration

Fed4Fire

OpenStack AM

TUBA Tuba Living Lab

Connector

TheLab TheLab Living Lab

Connector

Component developed (first release)

Component in development phase

Component in design phase

The next sections describe the details of the reusable components listed above and the integration activities

performed.

21




2. IoT gateway: integration between SensiNact and FESTIVAL

testbeds

This section describes the activities related to development of the components that will support the

federation of the resources provided by the IoT testbeds. The first part of this section is dedicated to

SensiNact, the gateway chosen as reference implementation for the FESTIVAL IoT gateway: in particular, it

will be presented the bridges that have been developed in order to support the IoT protocols of the FESTIVAL

testbeds. The last part of the section reports the experiments conducted to integrate and federate

environment monitoring sensors and applications in Japanese testbeds.

2.1. SensiNact integrations

SensiNact integrations are necessary whenever a communication with external systems or devices is

needed. This is an extensible part of SensiNact that allows the user to extend some particular behavior.

The extensibility of SensiNact is unbounded and allows bridges to extend or create new behaviors to

make it works on the benefit of a particular need.

Throughout the project, different needs are pointed out and the platform must be extended without

affecting functionalities already in place. Those extensions are developed without changing SensiNact

as an IoT platform. Some bridges developed during milestone (M14) are described in this chapter.

2.1.1. OpenHab communication bridge

OpenHab [6] is a largely used platform to control Smart Objects, with an important community of users

and developers behind the scene, and it has reached a significant slice of the smart -home software

market. Thus, an Openhab bridge for Sensinact was developed in order to allow SensiNact to run side-

by-side with an already running instance of OpenHab, enriching the OpenHab devices with SensiNact

Domain Specific Language (DSL) for Smart-Objects.

Another advantage of this bridge is that SensiNact supports (by transition) all protocols that are

supported nowadays by OpenHab without having to develop new bridges; this is an immediate

response to an imminent need of using SensiNact with devices not yet supported natively by it, but

that however are supported by OpenHab.

As Figure 4 depicts, devices connected to OpenHab can be accessed through SensiNact; OpenHab

provides full access to these devices and allows to manage and to control them directly via SensiNact

as well.

22




Figure 4: OpenHab device materialization in Sensinact

The general device materialization is shown in Figure 4; this process is accomplished via a ZeroConf discovery

system and a metadata query mechanism.

SensiNact subscribes on the network broadcast events that advertise the arrival of new OpenHab instances.

This mechanism of advertisement is present in the RFC 6667 [7] as DNS-SD protocol. Once the SensiNact

gateway instance receives an indication that a new OpenHab instance is present in the local network, a

metadata request is dispatched to the OpenHab REST interface in order to obtain all required information

necessary to materialize this device instance in the gateway. If that information is provided by OpenHab, the

bridge instantiates those devices as a SensiNact Device, making them ready to be used as supported devices

into SensiNact. This procedure is depicted in Figure 5.

Figure 5: Sensinact OpenHab bridge device materialization

23




2.1.2. EchoNet communication bridge

EchoNet is a communication protocol compliant with the standard IEEE 1888 [8] and currently

developed and maintained by Japan. Because it is the main protocol used in one of the iHouse testbeds

of FESTIVAL project, its integration is necessary.

In order to be able to control devices that use this protocol, a SensiNact piece of software, so called

bridge, had to be developed. This bridge is responsible for working as the contact point between the

EchoNet devices and SensiNact Gateway. This bridge is pluggable in SensiNact and can be removed at

runtime without affecting other functionalities, such as the ones linked to native EchoNet device

manipulation.

The bridge depends on a configuration service that allows device providers to specify the device

metadata. This is due to the fact that this information cannot be obtained automatically from the

network neither from the device. Figure 6 shows the declaration procedure for EchoNet devices.

Figure 6: EchoNet metadata device declaration

The EchoNet metadata is transmitted from ConfigAdmin (Felix component) to a Device Factory. ConfigAdmin

looks for a Device factory that can be used to represent the EchoNet device metadata; once this procedure

is finished, the device is available in SensiNact.

Since the device cannot exist without its metadata definition, once the metadata is invalidated by the

platform the device is removed at runtime.

It is possible to access EchoNet device in two manners: using the native implementation or non-native

which points to an OpenHab instance that controls the device. The word “native” implies that EchoNet

devices can be accessed directly through EchoNet bridge implementation, while the “non-native”

relies on an OpenHab to get the control over such devices.

24




The native implementation generates the header and creates the payload necessary to perform the

action requested on the device. This action concerns either action requests or query requests that are

sent through the network. It is important to be aware that UDP message access is necessary in order

be able to receive the “acknowledgement” of the device related to the action, which indicates if the

action was performed successfully or not.

This bridge is completely based on the ongoing implementation of EchoNet binding provided by iHouse

development team.

2.1.3. SmartSantander communication bridge

The SmartSantander sensors are deployed in the city of Santander. All of them are being managed by the

SmartSantander platform that exposes several methods to access their values. Within these methods, two

accessing options were identified: using the native REST services implemented for SmartSantander, or the

Generic Enablers from FIWARE (ORION, COSMOS and Short Term Historic) that provide context information

access. In that sense, the ORION instance used in SmartSantander was chosen to integrate this information

into FESTIVAL. Figure 7 shows the implementation of the FIWARE Generic Enablers in SmartSantander.

Figure 7: FIWARE GE's implementation in SmartSantander

25




The integration of the FIWARE GEs in SmartSantander relies mainly in two modules, Resource Configurator,

which is in charge of updating the context information of the devices from the deployment, and the Service

Storer module, which updates the last values gathered from the infrastructure. These modules update the

respective IoT agents that transform the different protocols into the corresponding queries in NGSI10. Old

configured devices using SensorML (Sensor Model Language) [9] format goes through the Device Gateway,

which updates ORION as well.

The other Generic Enabler used in the SmartSantander testbed are updated from ORION, including COSMOS

Big Data and STH (Short Term Historic). The update is made through a configured Cygnus module that

automatically updates the corresponding databases upon new context information.

ORION context broker instance in FIWARE LAB is providing the context access to the SmartSantander data in

FESTIVAL. This access is provided through the OMA (Open Mobile Alliance) NGSI9 [10] (deals with context

information availability) and OMA NGSI10 [11] specifications (deals with context information itself) and

presents an implementation of a context information broker with persistent storage.

Figure 8 NGSI10 Reference model for ORION

26




Figure 8 shows the queries available in ORION to use NGSI10 and access the context data. The interfaces

provide the methods for performing the next operations:

• Register context producers: add an entity with a set of capabilities.

• Update context information: update of the values associated to a determined entity.

• Query entity context information: get the set of attributes associated to an entity, as well as the

current values associated to this set of attributes.

• Discover context information: retrieve the current available entities with their attributes and the

latest values associated to each of them.

• Subscriptions/notifications: subscribe to context information when an event occurs (i.e., associates

to a time interval, a change in the attribute value), receiving the subscribed application (specific IP

and port), the corresponding asynchronous notification where appropriate.

ORION Context Broker is accessible through the IdM Keyrock Identity provider [12], which provides the

security token to access the resources. Once the user has obtained the token, it must be included in all the

queries using the RESTful interface of ORION to access the data.

In order to access sensors deployed in Santander an NGSI communication bridge was developed. This bridge

is capable of importing the sensor information from Santander and makes them available into SensiNact.

The Santander device bridge is in fact c is omposed of two different modules:

• The first one is the Java object implementation of the NGSI specification.

• The second one is connected to the specific Orion-NGSI node and instantiate the SensiNact

resource model.

The overall communication is described in Figure 9.

27




Figure 9: NGSI Sensinact device bridge

2.1.4. Ad-Hoc SensiNact

As the scalability is a big concern when dealing with a large and heterogeneous number of devices, a

SensiNact extension was designed as a proof of concept in order to validate the adoption of a non-centric

approach to manage a large quantity of devices.

This bridge allows a remote sensor data to be distributed among different SensiNact gateways that are

located in different networks, as long as the nodes are reachable among them, the data will be shared

between the peers. This was achieved with the use of a Peer-to-Peer (P2P) data distribution schema.

Under the hood of the P2P data schema distribution adopted in SensiNact, PIAX and JOSE tools are located

and each of them plays a different roles into P2P proof of concept: JOSE by providing the virtual machines

necessary to perform validation and PIAX as data distribution overlay over TCP/IP network.

The entities participating in the proof of concept are depicted in Figure 10. JOSE is located in outer element,

it provides the machines used to validate the concept. In JOSE there are 3 nodes (SnaUK, SnaFr, SnaJP) that

correspond to different independent machine instances that belong to the same network. Those nodes are

accessed via secure shell across a Gateway node. This node is mandatory and has the administrative role,

28




meaning that users connected to this node do not have the permission to run other applications; it is used

only as a mean to access the main nodes.

Each node runs one instance of SensiNact, except for the gateway node. The nodes running SensiNact are

called SensiNact nodes.

The sensor data is gathered via PIAX and can be visualized using external tools. As external tools to visualize

and query information, it is possible to mention a conventional web browser or other tool built exclusively

for that purpose like Kibana/ElasticSearch [13].

Figure 10: Peer to peer sensor data distribution

This implementation allows SensiNact to be widely deployed without losing data management capability over

the sensors and providing a solution for the scalability issue, the initial and main problem tackled by this

implementation.

29




Figure 11: Ad-hoc SensiNact sensor data query

In each SensiNact instance, a PIAX agent runs in order to keep the data synchronized among the SensiNact

instances. PIAX agents are responsible for handling the data synchronization instances independently of the

SensiNact gateway life-cycle. While the REST interface and SNA generic Machinery provide a SQL-Like

common language to fetch the sensor data information without specifying the target agent. This behaviour

is depicted in Figure 11.

Behind the scenes, PIAX is used as the base for implementing the P2P distribution, because it provides an

efficient P2P communication.

SensiNact abstracts the multitude of device types, while PIAX distributes the data in a non-centric approach

among SensiNact gateways.

Each SensiNact instance provides access to all sensor data available in the other gateways in the P2P network

without any further configuration. So, a user that has access to one SensiNact instance can access all sensor

data in the network. This was demonstrated during a demo session for the European commission at the

month 14 milestone.

30




2.2. Integration between Sensors, MQTT and Time Series Database

As described in the deliverable D1.3 [1], federation examples are investigated by KSU and ACUTUS based on

the environment monitoring application which targets the experiment field provided by JCOMM.

Environment monitoring application is simple but it basically includes main components of IoT applications,

e.g.: sensors, storage resources, computing resources and actuators. Since the monitoring results will be

visualized to the citizens as feedback, it also has interactions with them. From that point, it is a good

candidate to consider federation among the testbeds. As also described in the deliverable D3.1 [14],

federation experiment will also be examined by KSU, ACUTUS and JCOMM in order to integrate messaging,

storage and visualization platform. In this section more details about concrete implementation of the

integration are provided.

2.2.1. Integration target description

In the environment monitoring application, currently two types of sensors are prepared to monitor

the environment, one type (“Plug & Sense!” and “Meshlium”) is a set of commercial products provided

by Libelium Comunicaciones Distribuidas S.L. [15]. Meshlium is a gateway equipment and collects

sensor data from Plug & Sense! via ZigBee network. In the followings, we call them “Libelium sensors”.

The other type is “WiFi packet sensor” (AMP sensor) developed by JRISS and RU [16]. AMP sensor

collects WiFi probe packets and provides data for pedestrian flow analysis. The WiFi probe packets

include MAC address, but it is anonymized before recording in order to preserve pedestrians’ privacy.

Those two types of sensors will be integrated into environment monitoring system.

As a candidate messaging infrastructure, MQTT (MQ Telemetry Transport) is one of the target

protocols in FESTIVAL project. Several existing platforms support MQTT and Libel ium sensors also

support it. Furthermore, NICT (National Institute of Information and Communications Technology) has

a plan to add MQTT function to PIAX platform. Connecting existing platforms via MQTT is one of the

important tasks in the FESTIVAL project.

When the inter component connections will be established with messaging, protocol and data format,

the format conversion become a critical issues. For example, Libelium sensors output their data as

XML format. However, most of recent time series database, e.g. ElasticSearch, InfluxDB [17] and so on,

assumes JSON data format. XML to JSON conversion must be done in the messaging infrastructure.

Furthermore, not only sensors but also data storages must be connected into messaging in frastructure.

Since data storages like time series databases do not usually have MQTT subscription function, a

subscription layer in front of data storages is needed.

2.2.2. Fluent plugins for flexible platform federation

In order to provide such a federation function, KSU developed plugins for Fluentd [18]. Fluentd itself

is also a flexible messaging infrastructure. While it uses an original protocol to transfer messages

between Fluentd instances, because of its simple architecture, server operators often use it for log

data collection and monitoring. Fluentd has many kinds of adaptors for input and output data. In order

to utilize those connectivity and MQTT messaging infrastructure together, KSU has developed Fluentd

31




input and output plugin for MQTT [19]. KSU also developed XML parser plugin to convert XML data to

JSON format [20]. The plugins can be used for the following examples of federation.

(a) Commercial sensor data collection into time series database

There are many kinds of commercial sensor products on the market (e.g. Libelium). Major sensor

products support MQTT to upload sensor data. Figure 12 shows how to store uploaded Libelium sensor

data into ElasticSearch.

Figure 12: Libelium sensor data collection

As depicted in Figure 12, fluent-plugin-mqtt-io, fluent-plugin-xml-parser and fluent-plugin-

elasticsearch are used. Two types of messaging are supported in fluent-plugin-mqtt-io. One is

unbuffered and the other one is buffered. Unbuffered plugin forward messages immediately with the

same thread after receiving message from MQTT broker. On the other hand, buffered plugin forwards

messages periodically by specified number of threads, different from the subscriber thread, in order

to prevent blocking of MQTT subscription. Sequence diagrams of unbuffered and buffered plugin are

as follows (Figure 13 and Figure 14).

Figure 13: A sequence diagram of unbuffered plugin

32




Figure 14: A sequence diagram of buffered plugin

The type of the desired plugin can be selected through the “type” configuration field (e.g.: mqtt or

mqtt_buf). In the following example of Fluentd configuration, unbuffered plugin “mqtt” is used.

<source>

type mqtt

bind 192.168.1.100

port 1883

topic 'Libelium/+/+'

format xml

time_xpath '["cap:alert/cap:info/cap:onset", "text"]'

time_key '@timestamp'

attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]'

value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]'

@label @MQTT_OUT

</source>

<label @MQTT_OUT>

<match **>

type copy

<store>

type elasticsearch

host localhost

port 9200

index_name libelium

type_name smartcity

include_tag_key true

tag_key sensor_id

logstash_format false

</store>

</label>

Figure 15: An example fluentd configuration for sensor data collection

33




In this example, the following mapping (Figure 16) is assumed to be created at ElasticSearch.

curl -XPUT 'http://localhost:9200/libelium/_mapping/smartcity' -d '

{"smartcity":

{"properties":

{

"sensor_id":{"type": "string"},

"DUST":{"type": "float"},

"MCP":{"type": "float"},

"HUMA":{"type": "float"},

"TCA":{"type": "float"},

"BAT":{"type": "float"},

"@timestamp":{"type":"date","format":"dateOptionalTime"}

}

}

}'

Figure 16: An example ElasticSearch mapping for sensor data storing

(b) In-network MQTT message conversion

Sometimes, MQTT message conversion must be done in the network because the processing entities do not

have the conversion function. In that case, the configuration similar to the above example can be used. The

difference resides in output configuration. In this example (Figure 17 and Figure 18), since the same MQTT

broker is used to upload converted data, topic rewriting function is used for separating messages before and

after conversion.

Figure 17: MQTT message conversion

<source>

type mqtt

bind 192.168.1.100

port 1883

topic 'Libelium/+/+'

format xml

time_xpath '["cap:alert/cap:info/cap:onset", "text"]'

time_key '@timestamp'

attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]'

34




value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]'

@label @MQTT_OUT

</source>

<label @MQTT_OUT>

<match **>

type mqtt

bind 192.168.1.100

port 1883

topic_rewrite_pattern '^([\w\/]+)$'

topic_rewrite_replacement '\1/rewritten'

</match>

</label>

Figure 18: An example fluentd configuration for in-network MQTT message conversion

(c) Sensor data uploads from tiny computers, e.g. Raspberry Pi, Edison, etc

MQTT output plugin can be used as the following. In the case of a tiny computers (like Raspberry Pi)

equipped with sensors and data outputted as files, fluent-plugin-mqtt-io can be used for uploading

those data (Figure 19).

Figure 19: Sensor data uploads from tiny computers, e.g. Raspberry Pi, Edison, etc

35




2.2.3. Next steps

By using Fluentd plugin, the experimenter can federate existing assets flexibly. KSU will continue to

investigate in order to satisfy further requirements, e.g.: anonymizing process at sensor node which

can be implemented as a parser plugin, CKAN or some other service with REST API integration with

HTTP plugin and so on.

Sensor characteristics are also investigated by JCOMM and KSU. During the investigation, some of

sensors are recognized not to have enough granularities for detecting interesting phenomenon with

the default configuration. The sensor behavior needs to be tuned and it may cause more traffic into

the MQTT messaging infrastructure. The implemented Fluentd plugin for MQTT supports

buffered/unbuffered messaging for handling various kinds of requirements, e.g. : prompt responding,

batch analysis and so on. The performance of the plugins will also be investigated by KSU.

36




3. IT resource testbeds: integration of FIWARE GE and JOSE

platform

This section provides details about integration activities performed about IT resources; in particular it

provides a brief introduction about the FIWARE Lab [21] and the specific GE deployed on JOSE (section 0), a

report about the deployment of FIWARE’s GEs on JOSE platform and a brief description of the future

implementations to support the experimentation phase.

3.1. FIWARE Lab and Generic Enablers

FIWARE Lab provides a platform for testing and experimenting with FIWARE’s Generic Enablers and in

general with FIWARE’s technologies. Moreover, various cities provide their Open Data to FIWARE Lab

in order to establish an environment in which users can test their application on real data. The FIWARE

Lab is composed of series of federated nodes located in different areas among Europe and South

America; in details FIWARE Lab provides 1644 computational cores, 4611 GB RAM, 127 TB, 1234 public

IP addresses and 729 virtual machines [22].

The section 3.2. presents the results about the deployment of some Generic Enablers (GEs) in JOSE

testbed. In particular the selected GEs are:

• Orion Context Broker

• IDAS

• CKAN

• WireCloud

Orion Context Broker

Orion Context Broker [23] provides an implementation of the Publish/Subscribe Context Broker GE

and implements NGSI9 and NGSI10 interfaces. It allows several operations such as:

• registration of context producer applications (e.g.: a temperature sensor within a room);

• notification of changes on context information when they take place (e.g.: variation of a

temperature);

• update about context information (e.g.: provides updates about the temperature registered by a

sensor);

• retrieving context information; Orion Context Broker stores context information and relative updates

and provides that data when applications require them.

Orion Context Broker is able to manage the lifecycle of context information (updates, queries,

registrations and subscriptions). Through Orion Context Broker it is possible to register context

elements, managing them, and to subscribe to context information in order to receive notifications

about them.

37




Orion Context Broker mediates between producers (such as sensors) and consumer applications (such

as an applications based on data provided by the sensors).

In JP testbeds, Orion Context Broker is combined with Cygnus [24] connector; Cygnus provides

functionalities for persisting context data in third-party storages, such as HDFS [25], CKAN[26], MySQL

[27], MongoDB [28], etc. Cygnus takes advantage of the subscription/notification feature of Orion: in

Orion is initiated a subscription for Cygnus and it is configured in order to specify which entities will

be notified to Cygnus when events about them occur (such as update events).

IDAS

IDAS [29] provides an implementation of the Backend Device Management GE [30], so it can play a

central role in most common IoT scenarios. It is based on a modular architecture based on IoT Agents

for an easy installation and for reducing resources requirements. Thanks to this, it supports different

IoT protocols, such as: UL2.0/HTTP, MQTT, OMA LWM2M/CoAP, ThinkingThings Protocol and SIGFOX.

In particular, it provides functionalities for:

• Connecting IoT devices to a FIWARE platform (supporting different protocols).

• Managing IoT-related NGSI Context Entities (handling to a Data Context Broker, such as Orion).

• IoT Edge Management for storing and handling the communications and/or IoT-gateways topology.

Moreover, it provides:

• skeletons of IoT Agents in C++ language and for Node.js [31] framework;

• a client tool (FIGWAY [32]) for connecting physical and virtual devices;

• a specific component (IoT Agent Manager) for monitoring and management of IoT Agent instances.

IDAS is made of two main components:

• DCA-IDAS M2M platform, that consists of three sub-components:

o application services

o real-time event processing

o Device Communication Handling

• IoT-Agent, which plays the role of NGSI connector to external capable components.

38




CKAN

CKAN [26] is an open source data management system that provides tools and functionalities for

publishing, sharing, finding and using data. Its main features are:

• Catalogues system accessible through a web interface and APIs.

• Integration with CMS (Content Management System), such as Drupal [33] and WordPress [34].

• Tools for data visualization and data analytics.

• Workflow for data publishing management.

• Access control.

• Data storage and relative APIs.

Thanks to its open source nature, CKAN can be easily adapted and expanded; in particular, it can be

adapted and or modified by three ways:

• CKAN extensions: the common and recommended way; extensions are plugins that can hook into

different parts of CKAN and can modify its standard procedures. Examples of extensions are:

o Disqus Extension [35], that integrates CKAN with Disqus [36], a service for the management

of comments on web pages.

o Archive CKAN datastore to S3 [37], that provides functionalities for archiving CKAN resources

files on Amazon S3 [38].

o OAuth2 Support for CKAN [39], that provides functionalities for authenticating users through

OAuth2 [40].

• CKAN themes: it is particular type of extension that in contrast to CKAN extensions exposes and

overwrites static files of CKAN for customising the front end and graphic user interface.

• CKAN core: the core is the key component of CKAN and it can be changed and adapted if necessary

for implementing functionalities and features.

WireCloud

WireCloud [41] offers an end-user development tool for producing RIA (Rich Internet Application)

applications including semantic technologies. In particular, it provides a web application for service

mashup and for easily creating web applications through a dashboards/cockpits that integrates data,

application logic and components for building user interfaces. Its key feature is that it does not require

specific programming skills to create applications.

39




Figure 20: WireCloud mashup editor

WireCloud supports catalogues of widgets (reusable component for creating new applications), namely

marketplaces. Marketplaces can be added and removed in WireCloud, and users can explore them, find

needed widgets and import them in their projects. Furthermore, users can publish their application made

with WireCloud on marketplaces.

Moreover, WireCloud supports NGSI protocol as a source, so it is possible to handle sensors data for creating

new applications.

From version 0.8, WireCloud supports behaviours connection in making mashup; behaviours provide

additional features to produced application; an example of possible behaviours is provided in Figure 21,

where two behaviours are implemented: the first, starting from NGSI sources, display position of NGSI

sources on a map, while the second add an addition feature to the application, providing visualization

capabilities of historical data retrieved from a sensor.

Figure 21: Example of behaviours

40




The result of the mashup is depicted in Figure 22 where starting from NGSI sources, relative pinpoints

(representing the location of the sensors) are showed on a map and data retrieved from them are

represented as graphs.

Figure 22: Pinpoints representing location of NGSI sources and graphs

Reported example is extracted from WireCloud’s user guide [42].

3.2. First deployment of FIWARE GE on JOSE platform

3.2.1. General descriptions

At this section, first deployment of the EU reusable components (FIWARE GEs) on JP testbed JOSE is

explained. The testbed will accept sensors to be connected and will provide their data to

experimenters.

The implementation plan, described in the next sections, is based on the deployment of GEs and other

components on native JOSE VM (Virtual Machine), KVM (Kernel-based Virtual Machine), instead of

FIWARE standard OpenStack environment, because it has been demonstrated the fastest and easiest

way to deploy GE in JOSE in first phase of the FESTIVAL project . The component diagram for the plan

is depicted in Figure 23.

GEs and other components involved in the construction of the testbed are:

• Orion: context broker to collect and provide latest data from sensors.

• IDAS: gateway from MQTT to NGSI.

41




• Mosquitto: MQTT broker, receive data from sensors and publish to IDAS.

• Cygnus: fetch latest sensor data from Orion and send those to CKAN.

• CKAN: store and publish historical data.

• Wirecloud: Application Mashup, to create web application that use sensor data.

Experimenters will be able to provide their sensor data by connecting their sensors to Mosquitto and at the

same time they can use sensor data in their application by connecting to Orion and/or CKAN. Their

applications will be created and run on prepared Wirecloud. Experimenters will also be able to run

applications on their own developing environments.

Figure 23: Component diagram of the FIWARE in JOSE

42




3.2.2. Deploying GEs on JOSE

As JOSE’s KVM is run by Ubuntu 12.04 and no prebuilt packages of Orion and IDAS are available for

this platform, those GEs were built from source codes and deployed on JOSE’s KVMs.

Then, MQTT broker (Mosquitto) was placed at CKP (Cyber Kansai Project) [43] network in order to accept

sensor connections from the Internet (JOSE has a strict firewall policy, so public connectivity to the

components working on the JOSE VMs cannot be provided).

To support historical data, CKAN data management system has been deployed. Cygnus connector has also

been added to act as a gateway between Orion and CKAN.

The following source codes and packages are used:

• Orion: https://github.com/telefonicaid/fiware-orion/tree/release/0.22.0

• IDAS: https://github.com/telefonicaid/fiware-IoTAgent-Cplusplus/tree/release/1.0.2

• Mosquitto: Version 1.4.5, DPKG from mosquitto repository ppa:mosquitto-dev/mosquitto-ppa

• Cygnus: https://github.com/telefonicaid/fiware-cygnus/tree/release/0.9.0

• CKAN: Version 2.4, DPKG from http://packaging.ckan.org/python-ckan_2.4_amd64.deb

• Wirecloud: https://github.com/Wirecloud/wirecloud/tree/0.8.1

3.2.3. Connecting sensors

Thanks to the partners, two real sensors are connected to the system. First sensor to be connected is

the Libelium sensor at Kyoto Sangyo University 14 th building (KSU). The following sensing parameters

are collected by the sensor:

• Dust

• Noise

• Humidity

• Temperature

• Battery remaining

Figure 24 shows current deployment status of the Libelium sensors. Left side of the figure shows

“Meshlium” and right side of the figure shows “Plug & Sense!”. “Plug & Sense!” is connected to the

“Meshlium” via Zigbee network. As a default, it collects data every 30 seconds and then “Meshlium”

publish those data to MQTT broker using IBM MQTT client library [44] which is implemented in Java.

Current experiment results show that it is difficult to detect fine granularity environment changes like

increasing or decreasing of people around the sensor. Since “Plug & Sense!” is implemented by using

Arduino [45], the sensing behavior can be customized by modifying the code. a more detailed sensor

behavior investigation is required for extending the usage of the sensor.

43




Figure 24: Libelium sensor

Second sensor is the WiFi person counter at CKP-Umekita. Wi-Fi person counter measures

approximate number of person around the sensor based on WiFi signal from smartphones along with

them. The sensor collects observations as follows:

• number of persons in last period (default: 10 minutes);

• number of persons leaved until the last counting;

• number of persons entered until the last counting.

Figure 25 shows Wi-Fi person counter and its installation location CKP-Umekita. The sensor feature is

implemented on Raspberry Pi [46] with WiFi network interfaces. It collects unique identifier

information from received WiFi signal and uses them for counting existing person there in 10 minutes.

As the first field trial, it is installed at an office in CKP-Umekita. There is an office for researchers and

experiments. Since some persons are working there and sometimes visitors appear, it is a proper

location for the trial. All of the monitored data are transmitted to the platform. Applications and

visualizations tool based this information will be taken into account for implementation.

Figure 25: Wi-Fi person counter

44




3.2.4. The testbed

Figure 26 shows details about the testbed constructed on FIWARE GE, computing resources of JOSE

and CKP’s network resources.

Figure 26: The testbed constructed from FIWARE GEs and resources of JOSE and CKP

Experimenters, who want their sensors to be connected to this testbed, should request registration to the

testbed; then they can send their sensor data to MQTT broker (at lower left of the Figure 26). IDAS (at middle

left of the figure) receives sensor data from registered experimenter’s sensors and send the data to Orion (at

top left of the figure) by calling NGSI API. The Orion context broker holds latest sensor data and provides

them to other components. Cygnus (at top right of the figure) is one of such components; Cygnus fetches

new data from the registered sensors and send them to CKAN. CKAN stores both of the latest and older data

as “historical data” and provides them through its API.

Experimenters, who want to use the sensor data, can use them by accessing to Orion and/or to CKAN by

using NGSI and/or CKAN API respectively. Experimenters can use Wirecloud to create their own web

application using sensor data stored on Orion and on CKAN.

Components can be accessed via the following URLs:

• Orion: http://gw.jose.jp:4326 (at this moment, provided instance of Orion can only be accessed from

the available instance of Wirecloud. This issue will be fixed in the future)

• CKAN: http://ckan.festival.ckp.jp

• Wirecloud: http://wirecloud.festival.ckp.jp

45




• MQTT: mqtt://festival.ckp.jp:8883

3.2.5. User access controls of FIWARE in JOSE

For components involved in the testbed, controls on user access are configured as reported in the

following list:

• MQTT broker: username/password based authentication

o MQTT broker, mosquitto, has its own username and password list

• CKAN: username/password based authentication

o CKAN user can be created by accessing provided CKAN instance (http://ckan.festival.ckp.jp)

• Wirecloud: username/password based authentication

o Wirecloud user can be created by accessing provided Wirecloud instance

(http://wirecloud.festival.ckp.jp)

• Orion: host based authentication, at this time. Only applications on Wirecloud at CKP can connect to

the Orion. This is JOSE restriction and will be resolved.

Figure 27 shows user access controls of the FIWARE in JOSE.

Figure 27: User access controls of FIWARE in JOSE

46




3.3. Next steps

Next planned steps for integration of FIWARE’s GEs and JOSE platform are:

• Integration of PEP proxy and IDM (Identity Management) from FIWARE GE to add user management

capabilities to NGSI part of the testbed. Deploy user based authentication instead of host based

(Figure 28)

o When user based authentication will be prepared, in addition to the Wirecloud

authentication, it will be possible to access to Orion from any network locations.

Figure 28: Integrate PEP Proxy and IDM into FIWARE in JOSE

• Build new IaaS testbed using OpenStack in JOSE; the new IaaS testbed will be federated under

FESTIVAL’s federation layer. In particular, the JOSE management body decided to let their physical

machines to be used by FESTIVAL project: in this way an OpenStack environment will be provided

instead of JOSE native KVM by using JOSE physical machines. Connectivity to this OpenStack

environment will be provided by CKP (see section 5.1. ).

47




4. Open Data Federation: integration with FESTIVAL Open

data testbeds

Open Data Federation aims to federate Open Data repositories and portals (from now on named

ODMS: Open Data Management Systems) in order to provide a unique access point to the resources

they provide (Open Data and Linked Open Data). Open Data Federation is achieved through FODP

(Federated Open Data Platform). FODP is able to retrieve, search and visualize datasets from different

ODMS nodes.

Figure 29 depicts the general architecture of FODP.

Figure 29: Federated Open Data Platform - General architecture

48




Main components of FODP are:

• Federation Manager: it is the central component of FODP; it provides functionalities for;

o managing federated ODMS nodes;

o searching Open Data;

o querying Linked Open Data;

o managing configuration of the platform itself.

• Platform API: directly connected to Federation Manager, this component exposes RESTful APIs in

order to allows Federated Open Data Catalogue and or external tools or system to interact with

functionalities of Federation Manager.

• LOD Repository: it is the repository in which Linked Open Data retrieved from federated ODMS nodes

are stored.

• Federated Open Data Catalogue: it is a web application providing the GUI (Graphical User Interface)

for the functionalities of FODP to end users.

For interacting with federated ODMS nodes, Federation Manager uses specific software modules

(connectors), which represent a bridge between itself and federated ODMS nodes. Each module

provides a common set of methods and it is tailored on a precise class of ODMS nodes. From a technical

point of view, these modules are based on a Java interface, named ODMS Connector. Moreover, it is

important to underline that Federation Manager does not interact as a whole with connectors; as

described in D2.3 [2], only two components of Federation Manager can interact with connectors:

“Federation Core” and “OD/LOD Federated Search”. Details about ODMS Connector and its

implementations are provided in section 4.1.

4.1. ODMS Connectors

ODMS Connectors allow the Federated Open Data Platform (FODP) to interact with existent Open Data

systems (namely ODMS nodes) in order to retrieve metadata of their datasets through their specific

API and to mapping these metadata into a uniform format represented by a subset of DCAT vocabulary

properties. This mapping is useful in order to solve semantic ambiguities of metadata fields retrieved

from different ODMS nodes. Moreover, ODMS Connectors provide functionalities for performing live

searches on federated ODMS nodes.

ODMS Connectors implement a specific Java interface (ODMSConnector) in order to ensure

compliance with internal processes of FODP. ODMSConnector interface provides five public methods

for interacting with ODMS nodes:

• findDatasets: provides search capabilities in order to find specific datasets on a federated ODMS

node.

• countDatasets: provides functionalities for retrieving the number of available dataset from a

federated ODMS node.

• getDataset: provides functionalities for retrieving a specific dataset from a federated ODMS node.

49




• getAllDatasets: provides functionalities for retrieving all available datasets from a federated ODMS

node.

• getChangedDatasets: provides functionalities for retrieving new and modified datasets from a

federated ODMS node.

In Erreur ! Source du renvoi introuvable. Java code of ODMSConnector interface is reported.

Table 2: OMSConnector Java code

package it.eng.rspa.odf.connectors; import it.eng.rspa.odf.beans.ODMSNode; import it.eng.rspa.odf.dcat.DCATDataset; import java.util.HashMap; import java.util.List; public interface ODMSConnector { public List<DCATDataset> findDatasets(HashMap<String,Object> searchParameters) throws Exception; public int countDatasets() throws Exception; public DCATDataset getDataset(String datasetId) throws Exception; public List<DCATDataset> getAllDatasets() throws Exception; public HashMap<DCATDataset, String> getChangedDatasets(List<DCATDataset> oldDatasets, String startingDate) throws Exception; }

FODP provides two implementations (i.e. two connectors) of this Java interface, in order to support

natively open data portals based CKAN (CKANConnector) and Socrata (SocrataConnector). A third

implementation, named “OpenDataFederationNativeConnector” (ODFNConnector) is provided in

order to support “custom” ODMS nodes. In order to become part of the federation, custom ODMS

nodes are requested to implement and expose a specific set of API (Federation API).

Figure 30: Federation Manager and ODMSConnectors

Following sections provides technical details about implementation and internal operations of

CKanConnector, SocrataConnector and OpenDataFederationNativeConnector.

For more details about Federated Open Data Platform, refer to the FESTIVAL deliverable D2.3 Open

data guidelines and federated testbed policy [2].

ODMS

AP

ODMS

AP

ODMS

AP

ODMS

FEDERATION

API

Federation Manager

ODMS

AP

CKAN

ODMS

AP

CKAN

ODMS

AP

SOCRATA

CKANCon

SocrataCo

ODFNCon

50




51




4.1.1. CKANConnector

CKANConnector provides functionalities for interacting with CKAN based ODMS. It is built on top of a

customised version of CKAN_Client-J [47], a client for CKAN developed in Java. CKAN_Client-J allows

performing remotely basic operations on CKAN portals, such as adding, updating, deleting and

searching datasets. Moreover, CKAN_Client-J has been enriched in order to support:

• sorting and max number of results for search functionality;

• identification of recently deleted, modified and added datasets;

• retrieving the IDs of all available datasets.

CKANConnector provides functionalities for both searches: live or on cached data.

Figure 31: CKANConnector - General schema

Following sections describe the interaction between Federation Manager and CKANConnector and its

internal processes during the registration of a CKAN ODMS (section 4.1.1.1), the synchronization of

the internal cache of FODP with its datasets (section 4.1.1.2) and the execution of a live search on it

(section 4.1.1.3).

4.1.1.1. Registration of a CKAN ODMS

Subsequently the request for federating a CKAN ODMS, the Federation Manager checks the information

about the CKAN ODMS itself and then requires CKANConnector to check the node availability through the

count of its available datasets. The CKANConnector returns the count of available datasets, confirming

availability of the CKAN ODMS. Federation Manager registers the data of the CKAN ODMS and then starts

the first synchronization of its datasets. Federation Manager requests all available datasets to

CKANConnector; CKANConnector executes a search on federated CKAN ODMS: through the methods

“findDatasets” and “Create query”, it prepares a specific query and then requires CKAN_Client-J to execute

it. CKAN_Client-J forwards the query to CKAN ODMS that provides its available datasets. CKAN_Client-J

forwards the datasets to CKANConnector. For each dataset, CKANConnector performs a translation to DCAT

format and then provides the datasets in DCAT format to Federation Manager. Federation Manager registers

metadata of each dataset and if a dataset has associated RDF resources, Federation Manager downloads and

registers it.

Federation Manager

OD

AP

CKAN

Federation

Manager Internal

Compone

CKANConnector CKAN_Client-

52




Figure 32 depicts the sequence diagram that illustrates the steps involved during the registration of a CKAN

ODMS.

Figure 32: CKANConnector sequence diagram - registration of a CKAN ODMS

53




4.1.1.2. Datasets synchronization of CKAN ODMS

Periodically Federation Manager performs the synchronization of its internal cache of metadata with

metadata of available datasets of CKAN ODMSs. For each federated CKAM ODMSs, Federation Manager

retrieves previously cached datasets. After that, Federation Manager requests CKANConnector to provide

the modified datasets of the federated CKAN ODMS. CKANConnector forwards the request to CKAN-Client-J

that interacts with the federated CKAN ODMS in order to obtain the modified datasets. CKAN-Client-J

provides the modified datasets to CKANConnector, then CKANConnector translates the dataset in to DCAT

format. After that CKANConnector provides datasets to Federation Manager. For each dataset, Federation

Manager verifies if it is a new added dataset, if it is an updated dataset or if it is a deleted one, and performs

the corresponding operation.

Figure 33 depicts the sequence diagram that illustrates the steps involved during the synchronization of a

CKAN ODMS.

Figure 33: CKANConnector sequence diagram - Synchronization of a CKAN ODMS

54




4.1.1.3. Live search on a CKAN ODMS

If a live search involves a CKAN ODMS, Federation Manager requires CKANConnector to find datasets

corresponding to search criteria specified by the user (for details about this aspect refer to D2.3 [2]).

CKANConnector build a query on the basis of the search criteria and requires CKAN_Client-J to execute it.

CKAN_Client-J forwards the query to the CKAN ODMS that provides datasets corresponding to the executed

query. CKAN_Client-J forwards obtained datasets to CKANConnector. After that CKANConnector translates

datasets to DCAT format and then provides them to Federation Manager.

Figure 34 depicts the sequence diagram that illustrates the steps involved during a live search on a CKAN

ODMS.

Figure 34: CKANConnector sequence diagram – Live search on a CKAN ODMS

55




4.1.2. SocrataConnector

SocrataConnector provides functionalities for interacting with Socrata based ODMS. Unlike the

CKANConnector, SocrataConnector is not based on a java client for Socrata; for technical reasons, it is

based on a REST API provided by Socrata based portals. This API (/api/data.json) provides a JSON

containing data of all available datasets. SocrataConnector makes use of an HTTP Client in order to

consume this API. For this reason, SocrataConnector supports only cache capabilities. Live search is

not supported.

Figure 35: SocrataConnector - General schema

Following sections describe the interaction between Federation Manager and SocrataConnector and

its internal processes during the registration of a Socrata ODMS (section 4.1.2.1) and the

synchronization of the internal cache of FODP with its datasets (section 4.1.2.2).

4.1.2.1. Registration of a Socrata ODMS

Subsequently the request for federating a Socrata ODMS, the Federation Manager checks the information

about the Socrata ODMS itself and then requires SocrataConnector to check the node availability through

the count of its available datasets. The SocrataConnector internally invokes getAllDatasets method and then

getJSONDataset method. This last method invokes the API “/api/data.json” of the Socrata ODMS. Socrata

ODMS provides the JSON data of available datasets. SocrataConnector translates the data in to the DCAT

format and after that it returns the count of available dataset to Federation Manager.

Federation Manager registers the data of the Socrata ODMS and then starts the first synchronization of its

datasets. Federation Manager requests all available datasets to SocrataConnector; SocrataConnector invokes

getJSONDataset method and then the API “/api/data.json” of the Socrata ODMS. Socrata ODMS provides

JSON data of available datasets to SocrataConnector. After that SocrataConnector translates the data in to

the DCAT format and then provides the datasets in DCAT format to Federation Manager. Federation Manager

registers metadata of each dataset and if a dataset has associated RDF resources, Federation Manager

downloads and registers it.

Figure 36 depicts the sequence diagram that illustrates the steps involved during the registration of a Socrata

ODMS.

Federation Manager

OD

AP

Socrata

Federation

Manager Internal

Components

SocrataConnector HTT

P

56




Figure 36: SocrataConnector sequence diagram - Registration of a Socrata ODMS

57




4.1.2.2. Datasets synchronization of Socrata ODMS

As for CKAN ODMSs, Federation Manager performs the synchronization of its internal cache of metadata

with metadata of available datasets of Socrata ODMSs at periodic time interval. For each federated Socrata

ODMSs, Federation Manager retrieves previously cached datasets. After that, Federation Manager requests

SocrataConnector to provide the modified datasets of the federated Socrata ODMS. SocrataConnector calls

its internal method getAllDatasets that in turn calls getJSONDatasets method. This last method invokes the

API “/api/data.json” of the Socrata ODMS. Socrata ODMS provides the JSON data of available datasets.

SocrataConnector translates the data in to the DCAT format and after that it identifies new datasets (added

datasets), datasets that have been removed on the federated Socrata ODMS (deleted datasets) and updated

datasets. At the end of this process, SocrataConnector returns the list of modified datasets to Federation

Manager. For each dataset, Federation Manager verifies if it is a new added dataset, if it is an updated dataset

or if it is a deleted one, and performs the corresponding operation.


CKAN ODMS.

58




Figure 37: SocrataConnector sequence diagram - Synchronization of a Socrata ODMS

59




4.1.3. OpenDataFederationNativeConnector

OpenDataFederationNativeConnector (ODFNConnector) provides functionalities for interacting with

custom ODMS. It provides an implementation of ODMSConnector Java interface. ODFNConnector

interacts with custom federated ODMS that expose API following the Federation API specifications.

Custom ODMSs that want to join the federation have to expose Federation API through a RESTful

interface. Federation API provides different sets of method for searching datasets (Live Search) and

for synchronizing metadata of datasets (more details about Federation API specifications are provided

in section 4.2. ). On the basis of the implemented set of method on the federated custom ODMS,

Federation Manager invokes the correct method of ODFNConnector. ODFNConnector makes us of a

custom Java client, named Native Client, which in turn is based on HTTP Client, in order to consume

Federation API provided by custom federated ODMSs. This allows to separate programming logic and

to improve code maintainability.

Figure 38: OpenDataFederationNativeConnector - General schema

Following sections illustrate interaction between Federation Manager and ODFNConnector and its

internal processes during the registration of a custom ODMS (section 4.1.3.1), the synchronization of

the internal cache of FODP with its datasets (section 4.1.3.2) and the execution of a live search on it

(section 4.1.3.3).

4.1.3.1. Registration of a custom ODMS

Subsequently the request for federating a custom ODMS that provides dataset cache capabilities, the

Federation Manager checks the information about the custom ODMS itself and then requires

ODFNConnector to check the node availability through the count of its available datasets. The

ODFNConnector forwards the request to Native Client that requires the custom ODMS to provide the

available datasets. Native Client provides the datasets to ODFNConnector. ODFNConnector returns the

number of available datasets, confirming availability of the custom ODMS. Federation Manager registers the

data of the custom ODMS and then starts the first synchronization of its datasets. Federation Manager

requests all available datasets to ODFNConnector; ODFNConnector forwards the request to Native Client that

requests the custom ODMS to provide its available datasets. The custom ODMS provides the dataset in DCAT

format to Native Client and it forwards the dataset to ODFNConnector. ODFNConnector provides the dataset

FEDERATION API

Federation Manager

OD

Custom

Federation

Manager Internal

Components

ODFNConnecto

r

Native

ClieHTTP

60




to Federation Manager. Federation Manager registers metadata of each dataset and if a dataset has

associated RDF resources, Federation Manager downloads and registers it.

Figure 39 depicts the sequence diagram that illustrates the steps involved during the registration of a custom

ODMS.

Figure 39: ODFNConnector sequence diagram - Registration of a custom ODMS

4.1.3.2. Datasets synchronization of custom ODMS

At a periodic time interval, Federation Manager performs the synchronization of its internal cache of

metadata with metadata of available datasets of custom ODMSs that provides dataset cache capabilities. For

each of them, Federation Manager retrieves previously cached datasets. After that, Federation Manager

requests ODFNConnector to provide the modified datasets of the federated custom ODMS. ODFNConnector

forwards the request to Native Client that interacts with the federated custom ODMS in order to obtain the

modified datasets. Native Client provides the modified datasets to ODFNConnector, and then

ODFNConnector identifies new, updated and deleted datasets. After that ODFNConnector retrives metadata

of datasets through Native Client; Native Client request custom ODMS to provide metadata of datasets. The

custom ODMS provides metadata in DCAT format to Native Client and Native Client provides them to

ODFNConnector. After that, ODFNConnector provides datasets to Federation Manager. For each dataset,

61




Federation Manager verifies if it is a new added dataset, if it is an updated dataset or if it is a deleted one,

and performs the corresponding operation.


CKAN ODMS.

Figure 40: ODFNConnector sequence diagram - Synchronization of a custom ODMS

62




4.1.3.3. Live search on a custom ODMS

If a live search involves a custom ODMS, Federation Manager requires ODFNConnector to find datasets

corresponding to search criteria specified by the user (as for CKANConnector, for details about this aspect

refer to D2.3 [2]). ODFNConnector builds a query on the basis of the search criteria and requires Native Client

to execute it. Native Client forwards the query to the custom ODMS that provides datasets corresponding to

the executed query; in this case provided datasets are compliant to the DCAT format. Native Client forwards

obtained datasets to ODFNConnector. After that ODFNConnector instantiate for each datasets a

corresponding object representing the datasets itself and then provides them to Federation Manager.

Figure 41 depicts the sequence diagram that illustrates the steps involved during a live search on a custom

ODMS.

Figure 41: ODFNConnector sequence diagram – Live search on a custom ODMS

4.2. Federation API

This section provides a more detailed description of Federation API specification that was introduced

in D2.3 [2]: through the implementation and exposure of Federation API, custom ODMs can be

federated by Open Data Federation. The Federation API specification is described using RAML (RESTful

API Modelling Language) [48], a language for describing RESTful APIs easy readable from both humans

and machines. Description of Federation API was made following version 0.8 of RAML specification.

Furthermore, RAML is supported by a wide range of developer and different open source tools are

available for managing RAML documents, such as tools for automatic generation of code and

documentation.

63




Federation API are organised in two sets: Live Search (providing search capabilities on the custom

ODMS) and Metadata Cache (providing capabilities on the custom ODMS enabling Open Data

Federation to gather metadata about datasets available on the ODMS itself).

Live Search set consists in one method for searching datasets on the custom ODMS:

“/odf/odms/search”. It accepts an HTTP POST request and in input a JSON string representing the

search parameters; these parameters are:

• query: the query for searching datasets; the query is composed of one or more keywords;

• sort: criteria for sorting obtained results;

• rows: the maximum number of results returned;

Possible responses are:

• HTTP 500 if an error occurs; in this case the service returns a JSON string containing information

about the error; in particular the following field are represented:

o errorMessage, providing textual information about the error;

o errorCode, providing the code of the occurred error; in this particular case the provided code

is 500.

• HTTP 200 if the request is correctly executed; in this case the service returns a JSON string

representing collected results containing two main field:

o count: providing the total number of returned results;

results: an array containing the obtained results: the datasets that satisfy the search criteria;

an example of element of this array in reported in

o Table 3.

64




Table 3: Example of JSON representation of a dataset

{

"identifier": "A unique identifier of the dataset",

"spatial": "Spatial coverage of the dataset",

"keyword": [

"pollution",

"metro"

],

"issued": "2015-10-12T12:03:22Z",

"landingPage": "baseUrl/datasets/identifier",

"publisher": {

"name": "The entity responsible for publishing the dataset",

"mbox": "Publisher Mail box",

"homepage": "Publisher homepage",

"type": "Publisher Type"

},

"modified": "2015-10-12T12:03:22Z",

"title": "title of the dataset",

"accrualPeriodicity": "The frequency at which the dataset is updated and published",

"description": "text description of the dataset",

"temporal": "The temporal period that the dataset covers",

"versionNotes": "Description of changes between this version and the previous one",

65




"contactPoint": {

"fn": "Contact Full Name",

"hasEmail": "Contact Mail Box"

},

"language": "dataset language",

"distribution": [

{

"title": "Distribution Title",

"accessURL": "Distribution URL ",

"description": "Distribution description",

"mediaType": "The media type of the distribution as defined by IANA",

"format": "The file format of the distribution",

"license": "Link to license document under the which the resource is made available",

"issued": "2015-10-12T12:03:14Z",

"modified": "2015-10-12T12:03:14Z",

"byteSize": "1024"

}

]

}

Metadata Cache set consists of three methods for retrieving datasets on the custom ODMS:

• "/odf/odms/datasets/{id}": this method provides the metadata under the form of a JSON string of a

specific dataset; it accepts HTTP GET requests. The “id” parameter of the method is the identifier of

the requested dataset. Possible responses are:

HTTP 200 if the request is correctly executed; in this case the service returns a JSON string

representing the requested dataset; an example of JSON representation of a dataset is provided in

66




o Table 3.

o HTTP 500 if an error occurs; as reported for “/odf/odms/search” the service returns a JSON

string containing information about the error: errorMessage field provides the textual

description of the occurred error and errorCode field reports the error code 500.

o HTTP 404 if the required dataset is not available; as described for HTTP 500 response, the

service returns a JSON string containing information about the error; in this case errorCode

field reports the value 404.

• "/odf/odms/datasets/info": this method provides basic information about dataset available in the

ODMS that expose the Federation API; it accepts HTTP GET requests. Possible responses are:

o HTTP 200 if the request is correctly executed; in this case the service returns a JSON string

containing an array which elements represent available datasets, their issued date and

modified date; an example of element of this array is provided in Table 4.

Table 4: Example of JSON representation basic information of a dataset

{ "issued": "2015-10-12T12:03:14Z", "identifier": "unique numeric identifier of the dataset", "modified": "2015-10-12T12:03:14Z" }

o HTTP 500 if an error occurs; as described for previous methods, service returns a JSON string

containing information about the error, where errorMessage field provides the textual

description of the occurred error, while errorCode field provides the value 500.

• "/odf/odms/datasets": this method provides the complete description of datasets available in the

ODMS; it accepts HTTP POST requests containing two input parameters providing pagination

features: rows (representing the number of dataset returned per page) and offset (representing the

requested page). Possible responses are:

HTTP 200 if the request is correctly executed; in this case the service returns a JSON string

containing an array which elements represent provided datasets; an example of dataset is provided in

67




o Table 3.

o HTTP 500 if an error occurs; the service returns a JSON string containing information about

the error: in the field errorMessage the textual description of the occurred error is provided,

while in the field errorCode the code corresponding to the occurred error, in this case 500.

The complete technical description of Federation API in RAML is reported in Annex 1: Federation API – RAML description.

4.3. Federated ODMS

This section provides details about the status of the federation of the Open Data Management System

that participate as testbeds in FESTIVAL project.

4.3.1. Santander Datos Abiertos

As described in D1.2 [49], municipality of Santander provides an ODMS for publishing Open Data

(Santander Datos Abiertos) [50] about the city itself and related to the following topics:

• transport

• urban planning and infrastructures

• shops

• demography

• society and well-being

• culture and leisure.

From a technological point of view, the ODMS is based on Wordpress (for implementing the graphical user

interface), CKAN (for meta-dating and managing datasets and resources and for supplying APIs in order to

allow developers to consume data), Virtuoso [51] (for proving Semantic Web capabilities), iCMS (a data

gathering system for proving updated data to the portal) and Marmota/Neogolsim (for proving the SPARQL

engine).

Thanks to the presence of CKAN and to the exposure of its APIs, this ODMS is supported by Open Data

Federation (Open Data Federation supports natively CKAN ODMS). Moreover, ODMS of Santader played a

key role during the development of CKANConnector, because it and other CKAN based ODMS publically

available (such as FIWARE Lab Open Data portal [52] and CKAN node for FESTIVAL in JOSE testbed [53]) was

used for testing the development of the connector.

68




4.3.2. Greater Lyon Data

As reported in D1.2 [49], Greater Lyon Data [54] is a platform for providing Open Data relative to Lyon

city and about a wide range of sector, such as:

• land register

• metropolitan area

• greenery

• shared bikes (Vélo´V)

• automatic car-sharing stations

• real-time traffic

• highway events

• traffic history

• aerial photographs.

In this case, this ODMS is based on a proprietary technology, so it represents a “custom ODMS”. In

order to be federated it has to implement and expose Federation API (described in section 4.2. ). The

implementation of the federation API for Greater Lyon Data is an ongoing activity and will be

completed in the first period of 2016.

4.3.3. FESTIVAL Japanese Open Data Platform

This platform [55] will be used to collect and share the information of the experiments and applications

deployed in Japan. The information that will be collected during the experiments performed using the

Japanese testbeds involved in FESTIVAL will be stored in this platform based on CKAN that exposes

RESTful API.

4.3.4. FIWARE lab data portal

FIWARE Lab Open Data portal collects Open Data from a various cities and territories in Europe collected

using Generic Enablers through the FIWARE-Lab. This information can be considered very interesting for

experimenters that want to use it in the Smart Cities context. FIWARE Lab Open Data portal is based on CKAN

Generic Enabler.

69




5. Testbeds access integration

5.1. Exposing JOSE VMs to the Internet

As already discussed in D1.2 [49] and D1.3 [1], JOSE plays an important role in FESTIVAL as

Infrastructure as a Service testbeds. While JOSE testbed is operated by NICT, VMs (Virtual Machines)

are basically operated by the experimenters of JOSE. To keep the security of resources of VMs, the

security policy of NICT requires that when JOSE VMs are connected to the internet, they must have

assigned IP address associated to the relative experimenter and the security incidents must be handled

by the experimenter. To establish such environment, Cyber Kansai Project (CKP) provides VMs in the

CKP Umekita network testbed. Figure 42 shows the current solution. Access from outside networks is

authenticated and controlled at CKP VM node. Any traffic exchanged from JOSE VMs to the Internet

has source IP address from CKP network. It satisfies NICT’s requirement and enables us to access JOSE

testbed from any node in the Internet safely.

Figure 42: Exposing JOSE VMs to the Internet

In the current implementation, Nginx [56] is used as a HTTP Proxy to support Web Service and REST

API exposure. A sub domain of “festival.ckp.jp” is provided for each service (e.g.:

sensinact.festival.ckp.jp, ckan.festival.ckp.jp, and so on). Figure 43 shows sequence diagram of the

proxy service.

70




Figure 43: Exposing JOSE to the Internet (Web Service/REST API)

For messaging service (e.g.: MQTT broker) Proxy Message Broker is introduced at CKP VMs. The

sequence diagram is shown in Figure 44.

Figure 44: Exposing JOSE to the Internet (sensors/actuators)

5.1.1. Next step

Since sub domain based service proxy is easy to understand for experimenters and end-users to

identify services, it is desirable to provide it as a testbed function. However, since it is difficult to allow

dynamic service proxy in the current implementation, it requires some delay to configure a new service

proxy. The dynamic service proxy should be handled in developing IaaS service based on JOSE as future

work. The security policy must be also discussed there.

71




5.2. ACL for FIAPStorage for JOSE

5.2.1. General Descriptions

FIAPStorage for JOSE is one of the reusable components provided by JOSE testbed and it is an

extension of FIAPStorage, the storage component for the standard IEEE1888 protocol. FIAPStorage for

JOSE is also an integrated component built from plain FIAPStorage, its wrapper module and extension

module are implemented as a PIAX agent program.

The PIAX extension module provides the following functions:

• Host sensor agents to be found by PIAX conditional queries.

o Sensor agents are Agent programs of the PIAX.

o Sensor agents will be created per sensors triggered by receiving sensor data from sensors.

o Sensor data from sensors will update data in the sensor agents.

o Sensor agents reply to PIAX conditional queries if conditions are matched.

o Replies of sensor agents could contain sensor data from FIAPStorage.

• Provide RESTful interface to issue PIAX conditional queries to find and fetch sensor data.

o User can issue conditional queries by accessing RESTful interface and get replies from sensor

agents as HTTP replies.

Despite in principle the RESTful interface of the FIAPStorage for JOSE has functionality to expose all

data in FIAPStorage, it has no user access control. So any user can issue PIAX conditional queries and

fetch any sensor data from original FIAPStorage for JOSE without authentication. Next section (5.2.2)

provides details about adding access control list (ACL) functionality to FIAPStorage for JOSE.

5.2.2. Implementation of ACL for FIAPStorage for JOSE

Component diagram of the original FIAPStorage for JOSE is depicted in Figure 45. Following numbered

list describe how user can get data of sensors from the current version of the software:

1. When sensors send their data to FIAPStorage wrapper (at center of the diagram), wrapper

requests to RESTful peer to create sensor agents.

2. RESTful peer (at upper middle of the diagram) spawns sensor agents on requests.

3. User (at top right of the diagram) can issue PIAX conditional queries over PIAX overlay network

by accessing RESTful peer (middle of the diagram) via RESTful interface (middle right) without

authentications, fetching sensor data from FIAPStorage (at bottom of the diagram) or

FIAPStorages included in other FIAPStorages for JOSE (at upper left of the diagram).

72




Figure 45: Original FIAPStorage for JOSE

To be a more secure system, the user access control list function has been added to the RESTful interface. As

a first step, the system is dependent on authentication mechanisms of PIAX Testbed [57]. So to issue PIAX

queries to RESTful peer at this time an account of the PIAX Testbed is needed. Figure 46 shows the component

diagram of the FIAPStorage for JOSE with ACL. The RESTful peer of this version of the FIAPStorage for JOSE

implements the following procedure to accept and reply PIAX query requests:

1. RESTful peer accepts PIAX query requests with “API key”. “API key” is a character string that

users can obtain from user account manager of PIAX Testbed. So users need accounts of the

PIAX Testbed.

2. RESTful peer asks user account manager of the PIAX Testbed who is the owner of the “API key”

string.

3. RESTful peer issues PIAX query made from RESTful request with owner’s name of the API key.

4. Each sensor agent on the PIAX overlay network consults its relative ACL to decide whether to

reply to the query or not.

73




5. Original user will get replies from sensor agents that have user name in “authorized user list” of

the ACL and public sensor agents that allow any users to get their sensor data.

Figure 46: FIAPStorage for JOSE with ACL

5.2.3. Next steps

The next steps that will be performed are:

• Implement user account database to be independent from PIAX Testbed.

• Support OAuth 2.0 to allow users to use standard procedure to get access authorizations.

Figure 47 shows the system with above extensions. RESTful peer of this version will include its own

user database instead of using PIAX Testbed’s account system. The RESTful peer will consult the user

database when authentication is needed. The RESTful peer will also provide OAuth 2.0 authorization

74




(at upper right of the diagram). Therefore, users in the user database can authorize their applications

to access to RESTful interface with their privileges via the OAuth 2.0 interface.

Figure 47: FIAPStorage for JOSE with OAuth2.0

75




6. Future implementation plan

The finalization of the FESTIVAL Architecture, in order to start the first phase of the experimentation,

will be achieved by means of two main macro-activities:

1. Finalization of Resource-based Federation.

2. Implementation of Experiment-based Federation.

The first point includes all the activities needed to finalize, for each resource category (Open Data,

IoT, IT and Living Lab), the federation at the first level, that has the objective to describe each resource

category in a single way to the higher level of the architecture; for each category, the work already

completed and those still being finalized is reported in this deliverable.

For the Open Data category most of integration activities are completed; from the architecture side

the components developed are:

• The Open Data Federation Platform.

• The Open Data Federation Connectors to interact with CKAN ODMS and SOCRATA ODMS.

• The Open Data Federation Native Connector to interact with custom ODMS and Federation API.

In this sense, the development activities are almost completed.

From the Testbed side:

• Santander City has an ODMS (Santander Datos Abiertos) based on CKAN, so its federation in ODF

Platform is already available.

• Lyon City has custom ODMS (Greater Lyon Data), so the next activity will be the implementation of

the ODF Federation API in order to achieve its federation.

Once both testbeds are federated, for the first phase of experimentation it will be possible to use their

datasets in the experiments.

For the IoT category the integration activities aimed to connect the Festival IoT Testbeds to the

SensinAct platform, the IoT gateway responsible to achieve the Resource-IoT federation. From the

architecture side, the components developed are:

• KNX Bridge (testing phase)

• Orion Bridge

• Echonet Bridge

• OpenHab Bridge

76




Moreover, some investigations were performed with the objective to experimenting integration

between messaging, storage and visualization platform and for integrating Sensors, MQTT and Time

Series Database. These investigations will be carried on in the months in order to satisfy further

requirements, such as anonymizing process, integration with HTTP plugin for CKAN, etc.

For the IT category, the integration activities were focused on the work done on testbeds side. In this

sense, the experiment reported in the previous section of this document demonstrated the feasibility

of the integration of European software components (Generic Enablers from FIWARE) on the Japanese

IT platform (JOSE).

The ongoing activities related to this category are related to the definition of the technical architecture

for the IT Manager component. The most suitable solution seems to be the one adopted in Fed4Fire

project [4], which bases his architecture on the concept of Slice-based Federation. Reusing the

approach and technologies of Fed4Fire in FESTIVAL can be a good opportunity to capitalize the already

achieved results as described in D1.2 [49].

The implementation activities will focus mainly in the development of a centralized component, the

IT Resource Manager Core, that:

• manages communications with the testbeds using the Federation Aggregate Manager (AM) API, a

set of API defined in Fed4Fire project.

• manages the description of resources by using RSpec language.

• provides to the higher architecture levels a unique model (based on RSpec language) which describes

a generic IT resource.

From the testbeds side, the Fed4Fire federation can be achieved implementing an Aggregate Manager,

a component that performs the discovering and the provisioning of resources by means the Federation

AM API.

Fed4Fire’s approach makes easier the federation of IT resources thanks to a set of tools and libraries

that allow the integration of the testbeds based on specific infrastructures. Two possible open source

solutions that could help the IT federation objective are:

• SFAwrap [58]: a free software that supports SFA (Slice based Federation Architecture) architecture;

it requires a set of API exposed in the testbeds for interacting with them and for allowing the

implementation of wrappers for specific testbeds.

• FITeagle [59]: a toolkit for federating infrastructures compatible with the Fed4Fire Aggregate Manger

API.

Both tools provide connectivity functionalities to testbeds based on Open Stack infrastructure. This

feature is very important in Festival context, because FIWARE Lab is already based on Open Stack and

JOSE is working for implementing an Open Stack infrastructure on its platform, exposing its virtual

machines using it.

77




Therefore, the other integration activities apart the development of the IT Resource Manager Core,

that will be conducted are:

• The adoption of one of the tools listed above, implementing specific connector to Open Stack

infrastructure.

• The development of Aggregate Manger on the Open Stack infrastructure of each testbed.

The activities regarding the Living Lab category will be focused on two parts:

• Development of a centralized component, the Living Lab Manager Core.

• Development of a connector for federating each Living Lab testbed (LL Connector).

The LL Manager Core will provide the following functionalities:

• Provides a unique description of Living Labs to the higher level of the Festival architecture.

• Manages interactions between the experiment and the Living Labs, providing:

o Automatic interactions for specific activities, as the discovery of resources and services or

generic assets provided by Living Labs, in the configuration phase of an experiment.

o Manual interactions when the experiments include manual tasks involving the human

resources of the living lab; these activities can’t be managed fully automatically but they

need asynchronous interactions.

For instance, the organization of a focus group is composed of different steps in which the experiment

requires communication with a member of the involved Living Lab responsible for the specific activity;

each step has different features and could delay depending on involved end-users. The LL Connector

provides the tools that an experimenter can use for communicating with Living Labs during an

experiment; the communication can be performed by using several means (e.g.: Email, Notification

System, Form, Survey, etc). The experimenter can choose the communication system more suitable to

the task in execution, depending on the availability of the specific Living Lab involved in the

experiment.

In order to achieve the federation objective, each Living Lab will provide a representation of its

activities in terms of services or assets; for this reason, the Living Lab Manager Core will provide:

• A Registration Section containing all the fields needed to describe a Living Lab.

• A Living Lab Repository that provides to the experimenters an overview of the Living Labs federated

in Festival.

Moreover, each Living Lab will provide a system that allows to manage incoming and outcoming

messages exchanged with the experimenters.

The second macro-activity to be performed in the next months is the implementation of the second

level of the Festival Architecture: Experiment-based Federation.

78




This activity includes the development of two pieces of FESTIVAL’s architecture:

• FESTIVAL Uniform Access Layer.

• FESTIVAL EaaS Layer.

For the first one, the integration activities will regard:

• The final definition of a FESTIVAL Resource Model.

• The FESTIVAL Uniform API that will allow the EaaS layer to manage the resources from the four

resource aggregators (Open Data Federation, IoT Gateway, IT Resources Manager and Living Lab

Manager).

For the EaaS layer, the integration activities will regard the development of the sub-components

shown in the Figure 1: Experiment Control, Experiment Scheduling, Experiment Monitoring, Resource

Data Broker, Storage Service, Business Façade, Resource Discovery and Resource Access Manager.

In order to achieve the second federation level, the approach that will be followed will not be only

bottom-up but also top-down: starting from the definition of the main concepts and features of

Festival Portal (which are functionalities used by experimenters) and from the definition of FESTIVAL

EaaS Layer, the contact point between these two modules will be identified and the development will

be based on it.

79




7. Conclusion

This deliverable reported the technical details and the status of the components developed as reference

implementation of the FESTIVAL platform. In this document it has been presented an overview of the

activities performed related to different aspects of the testbed federation. In particular in the first 14 months

of the project the attention was paid on the achievement of the “Resource based federation” that has the

aim of federating the testbed resources of four different categories: Open Data, IoT, IT and Living Labs. The

different chapter of the deliverable presented the software artefacts related to these categories.

In the chapter 2 it has been reported the status of the IoT testbed integration that is achieved through the

use of SensiNact gateway and a set of bridges to support specific IoT protocols to federate the involved

testbed (i.e. Smart Santander, PTL and iHouse).

In the chapter 3 it has been presented the concrete integration between FIWARE Generic Enabler and the

JOSE testbed, demonstrating the possibility to reuse the FIWARE technology on different execution

environment in order to provide them as a key resource for the FESTIVAL EaaS model.

The chapter 4 showed the integration related to the Open Data testbeds. While the general technical details

about Federated Open Data Platform has been included in D.23 “Open data guidelines and federated testbed

policy” in this document were described the functions and internal structure of open data connectors that

allows the federation of the Open Data testbeds, but also external open data sources that can be interesting

for an experimenter.

During the first year of the FESTIVAL project it have been tackled different technical problems related to

interoperability and security of European and Japanese testbeds: in the chapter five were reported some of

the performed activities devoted to overcome these issues allowing, for instance, the secure access of the

JOSE VM to the internet or the implementation of an access control list for the FIAPStorage component.

The last part of the deliverable presented a plan for the next components implementation: in particular,

while the Open Data and IoT federation can be considered in advanced status, the IT resource federation and

the Living Lab one needed further technical work to be done in the next months: in particular for the IT

resources federation it has to be implemented, starting from existent identified components, a connector for

OpenStack to support Fed4Fire specification. From the Living Lab side, the next months will be crucial to

identify the service and specific resources that every Living Lab can provide to support the FESTIVAL

experiment.

In parallel with the aforementioned activities, the technical work to be performed in the first months of 2016

will be also focused on the achievement of the “Experiment based federation” that will harmonise the

description of the different resources federated through “Resource based federation” in order to be managed

and used by the experimenters through the FESTIVAL portal based on the “Experimentation as a Service”

approach.

80




8. References

[1] «FESTIVAL, "D1.3 - First architecture"».

[2] «FESTIVAL, "D2.3 Open data guidelines and federated testbed policy"».

[3] «KNX,» [Online]. Available: http://www.knx.org/knx-en/index.php.

[4] «FED4FIRE,» [Online]. Available: http://www.fed4fire.eu/.

[5] «Resource Specification (RSpec),» [Online]. Available: http://groups.geni.net/geni/wiki/GeniRspec.

[6] «OpenHab,» [Online]. Available: http://www.openhab.org/.

[7] «DNS-Based Service Discovery,» [Online]. Available: https://tools.ietf.org/html/rfc6763.

[8] «1888-2014 - IEEE Standard for Ubiquitous Green Community Control Network Protocol,» [Online].

Available: https://standards.ieee.org/findstds/standard/1888-2014.html.

[9] «Sensor Model Language (SensorML),» [Online]. Available:

http://www.opengeospatial.org/standards/sensorml.

[10] «FI-WARE NGSI-9 Open RESTful API Specification,» [Online]. Available:

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI-

9_Open_RESTful_API_Specification.

[11] «FI-WARE NGSI-10 Open RESTful API Specification,» [Online]. Available:

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FI-WARE_NGSI-

10_Open_RESTful_API_Specification.

[12] «Identity Management - KeyRock,» [Online]. Available: http://catalogue.fiware.org/enablers/identity-

management-keyrock.

[13] «Kibana | Explore & Visualize Your Data,» [Online]. Available: https://www.elastic.co/products/kibana.

[14] «FESTIVAL, "D3.1 Baseline report of selected applications and experiments"».

[15] «Libelium Comunicaciones Distribuidas S.L.,» [Online]. Available: http://www.libelium.com/.

81




[16] M. Mochizuki, J. Tsuneo, J. Nishida, H. Nakano e N. Nishio, «A Construction of Anonymized Pedestrian

Flow Analysis System using WiFi Packet Sensors,» SIG Technical Reports, Vol. %1 di %22014-MBL-70, n.

45, pp. 1-8, 2014.

[17] «influxdb - TIME-SERIES DATA STORAGE,» [Online]. Available: https://influxdata.com/time-series-

platform/influxdb/.

[18] «fluentd,» [Online]. Available: http://www.fluentd.org/.

[19] «Fluent plugin for MQTT Input/Output,» [Online]. Available: https://github.com/toyokazu/fluent-

plugin-mqtt-io.

[20] «Fluent plugin for XML Input,» [Online]. Available: https://github.com/toyokazu/fluent-plugin-xml-

parser.

[21] «FIWARE Lab,» [Online]. Available: https://account.lab.fiware.org/.

[22] «FIWARE Lab Nodes,» [Online]. Available: http://infographic.lab.fiware.org/.

[23] «Publish/Subscribe Context Broker - Orion Context Broker,» [Online]. Available:

http://catalogue.fiware.org/enablers/publishsubscribe-context-broker-orion-context-broker.

[24] «Cygnus,» [Online]. Available: https://github.com/telefonicaid/fiware-cygnus.

[25] «HDFS Users Guide,» [Online]. Available: http://hadoop.apache.org/docs/current/hadoop-project-

dist/hadoop-hdfs/HdfsUserGuide.html.

[26] «CKAN,» [Online]. Available: http://catalogue.fiware.org/enablers/ckan.

[27] «MySql,» [Online]. Available: https://www.mysql.com/.

[28] «MongoDB,» [Online]. Available: https://www.mongodb.org/.

[29] «Backend Device Management - IDAS,» [Online]. Available:

http://catalogue.fiware.org/enablers/backend-device-management-idas.

[30] «FIWARE Architecture Description IoT Backend Device Management,» [Online]. Available:

https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE.ArchitectureDescription.I

oT.Backend.DeviceManagement.

[31] «Node.js,» [Online]. Available: https://nodejs.org/en/.

[32] «fiware-figway,» [Online]. Available: https://github.com/telefonicaid/fiware-figway/.

82




[33] «Drupal,» [Online]. Available: https://www.drupal.org/.

[34] «WordPress.org,» [Online]. Available: https://wordpress.org/.

[35] «Disqus Extension,» [Online]. Available: http://extensions.ckan.org/extension/disqus/.

[36] «Disqus,» [Online]. Available: https://disqus.com/).

[37] «Archive CKAN datastore to S3,» [Online]. Available: http://extensions.ckan.org/extension/s3archive/.

[38] «Amazon S3,» [Online]. Available:

http://aws.amazon.com/it/s3/?utm_source=AWSBlog&utm_medium=socialmedia&SM_AWS_Blog_S

3Glacier.

[39] «OAuth2 Support for CKAN,» [Online]. Available:

http://extensions.ckan.org/extension/oauth2conwet/.

[40] «OAuth 2.0,» [Online]. Available: http://oauth.net/2/.

[41] «Application Mashup - Wirecloud,» [Online]. Available:

http://catalogue.fiware.org/enablers/application-mashup-wirecloud.

[42] «WireCloud - User Guide,» [Online]. Available:

https://wirecloud.readthedocs.org/en/latest/user_guide/.

[43] «Cyber Kansai Project,» [Online]. Available: http://www.ckp.or.jp.

[44] «Getting started with the MQTT client for Java,» IBM, [Online]. Available: http://www-

01.ibm.com/support/knowledgecenter/SS9D84_1.0.0/com.ibm.mm.tc.doc/tc10100_.htm.

[45] «Arduino,» [Online]. Available: https://www.arduino.cc/.

[46] «RASPBERRY PI,» [Online]. Available: http://www.raspberrypi.org/.

[47] «JCKANClient,» [Online]. Available: https://github.com/okfn/ckanclient-j.

[48] «RESTful API Modeling Language (RAML),» [Online]. Available: http://raml.org/.

[49] «FESTIVAL, "D1.2 Analysis of existing testbeds and SW HW components"».

[50] «Santander Datos Abiertos,» [Online]. Available: http://datos.santander.es/catalogo.

[51] «Virtuoso,» [Online]. Available: http://virtuoso.openlinksw.com/.

83




[52] «FiWARE Lab Open Data portal,» [Online]. Available: https://data.lab.fiware.org/.

[53] «CKAN node for FESTIVAL in JOSE testbed,» [Online]. Available: http://festival.ckp.jp/.

[54] «Data Grand Lyon,» [Online]. Available: http://data.grandlyon.com/.

[55] «FESTIVAL Japanese Open Data Platform,» [Online]. Available: http://festival.ckp.jp/.

[56] «nginx,» [Online]. Available: http://nginx.org/en/.

[57] «PIAX Testbed,» [Online]. Available: https://piax.jgn-x.jp/.

[58] «SFAWRAP,» [Online]. Available: http://sfawrap.info/.

[59] «FITeagle,» [Online]. Available: http://fiteagle.github.io/.

84




Annex 1: Federation API – RAML description

This section reports complete specification of Federation API in RAML language version 0.8.

#%RAML 0.8 title: Federation API version: 1.0 schemas: - "SearchCriteria": ' { "type": "object", "title": "SearchCriteria", "description": "The object containing the search criteria", "properties": { "query": { "type": "string", "required": true, "title": "query", "description": "The query string representing search criteria with SOLR notation - e.g. field:keyword OR other-keyword" }, "sort": { "type": "string", "required": true, "title": "sort", "description": "Sort criteria of the search results with SOLR notation - e.g. field asc/desc" }, "rows": { "type": "string", "required": true, "title": "rows", "description": "The number of search results to return" } } }' "SearchResults": '{ "type": "object", "description": "The object representing the results coming from the execution of a search", "title": "SearchResults", "properties": { "results": { "type": "array", "required": true, "title": "results", "description": "The list of the dataset metadata representing a search result", "items": {

85




"type": "object", "$ref": "#/schemas/DCATDataset" }, "uniqueItems": true }, "count": { "type": "string", "required": true, "title": "count", "description": "The total count of matched datasets" } } }' "DCATDataset": '{ "type": "object", "description": "Representation of a dataset metadata object compliant to DCAT specification http://www.w3.org/TR/vocab-dcat", "title": "DCATDataset", "properties": { "title": { "type": "string", "required": true, "title": "title", "description": "The title of the dataset" }, "description": { "type": "string", "required": true, "title": "description", "description": "free-text account of the dataset" }, "identifier": { "type": "string", "required": true, "title": "identifier", "description": "A unique identifier of the dataset" }, "issued": { "type": "string", "required": true, "title": "issued", "description": "Date of formal issuance (e.g., publication) of the dataset. Compliant to ISO 8601 specification" }, "modified": { "type": "string", "required": true,

86




"title": "modified", "description": "Most recent date on which the dataset was changed, updated or modified. Compliant to ISO 8601 specification" }, "landingPage": { "type": "string", "required": true, "title": "landingPage", "description": "A Web page that can be navigated to in a Web browser to gain access to the dataset, its distributions and/or additional information" }, "versionNotes": { "type": "string", "required": true, "title": "versionNotes", "description": "A description of changes between this version and the previous version of the dataset, according to ADMS profile http://www.w3.org/TR/vocab-adms" }, "accrualPeriodicity": { "type": "string", "required": true, "title": "accrualPeriodicity", "description": "The frequency at which the dataset is published" }, "spatial": { "type": "string", "required": true, "title": "spatial", "description": "Spatial coverage of the dataset" }, "temporal": { "type": "string", "required": true, "title": "temporal", "description": "The temporal period that the dataset covers" }, "language": { "type": "string", "required": true, "title": "language", "description": "The language of the dataset" }, "keyword": { "type": "array", "required": true, "title": "keyword", "description": "A keyword or tag describing the dataset", "items": {

87




"type": "string", "title": "keyword" }, "uniqueItems": true }, "publisher": { "type": "object", "$ref": "#/schemas/FOAFAgent", "required": true, "title": "publisher", "description": "The entity responsible for making the catalog online" }, "contactPoint": { "type": "object", "$ref": "#/schemas/VCardKind", "required": true, "title": "contactPoint", "description": "Link a dataset to relevant contact information which is provided using VCard" }, "distribution": { "type": "array", "required": true, "title": "distribution", "description": "Represents a specific available form of a dataset. Each dataset might be available in different forms, representing different formats of the dataset or different endpoints", "items": { "type": "object", "$ref": "#/schemas/DCATDistribution" }, "uniqueItems": true }, "versionInfo": { "type": "string", "required": true, "title": "versionInfo" } } }' "FOAFAgent": '{ "type": "object", "description": "Agent class of FOAF Vocabulary Specification", "title": "FOAFAgent", "properties": { "name": { "type": "string", "required": true, "title": "name"

88




}, "mbox": { "type": "string", "required": true, "title": "mbox" }, "homepage": { "type": "string", "required": true, "title": "homepage" }, "type": { "type": "string", "required": true, "title": "type" } } }' "VCardKind": '{ "type": "object", "description": "VCard object compliant to specification http://www.w3.org/TR/vcard-rdf", "title": "VCardKind", "properties": { "fn": { "type": "string", "required": true, "title": "fn" }, "hasEmail": { "type": "string", "required": true, "title": "hasEmail" } } }' "DCATDistribution": '{ "type": "object", "description": "Represents a specific available form of a dataset. Each dataset might be available in different forms, representing different formats of the dataset or different endpoi nts. Examples of distributions include a downloadable CSV file, an API or an RSS feed", "title": "DCATDistribution", "properties": { "license": { "type": "string", "required": true, "title": "license",

89




"description": "This links to the license document under which the distribution is made available" }, "issued": { "type": "string", "required": true, "title": "issued", "description": "Date of formal issuance (e.g., publication) of the distribution" }, "modified": { "type": "string", "required": true, "title": "modified", "description": "Most recent date on which the distribution was changed, updated or modified" }, "byteSize": { "type": "string", "required": true, "title": "byteSize", "description": "The size of a distribution in bytes" }, "title": { "type": "string", "required": true, "title": "title", "description": "A name given to the distribution" }, "description": { "type": "string", "required": true, "title": "description", "description": "free-text account of the distribution" }, "accessURL": { "type": "string", "required": true, "title": "accessURL", "description": "A landing page, feed, SPARQL endpoint or other type of resource that gives access to the distribution of the dataset" }, "mediaType": { "type": "string", "required": true, "title": "mediaType", "description": "The media type of the distribution as defined by IANA" }, "format": { "type": "string", "required": true,

90




"title": "format", "description": "The file format of the distribution" } } }' "ErrorReport": '{ "type": "object", "description": "It represents an error to be reported", "title": "ErrorReport", "properties": { "errorCode": { "type": "string", "required": true, "title": "errorCode", "description": "The numeric code of the error occurred in the system" }, "errorMessage": { "type": "string", "required": true, "title": "errorMessage", "description": "Descriptive message related to the occured error" } } }' "datasetPresenceInfo": '{ "type": "object", "description": "Object representing the info about the presence of a dataset in the node", "title": "datasetPresenceInfo", "properties": { "identifier": { "type": "string", "required": true, "title": "identifier", "description": "The unique numeric identifier of the dataset" }, "issued": { "type": "string", "required": true, "title": "issued", "description": "Date of formal issuance (e.g., publication) of the present dat aset" }, "modified": { "type": "string", "required": true, "title": "modified",

91




"descrption": "Most recent date on which the present dataset was changed, updated or modified" } } }' "/odf/odms/search": displayName: Search description: Perform the full-text search on datasets metadata post: description: Perform metadata search body: application/json: schema: SearchCriteria example: ' { "sort": "title asc", "query": "field:keyword OR field:keyword OR keyword", "rows": "200" }' responses: "500": description: The response returned when an error occurred body: application/json: example: '{"errorMessage":"Internal Server Error","errorCode":"500"}' "200": description: The response returned when a search is successfully completed body: application/json: schema: SearchResults example: ' { "count": "The total count of the matched datasets", "results": [ { "identifier": "A unique identifier of the dataset", "spatial": "Spatial coverage of the dataset", "keyword": [ "pollution", "metro" ], "issued": "2015-10-12T12:03:22Z", "landingPage": "baseUrl/datasets/identifier", "publisher": { "name": "The entity responsible for making the catalog online", "mbox": "Publisher Mail box", "homepage": "Publisher Homepage",

92




"type": "Publisher Type" }, "modified": "2015-10-12T12:03:22Z", "title": "title of the dataset", "accrualPeriodicity": "The frequency at which the dataset is published", "description": "text description of the dataset", "temporal": "The temporal period that the dataset covers", "versionNotes": "A description of changes between this version and the previous version", "contactPoint": { "fn": "Contact Full Name", "hasEmail": "Contact Mail Box" }, "language": "dataset language", "distribution": [ { "title": "Distribution Title", "accessURL": "Distribution URL of access", "description": "Distribution description", "mediaType": "The media type of the distribution as defined by IANA", "format": "The file format of the distribution", "license": "Link of license document under which the resource is made available", "issued": "2015-10-12T12:03:14Z", "modified": "2015-10-12T12:03:14Z", "byteSize": "1024" } ] } ] }' "/odf/odms/datasets/{id}": displayName: GetDataset description: Get the dataset with the provided id uriParameters: id: displayName: id description: The id of the requested dataset type: integer required: true repeat: false get: description: Get a dataset responses: "200": description: Return the requested dataset body: application/json: schema: DCATDataset example: '

93




{ "identifier": "A unique numeric identifier of the dataset", "spatial": "Spatial coverage of the dataset", "keyword": [ "pollution", "metro" ], "issued": "2015-10-12T12:03:22Z", "landingPage": "baseUrl/datasets/identifier", "publisher": { "name": "The entity responsible for making the catalog online", "mbox": "Publisher Mail box", "homepage": "Publisher Homepage", "type": "Publisher Type" }, "modified": "2015-10-12T12:03:22Z", "title": "title of the dataset", "accrualPeriodicity": "The frequency at which the dataset is published", "description": "text description of the dataset", "temporal": "The temporal period that the dataset covers", "versionNotes": "A description of changes between this version and the previous version", "contactPoint": { "fn": "Contact Full Name", "hasEmail": "Contact Mail Box" }, "language": "dataset language", "distribution": [ { "title": "Distribution Title", "accessURL": "Distribution URL of access", "description": "Distribution description", "mediaType": "The media type of the distribution as defined by IANA", "format": "The file format of the distribution", "license": "Link of license document under which the resource is made available", "issued": "2015-10-12T12:03:14Z", "modified": "2015-10-12T12:03:14Z", "byteSize": "1024" } ] }' "500": description: An error occurred during the request processing body: application/json: example: '{"errorMessage":"Internal Server Error","errorCode":"500"}' "404": description: The error occurred if the dataset with the provided id does not exist body:

94




application/json: example: '{"errorMessage":"The dataset was not found","errorCode":"404"}' "/odf/odms/datasets/info": displayName: GetAllDatasetsID description: Get the list of all available datasets (id and date info) get: responses: "200": description: Get the list of all available datasets (id and date info) body: application/json: schema: listDataset example: ' [ { "issued": "2015-10-12T12:03:14Z", "identifier": "unique numeric identifier of the dataset", "modified": "2015-10-12T12:03:14Z" } ]' "500": description: An error occurred during the request processing body: application/json: example: '{"errorMessage":"Internal Server Error","errorCode":"500"}' "/odf/odms/datasets": displayName: GetAllDatasets description: Get the list of all datasets on the node get: description: Get all datasets metadata (paginated). The list of datasets will be divided by page and number of datasets per page, avoiding the one-time discharge of a great number of datasets. queryParameters: offset: displayName: offset description: The offset (page) of the list of all datasets type: string required: true repeat: false rows: displayName: rows description: The number of datasets returned per page type: string required: true repeat: false responses: "200": description: The list of datasets metadata, compliant to DCAT Specification body:

95




application/json: schema: DCATDataset example: ' [ { "identifier": "A unique numeric identifier of the dataset", "spatial": "Spatial coverage of the dataset", "keyword": [ "pollution", "metro" ], "issued": "2015-10-12T12:03:22Z", "landingPage": "baseUrl/datasets/identifier", "publisher": { "name": "The entity responsible for making the catalog online", "mbox": "Publisher Mail box", "homepage": "Publisher Homepage", "type": "Publisher Type" }, "modified": "2015-10-12T12:03:22Z", "title": "title of the dataset", "accrualPeriodicity": "The frequency at which the dataset is published", "description": "text description of the dataset", "temporal": "The temporal period that the dataset covers", "versionNotes": "A description of changes between this version and the previous version", "contactPoint": { "fn": "Contact Full Name", "hasEmail": "Contact Mail Box" }, "language": "dataset language", "distribution": [ { "title": "Distribution Title", "accessURL": "Distribution URL of access", "description": "Distribution description", "mediaType": "The media type of the distribution as defined by IANA", "format": "The file format of the distribution", "license": "Link of license document under which the resource is made available", "issued": "2015-10-12T12:03:14Z", "modified": "2015-10-12T12:03:14Z", "byteSize": "1024" } ] } ]' "500": description: An error occurred during the request processing body:

96




application/json: example: '{"errorMessage":"Internal Server Error","errorCode":"500"}'

Documents

Project · PDF fileLOD Linked Open Data ... Figure 16: An example ElasticSearch mapping for sensor data storing ..... 33 Figure 17: MQTT message conversion