The FI-WARE Project – Mediator GE Webinar June 2013 Mediator GE Webinar

The FI-WARE Project – Mediator GE Webinar

June 2013

Mediator GE Webinar


Agenda

2

Mediator GE Overview

Mediator GE: Mediation Engine (TI)

Mediator GE: Dynamic Mediation API (Thales)

Demo: Event Brokering

Questions & Answers


Overview

FI-WARE platform should be able to support services exposed through different protocols and technologies and enable the creation of new composite services. The mediator shall provide the glue between the service layer and the composition layer in order to enhance the composition capabilities of the compositor GEs.


Capabilities

4

The main capability of the Mediator GE is to build an interoperability layer providing implementation for various enterprise integration patterns and data and protocol transformationsIt is a kind of broker between the service consumer and the service provider

Particularly it provides Data Mediation (Transformation), Protocol Mediation and routing capabilities using various mechanisms:

Using a library of built-in transformers in case of well-known formats Using templates for more flexible data/protocol transformation capabilities Using XML DSL (Domain Specific Language) in order to fulfill more

complex protocol/data transformation requirements or complex integration patterns (split/aggregations)

Using custom code (various programming languages are supported) to reach maximum flexibility in most complex scenarios


Role in FIWARE

5

Basically the Mediator GE could play the role of centralizing and abstracting the access towards the heterogeneity of different devices or target applications exposing different capabilities or the same capabilities through different interfaces and data models.

In this scenario the mediator GE can be configured to implement different functionalities:

exposing a uniform interface and data model towards a plethora of heterogeneous devices and sensors

routing incoming messages towards the right target device or service based on the content of the message or other criteria

performing the role of splitting a coarse grained task into many specific items that shall be executed by different actors (device or applications), dispatching the split tasks and aggregating all results in order to return the whole result to the caller.

performing the role of event broker. A client application interested in events from a particular set of devices or other applications can subscribe to them using a particular service exposed by the mediator where all relevant event sources are configured.

providing an API for automatically solving particular mediation issues (based on semantic meta-data)


Architecture

The mediator offers a set of available mediation tasks: the set of mediation capabilities that can be used via the mediator.

The mediator allows users to create and manage their mediation services: a mediation service is a virtual proxy towards a web service that executes a mediation task between the caller and the target service. The mediation task must be chosen from the set of available mediation tasks.

The mediator GE is used mainly by the Compositor Editor and Compositor Engine GEs within the FI-WARE platform. It provides a layer of virtual proxies to be used by the compositor instead of the target services in order to make the compositor able to support various kind of target services. Besides the FI-WARE platform also Future Internet applications or composite services on top of the FI-WARE platform can use the mediator as a service for their own purpose.


Mediation Engine (TI)

The reference implementation of GE-Mediator provided by Telecom Italia is based on two open source frameworks:

•WSO2 ESB (http://wso2.com/products/enterprise-service-bus)

•Apache Camel (http://camel.apache.org)

Particularly the value added by Telecom Italia is related to:

•A custom mediation module based on Apache Camel in order to enhance the mediation capabilities of WSO2 ESB.

• Basically it is possible to add Camel route to a running instance of WSO2 ESB through a customization of the WSO2 ESB Web Admin GUI

•An enhanced per user logging capability• Basically it provides data related the who is using what inside the ESB

• The integration with a monitoring and disaster recovery tool based on JMX

http://wso2.com/products/enterprise-service-bus


WSO2 Enterprise Bus Overview

The mediation built-in capabilities are provided through:

Service VirtualizationMessage Mediation

The main mediation capabilities are:Data TransformationProtocol mediationLoad Balancing and FailoverManipulating SOAP Headers (WS-Security) Iterate/clone messagesPolicy enforcement


How to configure Mediators

WSO2 ESB provide a built in Mediator Catalog Usually a mediator is configured using XMLWSO2 ESB provides a Web GUI to help to configure Mediators without or less XML ESB mediation framework is extensible. It is possible to write custom mediators using:

Pure Java Spring Bean Scripting Languages: javascript, ruby

Examples of Mediator functionalities: Drop Mediator : Drops a message, stopping further processing of the message Send Mediator :Send the message out, using information inside the message or according to some external

configuration Script Mediator : Executes a scripting language with the message Filter Mediator : Filter messages using XPath XSLT Mediator : Do XSLT transformations on messages


Mediators catalog: core mediators

Name Description

Send Send a message out

Log Logs a message

Property Set or remove properties associated with the message

Sequence Refer a sequence

Event Send event notifications to an event source

Drop Drops a message

Enrich Enriches a message

Enqueue Create an enqueue mediator


Mediators catalog: filter mediators

Name Description

Filter Filter a message using Xpath (if else logic)

Out Inbuilt filter for choosing messages in ESB out path

In Inbuilt filter for choosing messages in ESB path

Switch Filter a message using Xpath (switch logic)

Router Route messages based on XPath filtering

Conditional Router Route messages based on 'Condition'

Validate Schema validation for messages


Mediators catalog: transform mediators

Name Description

XSLT XSLT Transformations

XQuery XQuery Transformations

Header Set or remove SOAP Headers

Fault Create or remove SOAP Faults

Rewrite Create a rewrite mediators


Mediators catalog: advanced mediators

Name Description

Cache XSLT Transformations

Clone Clones a message

Iterate Splits a message

Aggregate Combines messages

Callout Blocking web services calls

Transaction Executes a set of mediators transactionally

Throttle Limit the number of messages

RMsequence Reliable messaging

DBReport Writes data to a database

DBLookup Retrieve data froma database

Rule Execute Rule

Other…


Mediators catalog: extensions

Name Description

Class Create and execute a custom mediator written in Java

POJOCommand Execute a custom command

Script Execute a mediator written in scripting language

Spring Create a mediator managed by spring

Other…


Example 1: managing WS-Security

<proxy xmlns="http://ws.apache.org/ns/synapse" name="smsService_ws_security" Transports="https" statistics="disable" trace="disable" startOnLoad="true">

<target> <outSequence> <send /> </outSequence> <endpoint> <address uri="http://localhost:9000/services/SmsMessageAdapter" /> </endpoint> </target> <enableSec /> <policy key="conf:/repository/axis2/service-groups/smsService_ws_security/

services/smsService_ws_security/policies/UTOverTransport" /></proxy>


Example 2: Data Transformation SOAP2REST

<proxy xmlns="http://ws.apache.org/ns/synapse" name="SoapToRest_Sms" transports="http" statistics="enable" trace="enable" startOnLoad="true"> <target> <inSequence> <property name="Accept" value="application/xml" scope="transport" /> <log level="full" /> <router> <route xmlns:soaptorest="http://telecomitalia.it/ictservice/smsService/soaptorest" expression="$body/soaptorest:sendSms"> <target sequence="anon"> <sequence> <xslt key="gov:/transformations/trasformSOAP2RESTPayload" /> <property name="HTTP_METHOD" value="POST" scope="axis2" type="STRING" /> <log level="full" /> <send> <endpoint> <address uri="http://polaris.cselt.it:9763/SmsServiceRest/rest/sms" format="rest" /> </endpoint> ….........................................

</proxy>


Example 2: Data Transformation SOAP2REST XSLT

<?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml" indent="yes"/>

<xsl:template match="/node()/node()"><xsl:copy>

<xsl:copy-of select="./*"/></xsl:copy>

</xsl:template>

</xsl:stylesheet>


Example 3: Protocol Transformation TCP2HTTP

<proxy xmlns="http://ws.apache.org/ns/synapse" name="TCPtoHTTP_mail" transports="tcp" statistics="enable" trace="enable" startOnLoad="true"> <target endpoint="fiwareMailproxy"> <inSequence> <log level="full" /> </inSequence> <outSequence> <send /> </outSequence> </target></proxy>


Example 4: Throttling capability

The throttling capability means that it is possible to restrict the usage of a mediation service using different parameters.

Range - The IP address range or the domain is restricted from accessing the service. Requests from such clients will be restricted based on the specified values.

The restriction can be : Deny or Control

Control means that it is possible to set usage limitation based on: - Maximum Request Count (MRC) - Prohibit Time Period (PTP)


Example 5: Loadbalancing capability

The loadbalancing capability means that it is possible to configure a mediation service to use different service endpoints in order to balance the calls using a round robin algorithm. More complex loadbalancing algorithms currently can be configured only using a custom mediator.

<session type="http|simpleClientSession"/>? <loadBalance [policy="roundRobin"]> <endpoint .../>+ </loadBalance>


Dynamic Mediation API (DMA)Optional Library provided by Thales

22

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

Dynamic Mediation API (DMA)

Main features and assumptions:• Optional library delivered as part of the Mediator GE

• Can be used “internally” by the Mediator as part of a created “Mediation Service”

• API can also be used directly by third-parties for solving particular mediation issues (as shown in this presentation)

• FI-PPP projects: use under the conditions established in the FI-PPP Collaboration Agreement. Distributed under Thales license as a binary-only closed-source library

• Useful when syntactic mismatch occurs (at design or runtime) between Service consumer’s needs and Target Service offer.

• Additional semantic information (meta-data) + domain ontology is used automatically by the DMA for bridging the syntactic gap between a Service Consumer and a Target service => automatic data conversion

23

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA Overview

SSL - Smart Systems Lab

Ontology

Automaticconversion

Mismatch

Service Request

TargetService

Data Request Data Target

DMA features

(1) Automatically deal with service (request / target) syntactic mismatch

Use semantic-based characterization of service features (methods, parameters,…)

Supports direct target service invocation through library API

Communication protocol agnostic (SOAP, REST)

(2) Automatic data mediation between services Data must be described using XSD and

semantically-annotated (3)Use of ontology as business model reference

(1) Automatically deal with service (request / target) syntactic mismatch

Use semantic-based characterization of service features (methods, parameters,…)

Supports direct target service invocation through library API

Communication protocol agnostic (SOAP, REST)

(2) Automatic data mediation between services Data must be described using XSD and

semantically-annotated (3)Use of ontology as business model reference

DMA Library

1

2

3is_equivalent

24

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA API typical use cases

1. Consumer needs to invoke a specific operation on an available target service and already has a structured payload he wants to send

Potential mismatch at data-model level payload needs to be automatically converted by the DMA

Semantic annotation of consumer-side payload and target service offer mandatory DMA will find the semantically matching operation in the target service

2. Consumer needs to invoke a specific operation on an available target service and doesn’t have an imposed structured payload to send

Consumer will provide DMA with a list of parameters with simple data type Consumer-side data-model undefined payload will be automatically created by

the DMA Semantic annotation of parameters and target service offer still mandatory DMA will find the semantically matching operation in the target service

3. In both cases, DMA can be used either to build the right payload (and stop), or to also deal with target service invocation

25

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA API (1/4)

DMA provides 4 Dynamic Mediation Tasks that are accessible as Java methods or SOAP endpoints:

String dynamicallybuildServicePayloadNoOPWithParameters( String usdlURI, String serviceID, String operationConcept , List<SemanticParamItem> inputs)

throws MediatorException;

Method description:• Purpose: using a list of unstructured data as input parameters, dynamically building an XML payload adapted to a given service and linked to an operation concept.

26

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA API (2/4)

String dynamicallybuildServicePayloadNoOPWithPayload( String usdlURI, String serviceID, String operationConcept , String saxd_payload, String payload )


Method description:• Purpose: using a structured payload (with its associated data schema) as input parameter, dynamically building an XML payload adapted to a given service and linked to an operation concept.

27

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA API (3/4)

String dynamicDataMediationInvocationNoOPWithParameters ( String usdlURI, String serviceID, String operationConcept , List<SemanticParamItem> inputs , List<SemanticParamItem> awaitedOutput)


Prototype description:• Purpose: dynamically searches and invokes a service capability based on the given operation concept. The operation of the service, if found, is called with a dynamically built payload based on the list of unstructured data passed as input parameters. • Note: the expected data format of the result(s) is defined by the last input parameter awaitedOutput. The result is then adapted before being returned.

28

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

DMA API (4/4)

String dynamicDataMediationInvocationNoOPWithPayload ( String saxsd_payload, String serviceID, String payload, String wanted_response, String operationConcept)


Prototype description:• Purpose: dynamically searches and invokes a service capability based on the given operation concept. The operation of the service, if found, is executed using the input payload, first adapted to the service. • Note: the data format of both input payload and expected result(s) is defined by the input parameter saxd_payload. The result is then adapted before being returned.

29

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

A: Consumer needs to invoke a specific operation on an available target service and doesn’t have an imposed structured payload to send

IMediator mediator = MediatorEntry.getInstance();;

String operationConcept = "http://www.example.com/Calculator.owl#Addition";

SemanticParamItem param1 = new SemanticParamItem();

param1.setConcept("http://www.example.com/Calculator.owl#OperandeGauche");

param1.setValue("12");


param2.setConcept("http://www.example.com/Calculator.owl#OperandeDroite");

param2.setValue("5");


param3.setConcept("http://www.example.com/Calculator.owl#Somme");

List<SemanticParamItem> result = null;

result = mediator.dynamicDataMediationInvocationNoOPWithParameters(usdlURI, serviceID, operationConcept, inputs, outputs);

for (SemanticParamItem item : result){

assertTrue(item.getValue().equals("17"));

System.out.println(item.getConcept());

System.out.println(item.getValue());

}

Use case: simple calculator exemple – Case 1

30

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

B: Consumer needs to invoke a specific operation on an available target service and already has a structured payload he wants to send

IMediator mediator = MediatorEntry.getInstance();;

String wanted_response = "AdditionResponse";

String payload = Util.getLocalPathFromResource("fiware/payload.xml");


param3.setConcept("http://www.example.com/Calculator.owl#Somme");

String saxsd_payload = Util.getLocalPathFromResource("fiware/saxsd_inputs_outputs.xml");

System.out.println(saxsd_payload);

String operationConcept = "http://www.example.com/Calculator.owl#Addition";

String result = null;

result = mediator.dynamicDataMediationInvocationNoOPWithPayload(

saxsd_payload, usdlURI, serviceID, payload, wanted_response, operationConcept);

assertTrue(result != null);

assertTrue(result.contains("15"));


31

Les

info

rma t

ion s

co

nten

ues

d an s

ce

doc

um

en t

de

meu

ren

t la

pro

prié

té e

xclu

siv e

du

gro

upe

TH

AL

ES

et n

e d

oiv

en t

pa

s ê

tre

divu

lgu

é es

à l 'e

x té

ri eu r

du

Gro

up

e .

Hypothesis of work:• Payload:

<Addition>

<operandeg>12</operandeg>

<operanded>3</operanded>

</Addition>

• Annotated schema for the payload and/or request/response using SAWSDL:<xs:element name="Addition">

<xs:complexType>

<xs:sequence>

<xs:element name="operandeg" sawsdl:modelReference="http://www.example.com/Calculator.owl#OperandeGauche" type="xs:string"/>

<xs:element name="operanded" sawsdl:modelReference="http://www.example.com/Calculator.owl#OperandeDroite" type="xs:string"/>

</xs:sequence>

</xs:complexType>

</xs:element>

<xs:element name="AdditionResponse">

<xs:complexType>

<xs:sequence>

<xs:element name="resultat" sawsdl:modelReference="http://www.example.com/Calculator.owl#Somme" type="xs:string"/>

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:schema>



DEMO


Example Scenario: Task Dispatcher and Event Brokering

33


Example Scenario: Task Dispatcher and Event Brokering

34

In this scenario the mediator can play the roles of task dispatcher using the split/aggregator pattern and event broker.

As a task dispatcher the mediator receive a coarse grained task from the task manager and , based on some criteria related to the content of the task, split it into specific items and dispatch them to the right target (human or application)

As an event broker the mediator exposes a service in order to receive events from one or more event sources and allows applications to subscribe to an event topic in order to get related events when they are received by the broker. In this way the event broker collect events from devices, optionally filter them and send them to the current subscribers.

Moreover, in this demo scenario the mediator play also the role of centralizing the calls to third party services used by the client (subscribed) applications (i.e. email and sms services).

In order to show the load-balancing capability of the mediator GE, the email service mediation can be configured to use two different endpoints balanced through a simple round robin algorithm


References

Fiware Apps Wiki Architecture https://forge.fi-ware.eu/plugins/mediawiki/wiki/fi-ware-private/index.php/FIWARE.OpenSpecification.Apps.Mediator

Fiware Apps Wik Open Spec: https://forge.fi-ware.eu/plugins/mediawiki/wiki/fi-ware-private/index.php/FIWARE.OpenSpecification.Apps.Mediator

WSO2 ESB docs site: http://wso2.com/products/enterprise-service-bus

http://wso2.com/products/enterprise-service-bus

Documents

The FI-WARE Project – Mediator GE Webinar June 2013 Mediator GE Webinar