Technical Note on Common Services, Intermediate …...H2020 – EINFRA – 2015 – 1 Page 1 of 37 Technical Note on Common Services, Intermediate Version Workpackage 5 VRE Infrastructure

H2020 – EINFRA – 2015 – 1 Page 1 of 37

Technical Note on Common Services,

Intermediate Version

Workpackage 5 VRE Infrastructure and Services Design and Development

Task 5.2 Common Services

Author (s) Ugo Di Giammatteo

Serena Avolio

Sergio de Gioia

ACS

ACS

ACS

Reviewer (s) Fulvio Marelli

Simone Mantovani

T2

MEEO

Approver (s) Pedro Gonçalves

Cristiano Silvagni

T2

ESA

Authorizer Mirko Albani ESA

Document Identifier EVER-EST DEL WP5-D5.2

Dissemination Level Public

Status Approved by the EC

Version 1.1

Date of issue 14 December 2018

H2020 – EINFRA – 2015 – 1 Page 2 of 37

Document Log

Date Author Changes Version Status

27/6/2016 Fulvio Marelli TOC 0.1 Draft

03/08/2016 Ugo Di Giammatteo Revised TOC 0.2 Draft

14/10/2016 Serena Avolio Enriched TOC 0.3 Draft

18/10/2016 Serena Avolio Section 2,3,4 (now 5) 0.4 Draft

27/10/2016 Serena Avolio Section 4 0.5 Draft

28/10/2016

Ugo Di giammatteo

Sergio de Gioia Issue for revision 0.6 Draft

23/11/2016

Serena Avolio

Sergio de Gioia

Implementation of changes from reviewers, annexes and some figures added

0.7 Draft

25/11/2016 Serena Avolio Implementation of minor changes from reviewers

0.8 Draft

08/12/2016 Serena Avolio Implementation of changes from Approvers

0.9 Draft

12/12/2016 Serena Avolio Implementation of final comments.

1.0 Draft to be approved by the EC

14/12/2018 Cristiano Silvagni Document status 1.1 Approved by the EC

H2020 – EINFRA – 2015 – 1 Page 3 of 37

Table of contents

1 Introduction ........................................................................................................................................... 9

1.1 Purpose and scope ........................................................................................................................... 9

1.2 Who shall read this document .......................................................................................................... 9

1.3 System context ................................................................................................................................ 9

1.4 Services list...................................................................................................................................... 9

2 WSO2 IS ............................................................................................................................................... 11

2.1 Services offered overview .............................................................................................................. 11

2.2 Design elements ............................................................................................................................ 11

2.3 Customisation ............................................................................................................................... 12

2.4 Installation guide ........................................................................................................................... 13

2.5 Using the service – quick overview ................................................................................................. 13

2.5.1 Getting started ............................................................................................................................... 16

2.6 Service public API’s ........................................................................................................................ 16

2.6.1 The logic ......................................................................................................................................... 17

2.6.2 APIs ................................................................................................................................................. 18

2.6.3 Troubleshooting ............................................................................................................................. 22

3 WSO2 ESB ............................................................................................................................................ 23


3.2 Design elements ............................................................................................................................ 23

3.3 Customisation ............................................................................................................................... 24





4 WSO2 DAS ........................................................................................................................................... 26


4.2 Design elements ............................................................................................................................ 26

4.3 Customisation ............................................................................................................................... 27




4.6.1 The logic ......................................................................................................................................... 28

4.6.2 DAS APIs for EVER-EST system ....................................................................................................... 28


5 SeaFile ................................................................................................................................................. 31


5.2 Design elements ............................................................................................................................ 31

5.3 Customisation ............................................................................................................................... 31

H2020 – EINFRA – 2015 – 1 Page 4 of 37





Annex A Installation Guides ................................................................................................................... 34

A.1. WSO2 IS Installation guide ............................................................................................................. 34

A.1.1 Software prerequisites ................................................................................................................... 34

A.1.2 Hardware prerequisites.................................................................................................................. 34

A.1.3 Uninstallation procedure ............................................................................................................... 34

A.2 WSO2 ESB Installation guide .......................................................................................................... 35




A.3 WSO2 DAS Installation Guide ......................................................................................................... 36




H2020 – EINFRA – 2015 – 1 Page 5 of 37

List of Figures

Figure 2-1 WSO2 IS architectural diagram in the orange box. The Service Providers box shows as the other EVER-

EST components interacts with the IS................................................................................................................. 12

Figure 2-2 Configuration of a service provider with the different possible authentication protocols....................... 13

Figure 2-3 VRE Portal, highlighted the login button that redirects to EVER-EST IS .................................................... 14

Figure 2-4 Login page of the EVER-EST IS .................................................................................................................... 15

Figure 2-5 Consent page on the EVER-EST IS .............................................................................................................. 15

Figure 2-6 User is authenticated and redirected back to the portal .......................................................................... 16

Figure 2-3Exposed WSDL ............................................................................................................................................. 17

Figure 2-3 Authentication Process and API protection ............................................................................................... 18

Figure 3-1The Service BUS........................................................................................................................................... 24

Figure 4-1The WSO2 DAS architecture. ...................................................................................................................... 27

Figure 4-2How to create an event receiver in DAS ..................................................................................................... 27

Figure 5-1Seafile Web Interface .................................................................................................................................. 32

Figure 5-2Seafile desktop application ......................................................................................................................... 32

List of Tables

Table 1 Operation name and its endpoint for SOAP request in version 1.1 ............................................................... 18

Table 2 Example of a validate action request with SOAP in version 1.1 ..................................................................... 19

Table 3 Operation name and its endpoint for SOAP request in version 1.2 ............................................................... 19

Table 4 Example of a validate action request with SOAP in version 1.2 ..................................................................... 20

Table 5 Examples of responses to a validate action request. The three different possibles statuses. ...................... 21

Table 6 Description of the required fileds to configure an event receiver in DAS ..................................................... 28

Table 7 Description of the APIs provided by DAS. ...................................................................................................... 29

Table 8 The parameters used in the different APIs listed and explained. .................................................................. 29

Table 9 The possibles error codes from the APIs ........................................................................................................ 29

Table 10 WSO2 IS software prerequisites ................................................................................................................... 34

Table 11 WSO2 IS hardware prerequisites ................................................................................................................. 34

Table 12 WSO2 ESB software prerequisites ................................................................................................................ 35

Table 13 WSO2 ESB hadware prerequisites ................................................................................................................ 35

Table 14 WSO2 DAS software prerequisites ............................................................................................................... 36

Table 15 WSO2 ESB software prerequisites ................................................................................................................ 36

Table 16 WSO2 DAS hardware prerequisites .............................................................................................................. 37

Table 17 WSO2 ESB hardware prerequisites .............................................................................................................. 37

H2020 – EINFRA – 2015 – 1 Page 6 of 37

Definitions and Acronyms

Acronym Description

ABAC Attribute Based Access Control

ACL Access Control List

AJAX Asynchronous JavaScript and XML

AMQP Advanced Message Queuing Protocol

API Application Programming Interface

BAPI Business Application Programming Interface

CMS Content Management System

COTS Commercial) Off-the-Shelf

CRL Certificate Revocation List

CRUD Create read Update Delete

CSV Comma-Separated Values

DAS Data Analytic Server

DOI Digital Object Identifier

E-PDSC Extended-Preservation Dataset Content

EAI Enterprise Application Integration

EDA Event Driven Architecture

EDI Electronic Data Interchange

EO Earth Observation

ERP Enterprise Resource Planning

ES Earth Science

ESA European Space Agency

ESB Enterprise Service Bus

EVAT Expert-user Visual Analysis Tool

EVER-EST European Virtual Environment for Research - Earth Science Themes

FTP File Transfer Protocol

FTPS FTP over SSL

GIF Graphics Interchange Format

GUI Graphical User Interface

HTML Hypertext Mark-up Language

HTTP Hypertext Transfer Protocol

HTTPS HTTP over TLS, HTTP over SSL, and HTTP Secure

IDE Integrated Development Environment

IIOP Internet Inter-ORB Protocol

IMAP Internet Message Access Protocol

H2020 – EINFRA – 2015 – 1 Page 7 of 37

IO Input Output

IPR Intellectual Property Rights

IS Identity Server

ISO International Organization for Standardization

IT Information Technology

IWA Integrated Windows Authentication

JIT Just In Time

JMX Java Management Extension

JPEG Joint Photographic Experts Group

JSON JS Object Notation

LDAP Lightweight Directory Access Protocol

LGPL Lesser General Public License

MEA Multi-sensor Evolution Analysis

MLLP Minimum Lower Layer Protocol

MOM Message Oriented Middleware

MQTT MQ Telemetry Transport

OAGIS Open Applications Group Integration Specification

OASIS Organization for the Advancement of Structured Information Standards

OCSP Online Certificate Status Protocol

OIDC Open ID Connect

OGC Open Geospatial Consortium

OPI Operator Panel Interface

PBAC Policy Based Access Control

PDSC Preservation Dataset Content

PNG Portable Network Graphics

PSNC Poznań Supercomputing and Networking Center

POP Point of presence

RBAC Role Based Access Control

RDF Resource Description Framework

REST Representational State Transfer

RO Research Objects

RODL Research Objects Digital Library

RSS Rich Site Summary

SFTP Secure File Transfer Protocol

SLA Service Level Agreement

SMTP Simple Mail Transfer Protocol

SPARQL Protocol and RDF Query Language

H2020 – EINFRA – 2015 – 1 Page 8 of 37

SQL Structured Query Language

SSO Single Sign-On

SVG Scalable Vector Graphics

TCP Transmission Control Protocol

UDP User Datagram Protocol

URI Uniform Resource Identifier

URL Uniform Resource Locator

VAC Visual Analysis Client

VRC Virtual Research Community

VRE Virtual Research Environment

WCF Windows Communication Foundation

WCPS Web Coverage Processing Service

WCS Web Coverage Service

WebCGM Web Computer Graphics Metafile

WFMS Workflow Management Systems

WFS Web Feature Service

WMS Web Map Service

WSDL Web Services Description Language

XML EXtensible Mark-up Language

Applicable Documents

Document ID Document Title

EVER-EST DEL WP5-D5.1 VRE Architecture and Interfaces Definition

Reference Documents

Document ID Document Title

EVER-EST DEL WP5-D5.1 VRE Architecture and Interfaces Definition

EVER-EST DEL WP5-D5.4 Technical Note on e-Research application services, Intermediate version

EVER-EST DEL WP5-D5.3 Technical Note on digital information and e-collaboration services

H2020 – EINFRA – 2015 – 1 Page 9 of 37

1 Introduction This Technical Note is the first deliverable of Task 5.2, whose objective is to design and implement the common services that will be used in the VRE.

This document takes into account the overall design sketched in the DoA, refined during the first 5 months of the project and reported in EVER-EST DEL WP5-D5.1. It takes also into account the subsequent work of analysis that allowed defining the detailed interfaces between main components the EVER-EST system. This interface definition has been carried out during dedicated technical meetings, plenary project meetings and numerous bi-lateral meetings and calls.

1.1 Purpose and scope

The purpose of this document is to describe the design of the Common Services that will be integrated and used within the first version of the EVER-EST system.

The scope comprises: the identification of all services that belong to this category, the grouping of these services according to the relevant components and the definition of interfaces between them and with services of other groups.

1.2 Who shall read this document

This document has been thought mainly to provide implementation and configuration information about the Common Services to the technical partners in the consortium. It contains also some information about the utilisation of the Common Services that can be useful for the VRC members.

1.3 System context

The Common Services category represents the family of services, which are used by different elements of the EVER-EST infrastructure. They refer to the category of services that will guarantee the correct functioning between various infrastructure components and between the infrastructure and the final users.

One of the most important goals of the common services is to provide a web services architecture satisfying the Service Oriented Architecture (SOA) principles. One of the SOA core concepts the loose coupling that means reducing the assumptions that two parties (components, applications, services, programs, users) make about each other when they exchange information. Following this concept the EVER-EST system is tolerant to changes because the parties are not tightly coupled to each other, it is also easily extensible, and, most important for a Virtual Research Environment, it allows the implemented infrastructure to be extended beyond the initial Earth Science communities to other scientific domains.

The SOA approach is achieved using middleware and interoperability standards. The common functionalities that have been identified to ease the integration of new/existing components are listed in the following paragraph.

1.4 Services list

A Server Component can offer multiple services.

The following services are offered to the EVER-EST users/components:

1. Authentication 2. Authorization 3. Account Provisioning

H2020 – EINFRA – 2015 – 1 Page 10 of 37

4. Identity Administration 5. Security 6. Message routing 7. Message transformation 8. Auditing 9. Data Discovery 10. Data Access 11. Data Storage

The first five Common Services are offered by the WSO2 Identity Server (IS). The component offering the message routing and transformation is the WSO2 Enterprise Service Bus (ESB). The components offering the Auditing service are the WSO2 Data Analytic Server (DAS)together with the WSO2 ESB. The personal Data Storage and Data Discovery services are provided by the SeaFile component, the Data Discovery and Data Access for what concerns EO data are guaranteed by the use of the Open Search specifications, OpendData protocol and OGC standards for discovering metadata on the different catalogues. The identified components that are going to be detailed in this document are:

1. WSO2 IS 2. WSO2 ESB 3. WSO2 DAS 4. SeaFile

The involved protocol, standard or specifications are the following:

1. OpenSearch Specification 2. OpenData protocol 3. OGC Standards

and they have been already detailed in the EVER-EST DEL WP5-D5.1 and in the EVER-EST DEL WP5-D5.3.

H2020 – EINFRA – 2015 – 1 Page 11 of 37

2 WSO2 IS WSO2 IS provides secure Identity Management for the EVER-EST Web Application services and APIs by managing identity and entitlements of the users securely and efficiently.

2.1 Services offered overview

Among the common services the ones offered by the IS for EVER-EST are briefly detailed hereafter:

1. Authentication, the process of identifying an individual. Authentication merely ensures that the individual is who he or she claims to be.

2. Authorization, the process of granting or denying access to a network resource. The authorization allows the user access to various resources based on the user's identity.

3. Account Provisioning, the process of providing users with access to data and technology resources. The process implies that the access rights and privileges are monitored and tracked to ensure the security of an enterprise's resources.

4. Identity Administration, the process of creating new and modifying or deleting existing identities as well as managing the security entitlements associated with those identities.

5. Security, the process to provide secure access to resources based on standards and model for security

All these functionalities are centralized to support the coordinated usage of the different EVER-EST components and allow their interoperability.In EVER-EST there are multiple applications that require authentication, users should be able to log in at one place and still have seamless access to all the other applications. The IS takes care of transformation of protocols and tokens.. Once the user provides its credentials it's automatically authenticated on all application enabling a Single Sign On (SSO) scenario between multiple heterogeneous authentication protocols.

The different components authenticated by the IS can choose to expose a System for Cross Domain Identity Management (SCIM) endpoint for the users provisioning/de-provisioning functionality that can be consumed by the IS. Alternatively the components can implement an own Just In Time (JIT) user provisioning to complete, for their necessity, the user information exposed for them by the IS. Currently the VRE portal and ROHUB Portal have implemented their own JIT provisioning.

2.2 Design elements

WSO2 IS is a COTS configured and extended for the EVER-EST environment. It enables customization and extension through its componentized architecture.

The WSO2 Identity Server is used directly by the administrator users, through its Management Console. Apart from such users, the Identity Server is mainly used in EVER-EST as an identity provider for the other components, services and applications.

The following diagram depicts inside the orange box the entire architecture of the Identity Server to show how its parts allow the integration of the existing and future EVER-EST components.

H2020 – EINFRA – 2015 – 1 Page 12 of 37

Figure 2-1 WSO2 IS architectural diagram in the orange box. The Service Providers box shows as the other EVER-

EST components interacts with the IS.

The IS works using inbound auhtenticators. The inbound authenticators parse incoming requests and construct corresponding responses for all the supported protocols. Among the supported authentication protocols there are the authentication protocols used by the others EVER-EST sub-systems, in particular Open Id Connect and SAML 2.0.

2.3 Customisation

The IS needs to be configured and customized to integrate the different EVER-EST components, applications and services.

The various EVER-EST components need to be registered in the IS as service provider and linked to the involved authentication protocol. The Open ID Connect and the SAML 2.0 authentication protocols are involved in the integration of the current EVER-EST components. The following figure highlights one of the configuration steps needed to register one of the EVER-EST components on the IS.

H2020 – EINFRA – 2015 – 1 Page 13 of 37

Figure 2-2 Configuration of a service provider with the different possible authentication protocols.

In addition to the registration processes, the login pages and the other pages like error, notification screens, and the redirection page for SSO needs to be customized for the EVER-EST project.

2.4 Installation guide

The installation / uninstallation procedures of the IS on the dedicated machine are summarised in Annex A.1 of this document.

2.5 Using the service – quick overview

Whenever an application is asked to deliver a service for which the user needs to be authenticated, it redirects the user to the login page of the EVER-EST IS.

Soon after the user logins to EVER-EST, the application that triggered the process receives a certified user identity issued by the identity provider as an automatic means to log the user into the application as well. Next the user is redirected back to the application where the service, requested initially, can now be delivered.

The following figures provides an example of the different phases of the authentication process from the VRE Portal. In the first step the VRE portal is asked to deliver a service for which the user needs to be authenticated, it

H2020 – EINFRA – 2015 – 1 Page 14 of 37

redirects the user to the login page of the EVER-EST IS. There the user is asked to enter its credentials. From then on, if credentials entered were valid, the user is authenticated and in this working session it will not be asked for its credentials any more till it logs out or the session times out. Soon after the user login to EVER-EST, the application (VRE Portal) that triggered such a process receives a certified user identity issued by the IS as an automatic means to log the user into the application as well. Next the user is redirected back to the application where the service, requested initially, can now be delivered.

Figure 2-3 VRE Portal, highlighted the login button that redirects to EVER-EST IS

H2020 – EINFRA – 2015 – 1 Page 15 of 37

Figure 2-4 Login page of the EVER-EST IS

Figure 2-5 Consent page on the EVER-EST IS

H2020 – EINFRA – 2015 – 1 Page 16 of 37

Figure 2-6 User is authenticated and redirected back to the portal

2.5.1 Getting started

To start using the services offered by the IS the generic EVER-EST application needs to be registered in the IS.

The registration process of an Open Id Connect (OIDC) relying party has the following steps:

• Registration of the RPs in the IS (redirect URL and Relying Party name needed).

• Communication of ClientId and secret to the Relying Party.

The registration process of a Service provider using the SAML 2.0 authentication protocol has the following steps:

• Registration of the SP in the IS.

• IS/SP Metadata exchanging.

2.6 Service public API’s The WSO2 Identity Server provides several web services APIs for managing identities. Among them only the APIs exposed to the EVER-EST components are documented within this deliverable. The APIs exposed in the EVER-EST system have the aim to protect some of the subsystem resources. The protection is offered via OAuth2.0 authorization protocol. The OAuth protocol is used to authorize websites or applications to access their resources on other applications. OAuth provides to clients a ‘secure delegated access’ to server resources. In particular OAuth2.0 provides also a specific authorization flow. The EVER-EST components that need to validate the access to their resources can be provided of credentials for using the APIs offered by the validation web service provided by the WSO2 IS. The WSDL of the web service is exposed at URL:https://sso.everest.psnc.pl/services/OAuth2TokenValidationService?wsdl

https://sso.everest.psnc.pl/services/OAuth2TokenValidationService?wsdl

H2020 – EINFRA – 2015 – 1 Page 17 of 37

Figure 2-7Exposed WSDL

The only operation to use for the purpose to validate the OAuth2.0 access token is the “validate”.

2.6.1 The logic This API provides a secure method to protect resources. In the EVER-EST system the ROHUB APIs are the resources that need protection. When an API consumer, in our concern the VRE Portal, want to access these resources the API owner, in our concern the ROHUB, following the OAuth2.0 specification and using the exposed API can validate the access to the resources. Following the OAuth2.0 specification, the API owner (Resource owner) owns the responsibility to prevent unauthorized users from accessing an API (or a Resource in general). For that to be possible the specification prescribes that API consumers send a valid OAuth2.0 access token along with every API call. Such access token must be sent as the HTTPS request parameter. Once the API Server gets the call it needs to validate the access token against the OAuth2.0 Authorization server, before processing the request and replying to the consumer API. The following figure provides an overview of the process, it contains the authentication process, steps 1-8, and the protected API call, steps 9-12.

H2020 – EINFRA – 2015 – 1 Page 18 of 37

Figure 2-8 Authentication Process and API protection

2.6.2 APIs

The OAuth2.0 access token validation web service is provided by WSO2 IS. The only operation to use for the purpose is “validate”.

Operation URL for SOAP v1.1

Soap Action:

urn:validate

https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationServic

eHttpsSoap11Endpoint/

Table 1 Operation name and its endpoint for SOAP request in version 1.1

Type REQUEST SAMPLE

SOAP

1.1

POST

https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationService

H2020 – EINFRA – 2015 – 1 Page 19 of 37

HttpsSoap11Endpoint/ HTTP/1.1

Accept-Encoding: gzip,deflate

Content-Type: text/xml;charset=UTF-8

SOAPAction: "urn:validate"

Content-Length: 579

Host: sso.everest.psnc.pl

Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

Cookie: JSESSIONID=76D417C89BB22FEC0AB76F446B1EE175

Cookie2: $Version=1

Authorization: Basic T0F1dGhUb2tlblZlcmlmaWVyOjgkenVeJ3o=

<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"xmlns:xsd="htt

p://org.apache.axis2/xsd" xmlns:xsd1="http://dto.oauth2.identity.carbon.wso2.org/xsd">

<soapenv:Header/>

<soapenv:Body>

<xsd:validate>

<xsd:validationReqDTO>

<xsd1:accessToken>

<xsd1:identifier>636489a7-8481-33fa-82cc-c5700e277106</xsd1:identifier>

<xsd1:tokenType>Bearer</xsd1:tokenType>

</xsd1:accessToken>

</xsd:validationReqDTO>

</xsd:validate>

</soapenv:Body>

</soapenv:Envelope>

Table 2 Example of a validate action request with SOAP in version 1.1

Operation URL for SOAP v1.2

Soap Action:

urn:validate

https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.O

Auth2TokenValidationServiceHttpsSoap12Endpoint/

Table 3 Operation name and its endpoint for SOAP request in version 1.2

Type REQUEST SAMPLE

SOAP

1.2

POST

https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationService

HttpsSoap12Endpoint/ HTTP/1.1

Accept-Encoding: gzip,deflate

H2020 – EINFRA – 2015 – 1 Page 20 of 37

Content-Type: application/soap+xml;charset=UTF-8;action="urn:validate"

Content-Length: 559

Host: sso.everest.psnc.pl

Connection: Keep-Alive

User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

Cookie: JSESSIONID=6EEE2C8538225FB26984056094A47A22

Cookie2: $Version=1

Authorization: Basic T0F1dGhUb2tlblZlcmlmaWVyOjgkenVeJ3o=

<soap:Envelopexmlns:soap="http://www.w3.org/2003/05/soap-

envelope"xmlns:xsd="http://org.apache.axis2/xsd"

xmlns:xsd1="http://dto.oauth2.identity.carbon.wso2.org/xsd">

<soap:Header/>

<soap:Body>

<xsd:validate>

<xsd:validationReqDTO>

<xsd1:accessToken>

<xsd1:identifier>536489a7-8481-33fa-82cc-c5700e277106</xsd1:identifier>

<xsd1:tokenType>Bearer</xsd1:tokenType>

</xsd1:accessToken>

</xsd:validationReqDTO>

</xsd:validate>

</soap:Body>

</soap:Envelope>

Table 4 Example of a validate action request with SOAP in version 1.2

Status Response

Expire

d

<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">

<soapenv:Body>

<ns:validateResponsexmlns:ns="http://org.apache.axis2/xsd">

<ns:returnxsi:type="ax2354:OAuth2TokenValidationResponseDTO"

xmlns:ax2354="http://dto.oauth2.identity.carbon.wso2.org/xsd"xmlns:xsi="http://www.w3.org/20

01/XMLSchema-instance">

<ax2354:authorizationContextTokenxsi:nil="true"/>

<ax2354:authorizedUserxsi:nil="true"/>

<ax2354:errorMsg>Access token expired</ax2354:errorMsg>

<ax2354:expiryTime>0</ax2354:expiryTime>

<ax2354:scopexsi:nil="true"/>

<ax2354:valid>false</ax2354:valid>

H2020 – EINFRA – 2015 – 1 Page 21 of 37

</ns:return>

</ns:validateResponse>

</soapenv:Body>

</soapenv:Envelope>

Valid <soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">

<soapenv:Body>






<ax2354:authorizedUser>[email protected]</ax2354:authorizedUser>

<ax2354:errorMsgxsi:nil="true"/>


<ax2354:scope>openid</ax2354:scope>

<ax2354:valid>true</ax2354:valid>

</ns:return>


</soapenv:Body>

</soapenv:Envelope>

Invali

d

<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">

<soapenv:Body>






<ax2354:authorizedUserxsi:nil="true"/>

<ax2354:errorMsg>Invalid access token</ax2354:errorMsg>


<ax2354:scopexsi:nil="true"/>

<ax2354:valid>false</ax2354:valid>

</ns:return>


</soapenv:Body>

</soapenv:Envelope>

Table 5 Examples of responses to a validate action request. The three different possibles statuses.

H2020 – EINFRA – 2015 – 1 Page 22 of 37

2.6.3 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running IS instance can be viewed.

• Through the Management Console. The administrator of the IS can access to the Management consol of the Identity Server and monitor the different actions in which IS has been involved. The monitor menu includes a list of features focused on providing logs and statistics related to monitoring the IS. The components of the monitor menu have the aim to monitor system statistics, system logs and trace the SOAP requests. The system statistics page shows certain statistics related to the WSO2 IS instance. These include free memory, request count, server name, server start time, system up time, active services, total memory, average response time, minimum response time, and maximum response time. The system logs page displays information regarding the log files of the current product. Furthermore, it has a feature that allows the administrator to view and download log files according to their preference. The SOAP Tracer shows SOAP messages, SOAP message requests, and SOAP message responses.

• If the Management console is not accessible troubleshooting can be performed through the log files that are stored in the <PRODUCT_HOME>/repository/logs folder. This folder contains current logs in a log file with a date stamp. Older logs are archived in the wso2carbon.log file.

• If the Management console is not accessible troubleshooting can be also performed through the command prompt/shell terminal that opens when running the script to start the server.

H2020 – EINFRA – 2015 – 1 Page 23 of 37

3 WSO2 ESB WSO2 Enterprise Service Bus provides, when it is needed, the bus for the messages exchanged between service provider and service consumer.


Among the common services the ones offered by the ESB and used by the EVER-EST system are briefly detailed here:

1. Message routing, the process of routing requests to correct service provider. 2. Message transformation, the process of transforming message formats between consumer and

provider.

These functionalities are centralized in EVER-EST to integrate the RoHub functionalities with the VRE portal.

The ROHub functionalities are offered to the end user of the VRE Portal using the mediation of the ESB that acts as a middleware.

The ESB performs a transformation of format, serves the ROHub output formats in a format easier to manage from the VRE Portal (xml to json), it routes the different requests from the VRE portal to the correct RoHub endpoint (REST API endpoint or SPARQL endpoint). The ESB also performs a mediation between VRE Portal and ROHub mapping a single call from VRE Portal into N calls to RoHub sending most of them in parallel.

3.2 Design elements

WSO2 ESB is a COTS configurable to work in the EVER-EST environment.

The role of ESB in the EVER-EST service oriented architecture is that of an enabler of communications among systems and components both currently involved in VRE and, taking in due care extensibility concern, upcoming in future evolutions of VRE. Such enablement consists in leveraging extensive connectivity technologies supported by WSO2 ESB thus making it possible to integrate very heterogeneous components. At the same time the use of ESB on VRE has been inspired by a low-intrusion philosophy, whereby components endowed with a mature level of communications technology are allowed to interoperate directly. The detailed analysis of existing components carried out during the design phase has thus reduced the need for ESB as a communication enabler just between the exposed RoHub functionalities and the VRE portal. The purpose of using ESB is twofold, concerning Design and Performance issues. About the design the ESB allows to streamline the usage of ROHUB services, repurposing them to the EVER-EST user-action semantics (Service Façade pattern). About the performance the ESB allows to gain in user-perceived speed of processing, splitting complex user actions commanded by the VRE portal into smaller units taken care of by the ESB asynchronously and concurrently (Mediation pattern).

H2020 – EINFRA – 2015 – 1 Page 24 of 37

Figure 3-1The Service BUS

3.3 Customisation

The ESB needs to be configured and customized to integrate the different EVER-EST components, applications and services. The major configuration is the message mediation. This functionality can be any combination of advanced mediations such as transformation and content-based routing. The configurations have been done to allow format management, routing, mediation and trasformation.

For the format management the transformations configured allow to transform the output format from ROHUB, XML, in JSON format for the VRE portal.

About the routing, the configuration done allows to redirect the requests among the different ROHUB backend (ROHUB REST API endpoint or ROHUB SPARQL endpoint) depending on the request from the VRE portal

About the mediation, the ESB is configured to do a mediation mapping one call on VRE to several calls on ROHUB.

Concerning the transformation the ESB sanitizes, following the ROHUB specifications, the filters coming from the VRE portal before passing them to RO.


The installation / uninstallation procedures of the ESB on the dedicated machine are summarised in Annex A.2 of this document.


The use of the service is completely hidden to the final user. The VRE portal will use the Façade API exposed by the ESB.

3.6 Service public API’s

The service public API’s are described in the EVER-EST DEL WP5-D5.3.

3.6.1 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running ESB instance can be viewed.

H2020 – EINFRA – 2015 – 1 Page 25 of 37

• Through the Management Console. The administrator of the ESB can access to the Management consol of the ESB and monitor the different actions in which ESB has been involved. The monitor menu includes a list of features to monitor and manage the server runtime. It is possible to see mediation tracing, mediation statistics and system logs. You can monitor messages that are mediated through the ESB using the Mediation Tracer in the management console. Mediation statistics allow a server administrator to collect and view runtime statistical information from sequences, proxy services, and endpoints through the ESB management console.


• If the Management console is not accessible troubleshooting can be also performed through the command prompt/shell terminal that opens when running the script to start the server.

H2020 – EINFRA – 2015 – 1 Page 26 of 37

4 WSO2 DAS WSO2 DAS and WSO2 ESB, working in synergy with each other, offer data analytics functionalities. WSO2 ESB is the component that mainly will be used to send event to WSO2 DAS for processing. The role of the ESB in this case is that of a data collector from the various systems that make up EVER-EST infrastructure.


The common service offered by the conjunction of DAS and ESB for EVER-EST is the auditing service.

This service, can show who has accessed the system and what operations he or she has performed during a given period of time. Auditing is useful both for maintaining security and for monitoring system performances.

4.2 Design elements

The WSO2 DAS and ESB are COTS configurable to work in the EVER-EST environment. The WSO2 DAS is a platform where different kinds of data analytics can be performed:

• Real-time: thanks to an embedded CEP engine “WSO2 Siddhi”, data received by DAS can be processed on the fly, with or without persistence.

• Batch: with the support of Apache Spark, scheduled batch processing can be implemented on data received and persisted on DAS.

• Interactive: with the support of indexing and searching power of Apache Lucene, users are able, on demand, to query data or monitor activities.

• Predictive: thanks to the embedded WSO2 Machine Learner, using persisted data as source dataset for the learning, new data sent to DAS are used to predict future events.

The WSO2 ESB is the component that will be mainly used to send event to WSO2 DAS for processing. The ESB plays, in this case, a role of a data collector from the various systems that made up EVER-EST infrastructure. Data will be collected by the ESB mainly through listening on log files and data repositories while such data are produced or as a second option (for systems that do not have a data repository shareable for the purpose) through receiving real-time data feeds via HTTP directly from the systems. Once ESB gets the data it sends such collected information to WSO2 Data Analytics Server where data analysis can take place.

H2020 – EINFRA – 2015 – 1 Page 27 of 37

Figure 4-1The WSO2 DAS architecture.

4.3 Customisation

The DAS and ESB need to be configured and customized to integrate the different EVER-EST components, applications and services.

The aggregation of data from the different components can be done configuring an event receiver as in the following figure.

Figure 4-2How to create an event receiver in DAS

H2020 – EINFRA – 2015 – 1 Page 28 of 37

Section Description

To The event stream from which the event receiver will fetch the events for processing.

Mapping

Configuration

Specific properties of the selected input event adapter.

From The event stream from which the event receiver will fetch the events for processing

Adapter

Properties

The format of the message that is received. You can configure custom mappings on the selected

format via advanced settings.

Table 6 Description of the required fileds to configure an event receiver in DAS


The installation / uninstallation procedures of the DAS on the dedicated machine are summarised in Annex A.3 of this document.


After collecting data, and performing various analysis on them to produce meaningful information, the final usage of the service is to present this information. WSO2 DAS queries the analysed information and shows it in various UIs. There are different ways to query and present analysed information, the ones selected for EVER-EST are:

1. Visualizing Results in WSO2 DAS, access the WSO2 DAS Analytics Dashboard, create new dashboard and new gadget or visualize the existing ones.

2. Communicating Results Through REST API, WSO2 DAS stores the received events and processed data in its underlying data storage system, where that data can be retrieved using the standards. Analytics REST API allows external parties to query this data.


The interface of the analytics data service as exposed as REST APIs.

4.6.1 The logic

Analytics REST API allows external parties to query this data.

4.6.2 DAS APIs for EVER-EST system

The REST APIs are secured with basic authentication and all the Response are in JSON format.

Function REST API

Checking if a table exists GET /analytics/table_exists?table={tableName}

Checking if the Given Records Store Supports

Pagination

GET /analytics/pagination/<recordstore-name>

Drilling down through hierarchical categories POST /analytics/facets

Getting the record count of a table GET /analytics/tables/{tableName}/recordcount

Getting the search record count of a table POST /analytics/search_count

H2020 – EINFRA – 2015 – 1 Page 29 of 37

Listing all tables GET /analytics/tables

Retrieving Aggregated Values of Given Records POST /analytics/aggregates

Retrieving All Record Stores GET /analytics/recordstores

Retrieving matching records through a drill down

search

POST /analytics/drilldown

Retrieving records of a table GET

/analytics/tables/{tableName}/{from}/{to}/{start}/{count}

Retrieving the count of matching records through a

drill down search

POST/analytics/drillDownScoreCount

Searching records of a table POST /analytics/search

Table 7 Description of the APIs provided by DAS.

Params Value

{tableName} The name of the table that you want to find in the DAS server.

{from} The starting time to retrieve records from. This is an optional

parameter.

{to} The ending time up to which records should be retrieved. This is

an optional parameter.

{start} The paginated index from value. This is an optional parameter.

{count} The paginated records count to be read. This is an optional

parameter.

{timeout} The timeout in seconds after waiting until the process completes.

{recordstore-name} The name of the record store for which you need to carry out a

check.

Table 8 The parameters used in the different APIs listed and explained.

Status Response

success 20*

error 40*/50*

Table 9 The possibles error codes from the APIs

4.6.3 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running DAS instance can be viewed.

• Through the Management Console. The administrator of the DAS can access to the Management console of the DAS and monitors the different actions in which DAS has been involved. The monitor menu includes a list of features to monitor and manage the server runtime. From the Monitor menu it is possible monitor events statistics and trace events. Event Statistics is an important feature which helps monitoring purposes of events.

https://docs.wso2.com/display/DAS300/Retrieving+All+Record+Stores+via+REST+API

https://docs.wso2.com/display/DAS300/Retrieving+Records+Based+on+a+Time+Range+via+REST+API

H2020 – EINFRA – 2015 – 1 Page 30 of 37

This presents realtime requests and responses vs time for all the incoming and outgoing topics. DAS Event Tracer provides huge functionality to trace the event in each and every component. Event tracer will help to check the event when travels along the components.


• If the Management console is not accessible troubleshooting can also be performed through the command prompt/shell terminal that opens when running the script to start the server.

H2020 – EINFRA – 2015 – 1 Page 31 of 37

5 SeaFile SeaFile1 offers a repository for the personal data storage. EVER-EST tools and services need to be able to reference and access these resources.


Users need a space where they can store complementary or auxiliary resources (e.g., documents, scripts, papers, intermediate results, etc.)

5.2 Design elements

All data, files, documents, scripts, uploaded in Seafile are available through a Web link, so that they can be referenced, particularly within a research object. More in general, EVER-EST applications and services can reference and access the stored resources. The solution is economic and scalable, and the component is able to increase its storage capacity seamlessly.

Seafile guarantees, among other features, file synchronisation, version control, public link sharing, desktop client and web API.

5.3 Customisation

A Seafile instance for EVER-EST is accessible at https://box.everest.psnc.pl/. This instance of Seafile is integrated with the EVER-EST Identity Service Provider by means of the SAML 2.0 authentication protocol. Possible future configurations/customisation are definition of user quotas or increasing of the initial size (5TB).


For a complete guide of Seafile Server, including service installation and uninstallation procedures:

https://www.gitbook.com/book/freeplant/seafile-server-manual/details


To start to use Seafile functionalities as EVER-EST user you need to be logged in the EVER-EST system.

SeaFile Web Services will be accessed in an authenticated session, the web interface will appear as in the following figure.

1 https://www.seafile.com/en/home/

https://www.gitbook.com/book/freeplant/seafile-server-manual/details

H2020 – EINFRA – 2015 – 1 Page 32 of 37

Figure 5-1Seafile Web Interface

The SeaFile desktop application can be downloaded to have the complete functionalities offered by the server component. It guarantee a reliable file Syncing, high performance, and easy upgrade. The client application will appear as in the following figure.

Figure 5-2Seafile desktop application


The other services, applications and components can access Seafile's functions remotely via REST calls. All API calls must be authenticated. The offered APIs are WEB, python and PHP respectively well documented at the following link:

https://manual.seafile.com/develop/web_api_v2.1.html

https://manual.seafile.com/develop/web_api_v2.1.html

H2020 – EINFRA – 2015 – 1 Page 33 of 37

https://manual.seafile.com/develop/python_api.html

https://github.com/rene-s/Seafile-PHP-SDK

5.6.1 Troubleshooting

If the installation did not finish successfully, the following file can be verified for errors: /root/[installer_name]_installation.log.

https://manual.seafile.com/develop/python_api.html

https://github.com/rene-s/Seafile-PHP-SDK

H2020 – EINFRA – 2015 – 1 Page 34 of 37

Annex A Installation Guides

A.1. WSO2 IS Installation guide

1. On the machine dedicated to IS:Download the Identity Server2 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred

to as <IS_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit

(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service 6. Install the Apache Web Server to put in front of the IS 7. Configure the Apache Web Server to be a proxy server for the IS

A.1.1 Software prerequisites

Application Purpose and Version

Oracle Java

SE

Development

Kit (JDK)

To launch the product as each product is a Java application.

Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running.

Web Browser To access the product's Management Console. The Web Browser must be JavaScript

enabled to take full advantage of the Management console

Table 10 WSO2 IS software prerequisites

A.1.2 Hardware prerequisites

Memory • ~ 2 GB minimum

~ 512 MB heap size. This is generally sufficient to process typical SOAP messages but

the requirements vary with larger message sizes and the number of messages processed

concurrently.

Disk • ~ 1 GB, excluding space allocated for log files and databases.

Table 11 WSO2 IS hardware prerequisites

A.1.3 Uninstallation procedure

1. Uninstall the service 2. Delete system properties 3. Remove the <IS_HOME>

2 http://wso2.com/products/identity-server/#download. At the time of writing this document, version 5.2 is available.

https://docs.wso2.com/display/IS520/Running+the+Product

http://wso2.com/products/identity-server/#download

H2020 – EINFRA – 2015 – 1 Page 35 of 37

4. Remove the Apache Web Server configuration for the IS

A.2 WSO2 ESB Installation guide

On the machine dedicated to ESB:

1. Download the Enterprise Service Bus3 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred

to as <PRODUCT_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit

(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service



Oracle Java

SE

Development

Kit (JDK)


Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running

(not building).



Table 12 WSO2 ESB software prerequisites



• ~ 1 GB heap size. This is generally sufficient to process typical SOAP messages but the

requirements vary with larger message sizes and the number of messages processed concurrently.


Table 13 WSO2 ESB hadware prerequisites



1. Uninstall the service 2. Delete system properties

4 http://wso2.com/products/enterprise-service-bus At the time of writing this document, version 5.0 is available


http://wso2.com/products/enterprise-service-bus

H2020 – EINFRA – 2015 – 1 Page 36 of 37

3. Remove the <PRODUCT_HOME>

A.3 WSO2 DAS Installation Guide

On the machine dedicated to DAS:

1. Download the Data Analytics Server4. 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred

to as <DAS_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit

(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service




Oracle Java

SE

Development

Kit (JDK)



(not building).



JDBC-

compliant

Connector

for Java

Required as a standardized database driver for Java platforms and development.

Supported Version: 1.7.0 or later

Table 14 WSO2 DAS software prerequisites


Oracle Java

SE

Development

Kit (JDK)



(not building).



Table 15 WSO2 ESB software prerequisites

4 https://docs.wso2.com/display/DAS310/Downloading+the+Product At the time of writing this document, version 3.1 is available



https://docs.wso2.com/display/DAS310/Downloading+the+Product

H2020 – EINFRA – 2015 – 1 Page 37 of 37




• ~ 4 GB minimum


Table 16 WSO2 DAS hardware prerequisites



• ~ 1 GB heap size. This is generally sufficient to process typical SOAP messages but the

requirements vary with larger message sizes and the number of messages processed concurrently.


Table 17 WSO2 ESB hardware prerequisites



1. Uninstall the service 2. Delete system properties 3. Remove the <DAS_HOME>