22
LOSD Publication and Capacity Building D4.1 Complete Documentation of the LOSD Service Date: 8 th April 2019 Web: www.derilinx.com Email: [email protected] Address: 11/12 Baggot Court, Dublin 2, D02 F891 Tel: +353 (0)1 254 4316 Mob: +353 (0)87 417 2318 Twitter: @derilinx This project has received funding from the ESSnet LOS under Grant Agreement No 11101.2017.001-2017.66

LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD Publication and Capacity Building

D4.1 Complete Documentation of the LOSD

Service

Date: 8th April 2019

Web: www.derilinx.com

Email: [email protected]

Address: 11/12 Baggot Court,

Dublin 2, D02 F891

Tel: +353 (0)1 254 4316

Mob: +353 (0)87 417 2318

Twitter: @derilinx

This project has received funding from the ESSnet LOS

under Grant Agreement No 11101.2017.001-2017.66

Page 2: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

2

Table of Contents

1 Revision History 3

2 Introduction 4

2.1 Purpose and Target Group of the deliverable 4

2.2 Relation to other Activities in the Project 4

2.3 Document outline 4

3 Complete LOSD Service 5

4 LOSD Publication Pipeline Installation and Setup 7

4.1 CKAN 7

4.2 Juma 8

4.3 Juma API 11

4.4 Juma Docker Image 12

4.5 Cube Visualizer 13

4.6 Cube Visualizer Docker Image 13

4.7 OLAP Browser 14

4.8 OLAP Browser Docker Image 14

4.9 JSON-QB API 14

4.10 JSON-QB API Docker Image 15

4.11 Virtuoso 15

4.12 Moodle 16

5 JSON-STAT to RDF Export 17

6 JSON-STAT to RDF API 21

7 LOSD Example Application 22

8 Conclusion 22

Page 3: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

3

1 Revision History

Version Date Authors Status Description of Changes

0.1 21/01/2019 Alan Meehan Draft Initial outline

0.2 31/03/2019 Deirdre Lee Draft Added sections on JSON-Stat export

0.3 03/04/2019 Deirdre Lee Final Finalised

Page 4: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

4

2 Introduction

This LOSD project is being delivered in the context of the European Statistical System’s (ESS) DIGICOM

project. DIGICOM contributes to two key areas of the ESS vision: (i) focus on users, and (ii) improve

dissemination and communication1. One of the four DIGICOM work packages is Open Data

Dissemination2. The objective of the Open Data Dissemination work package is to facilitate automated

access to European aggregate data for heavy users or re-disseminators and to improve access to

micro-data.

2.1 Purpose and Target Group of the deliverable

This deliverable provides complete documentation of the LOSD Service, including the LOSD Publishing

Pipeline and the LOSD Service Architecture.

The main target group of this deliverable are users who wish to create a new LOSD publication

pipeline.

2.2 Relation to other Activities in the Project

This deliverable is related to just about every other activity in the project as it specifies how to install

and setup the pipeline on which many of the other activities in the project are based.

2.3 Document outline

The remainder of this document is organised as follows:

● Section 2 presents the installation and setup of the various software tools within the LOSD

publication pipeline.

● Section 3 wraps up with a quick conclusion.

1 http://ec.europa.eu/eurostat/web/ess/about-us/ess-vision-2020/implementation-portfolio#DIGICOM 2 http://ec.europa.eu/eurostat/web/ess/digicom

Page 5: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

5

3 Complete LOSD Service

The complete LOSD Publishing Pipeline service is available at https://losd-data.staging.derilinx.com/

The LOSD Publishing Pipeline service facilitates all of the capability outlined at the beginning of the

project.

We have demonstrated this through the successful implementation of three LOSD use cases, namely:

● Use Case 1: NSO Statisticians Publishing LOSD

● Use Case 2: General Public Users Accessing LOSD

● Use Case 3: NSO Collaboration with 3rd Party Partner for Developing Added-Value Services

Page 6: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

6

The implementation of each of the use cases are described in detail in Deliverable 3.3 Implementation

of Use-cases from NSO Partners. The LOSD Publishing Pipeline comprises of the following components,

which are utilised corresponding to the specific use-case.

CKAN, LOSD Data Hub

CKAN serves as a data catalog for datasets and resources used in the LOSD project. It is the

front end for using the rest of the LOSD pipeline and to access datasets, Linked Data tools and

training materials.

Juma & Juma API

Juma is a CSV to RDF mapping and conversion tool build on top of R2RML technology - which

is a standard for the conversion of tabular data to RDF. It allows users with little RDF

knowledge to create their mappings using blocks.

Virtuoso Triple Store

Virtuoso has been added to the pipeline to expose data via a SPARQL endpoint. It hosts all the

RDF data generated by the LOSD pipeline. Virtuoso not only provides a GUI which allows users

to manually type queries and see the resulting data, it also provides an API for data access

which is important for data access via applications.

Cube Visualizer

Cube Visualizer is a visualization and visual analytics tool. It allows user to create and present

graphical visualizations of RDF data cube's one-dimensional slices. Users can interact with the

User Interface by choosing measure, free dimension and dimension values, which later are

transformed into appropriate API requests. Responses from the API are presented to the user

in a form of chart - bar chart, pie chart or area chart.

OLAP Browser

The OLAP Browser enables performing OLAP (Online Analytical Processing) operations (e.g.

pivot, drill-down, and roll-up) on top of multiple linked data cubes. The tool is using API

requests in order to receive the data based on user selection.

Page 7: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

7

JSON-stat to RDF Converter (UI/API)

The JSON-stat to RDF Converter allows users to convert statistical JSON-stat data to Linked

Data n-triples. Conversion can either be done through the CKAN UI or through an API. UI

conversion enables users to create an RDF file with no knowledge of Linked Data. An API can

also be used to convert valid JSON-stat to RDF files. The response from the API is presented

to the user either in RDF/XML or JSON-LD format.

A detailed description of the LOSD Publishing Pipeline components are available in D3.2 Centralised

LOSD Publication Platform.

4 LOSD Publication Pipeline Installation and Setup In this section, we present the requirements for the installation and setup of each component of the

LOSD publication pipeline. For each tool the required software for each tool is specified, configuration

and basic usage.

4.1 CKAN

Every component of CKAN is dockerised - CKAN installation can be divided into following categories -

Server/System Information:

● Linode - Ubuntu 16.04LTS

● Ram: 4GB

● Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

Required Software:

● CKAN

● LOSD-CKAN extensions

● PostgreSQL - v10.4

● Nginx - v1.13.12

● Solr

● Docker - v18.03

● Docker compose - v3.4

● Core CKAN - v2.8.1 and the required extensions

● PostgreSQL - v10.4

● Nginx - v1.13.12

The above-mentioned core CKAN and extensions are available for download at GitHub page:

Link: https://github.com/derilinx/losd-ckan

Branch: losd-ckan

The process of installing all the required software and packages are automated by docker compose.

Page 8: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

8

Configuration and Running of losd-ckan:

● Install Docker and Docker compose.

● Login to appropriate docker hub / repository.

● Clone the losd ckan code from the link: [email protected]:derilinx/ckan-docker-compose.git

● Switch to branch losd-ckan.

● Make an env file by running the script env.py.

● Type the command Make - will fetch all the necessary extensions and required software from

git/docker repository and dockerise each of the components.

○ docker-compose.yml file builds total of 8 docker containers -

■ ckan: which holds the core ckan and losd-extension

■ nginx-proxy: contains nginx for proxying and routing.

■ nginx-proxy-letsencrypt-sidecar: contains required ssl certificates for losd

website.

■ db: ckan database (postgreSQL DB).

■ datastore: contains postgres database called datastore which is used to

display data previews.

■ solr: this container contains solr which is used for indexing the contents of db

and resources.

■ datapusher: web service that pushes data files from CKAN sites resource into

datastore.

■ reddis: this is part of ckan to run time consuming tasks as background jobs.

4.2 All Applications as docker compose- Option 1 (except CKAN):

4.2.1 Application Setup Instructions:

Installing Applications using Docker and Docker Compose:

This installation process involves the following applications:

• Juma • Juma API • Cube Visualizer • OLAP Browser • JSON QB API • Virtuoso • Moodle • RDF Conversion API • Data Visualization App

Server/System Information:

1. Linode - Ubuntu 16.04LTS 2. Ram: 4GB 3. Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GH 4. Internet Access

Page 9: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

9

Required Software:

1. Docker - v18.03 2. Docker compose - v3.4

Installation Steps:

• Clone the GitHub repository to a folder called “losd-applications-docker-compose”

https://github.com/derilinx/losd-applications-docker-compose

• git submodule init • git clone --recurse-submodules

This will clone all submodules to the respective folders. All submodules are available in below GitHub Repo:

• Juma Editor and API: [email protected]:derilinx/juma-losd-docker_sp.git • Cube Visualizer: https://github.com/LOSD-Data/docker-cube-visualizer • OLAP Browser: https://github.com/LOSD-Data/docker-olap-browser.git • JSON AB API: https://github.com/LOSD-Data/docker-json-qb-api.git • Virtuoso: Used as a docker image - tenforce/virtuoso:1.3.1-virtuoso7.2.2 • Moodle: [email protected]:derilinx/docker-moodle.git • RDF Conversion API: [email protected]:derilinx/docker-rdfapi.git • Data Visualization App: [email protected]:derilinx/docker-dataviz.git

• Once all submodule is cloned to respective folder create a file called “.env”. This file contains all necessary environment variables, username and password required for the each of the applications. Parameters used in environment variables are given in the GitHub README.md file

(https://github.com/derilinx/losd-applications-docker-compose/tree/master).

• Create a file called “shiro-users.properties” in the path “juma-losd-docker_sp/Juma-losd-docker_sp/juma-uplift-master/src/main/resources”. This file is used for creating a username and password for Juma Editor. Further details are give in GitHub README.md file

(https://github.com/derilinx/losd-applications-docker-compose/tree/master).

• Go to the folder “losd-applications-docker-compose” • Create and folders to hold the data (As Volume in docker-compose.yml file) of moodle,

virtuoso and juma.

- Volume for Moodle:

- "./moodle_data/mysql:/var/lib/mysql"

- "./moodle_data/moodledata:/var/moodledata"

- "./moodle_data/config.php:/var/www/html/moodle/config.php"

- Volume for Virtuoso:

- "./data/virtuoso:/data"

- Volume for Juma:

- "./mysqldata/mysql:/var/lib/mysql"

Page 10: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

10

- "./mysqldata/mysql-files:/var/lib/mysql-files"

- "./mysqldata/mysql-keyring:/var/liv/mysql-keyring"

• Type the command “make build” • Then type “make”. This should bring all the applications up and running.

Note: All the required packages are of each applications are handled by docker and please make sure that you have an internet in the server. Further details on creating .env and shiro-users.properties are given in the gitHub link:

https://github.com/derilinx/losd-applications-docker-compose/tree/master

4.2.2 PORTS Used:

All applications run on different ports inside docker container and it is served by web server Nginx on port 443 (SSL) for the users. Details on ports are given below:

NGINX Listens to port 443 (SSL) and directs the requests to appropriate docker containers:

• Moodle request: https://losd-dck.staging.derilinx.com/moodle/ to container moodle exposed on port 80

• Virtuoso request: https://virtuoso.losd-dck.staging.derilinx.com/ to container virtuoso exposed on port 8890

• Cubeviz request: https://cubeviz.losd-dck.staging.derilinx.com/ to container cubeviz exposed on port 8000

• OLAP request: https://olap.losd-dck.staging.derilinx.com/ to container olap exposed on port 8000

• JSON-QB-API request: https://json-qb-api.losd-dck.staging.derilinx.com to container json-qb exposed on port 8000

• Data Viz Request: https://dataviz.losd-dck.staging.derilinx.com/ to container data-viz exposed on port 5000

• Juma editor and Juma API request:

https://juma.losd-dck.staging.derilinx.com/juma-editor/login and

https://juma.losd-dck.staging.derilinx.com/juma-api/ to container juma exposed on port 8889

• RDF API request: https://rdfapi.losd-dck.staging.derilinx.com/convert_to_rdf to container rdfconv_api on port 5000.

Note:

It is recommended to use OPTION 1 (i.e using docker and docker -compose). Because all packages and its versions are managed by docker containers.

Page 11: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

11

4.3 Running as Independent Applications - Option 2:

4.3.1 Juma

Juma is available for download from the following GitHub page:

https://github.com/Al-Meehan/juma-losd

Required Software:

● MySQL Server v5.7

● Apache Maven v3.2.5 (or later)

● Java v1.7 (or later)

Configuration and Running of Juma:

How to configure and run Juma:

1. Extract Juma to desired directory.

2. You need to specify connection details to mysql server for Juma. Open the "../JUMA/juma-uplift-master/src/main/resources/hibernate.cfg.xml" file and specify your mysql connection URL, username and password.

3. To add users for Juma, edit the "../JUMA/juma-uplift-master/src/main/resources/shiro-users.properties" file. Each user must be on a separate line and the syntax to declare a new user is as follows: "user.name = password". So for a user "foo" with password "bar" it would be specified as: "user.foo = bar".

4. To Run Juma, navigate to "../JUMA/juma-uplift-master". Run the following command: "mvn jetty:run", this will run Juma on the default port which is 8080. To run Juma on a specific port, run the following command instead: "mvn jetty:run -Djetty.port=xxxx".

5. In a web browser, go to http://localhost:8080 (or to whatever port you specified).

4.3.2 Juma API

Juma API is available for the following GitHub page:

https://github.com/Al-Meehan/juma-api

This tool provides a RESTful API for the Juma Editor (LOSD Version) which allows data to be quickly

turned into RDF through a mapping created in Juma.

Required Software:

● Apache Webserver

● php 7

● Juma Editor (LOSD Version)

Configuration of the juma-api:

1. Extract "juma-api" to webserver root directory.

Page 12: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

12

2. For the juma-api to communicate with the Juma Editor, you need to specify the path to the Juma Editor base directory. Open the "../juma-api/index.php" file and specify the path of the Juma Editor base directory for the "$path_to_juma_base" variable. For example, if Juma Editor is located at "/home/user1/" then the following should be specified: "$path_to_juma_base = '/home/user1/Juma/juma-uplift-master/';".

Usage of the juma-api:

● To use the juma API, POST your request to the address of the api with three parameters - which are specified via tokens. These three parameters are:

o Juma Editor user name (?user)

o Juma Editor mapping id number (?map)

o URL of the CSV source data that is to be converted into RDF (?source)

● The result from the api will be RDF data (RDF version of the CSV data in the source file, specified according the mapping from the Juma Editor) if all parameters are correct - otherwise it will return an error specifying what has gone wrong.

● An example call to the api, where the Juma Editor user name is "user1", the mapping ID number is "10" and the URL of the source CSV data is "http://example.com/data/csvfile1", would be as follows:

o 'POST /juma-api?user=user1&map=10&source=http://example.com/data/csvfile1'

4.3.3 Juma Docker Image

A docker image of Juma is also available which includes both the Juma editor and the Juma API (along

with all the required software for those two services) which is available on GitHub:

https://github.com/Al-Meehan/juma-losd-docker_sp .

Building and running the Juma docker container

To build the docker image use the following command:

docker image build -t juma_image .

When the image has been built, a container can be run using the following command:

docker run --name juma_container -p 8889:(HOST_PORT) -t juma_image

Both of the services run on port 8889 of the container. To map the container to a specific port of the

host, alter the above command so that (HOST_PORT) is replaced with the desired host port. For

example if you want the container to run on port 80 of the host - then change ‘-p

8889:(HOST_PORT)’ to ‘-p 8889:80’.

Configuration and Usage:

When the container is running: ● the Juma Editor service is available on:

http://(IP|domain):(HOST_PORT)/juma-editor/login

● the Juma API service is available on : http://(IP|domain):(HOST_PORT)/juma-api

Page 13: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

13

To add users to the Juma Editor, you will first need to connect to the container. This can be done with the following command:

docker exec -it juma_container bash

Once connected, edit the "../JUMA/juma-uplift-master/src/main/resources/shiro-users.properties" file. Each user must be on a separate line and the syntax to declare a new use is as follows: "user.name = password". So for a user "foo" with password "bar" it would be specified as: "user.foo = bar".

For usage of the Juma-API see Section 3.3.

4.3.4 Cube Visualizer

The Cube Visualizer is a web application that creates and presents to the user graphical

representations of an RDF data cube’s one-dimensional slices. It is built as a client of the JSON QB REST

API implementation. User’s choices (measure, free dimension and dimension values) are translated to

appropriate API calls. The returned data are presented to the user in the form of a chart, the type of

which (bar chart, pie chart, sorted pie chart, area chart) can be also selected.

Cube Visualizer code is available at:

https://github.com/LOSD-Data/CubeVisualizer

Requirements:

● Webserver

The simplest way to deploy and test the code is by using Python SimpleHTTPServer (simple HTTP server

that provides standard GET and HEAD request handlers):

python -m SimpleHTTPServer [port]

Configuration options:

The configuration file is located at ‘js/config.js’. Before the deployment, ‘jsonqbAPIuri’

needs to be specified. CubeVisualizer requires a JSON-stat API instance in order to process the Data

Cubes.

The additional configuration allows the user to change error messages text and the colours scheme.

4.3.5 Cube Visualizer Docker Image

A Docker image of CubeVisualizer (running on Alpine Linux) is available on GitHub:

https://github.com/LOSD-Data/docker-cube-visualizer

Usage

Update the configuration file config.js and provide JSON-QB API endpoint URL -

jsonqbAPIuri. See more at: https://github.com/LOSD-Data/docker-json-qb-api

Build an image from a Dockerfile:

docker build -t cube-visualizer .

Run Docker container:

Page 14: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

14

docker run -p 8000:8000 cube-visualizer

4.3.6 OLAP Browser

OLAP Browser is a web-based tool that enables data analysts to reuse existing data stored as RDF Data

cubes by presenting a two-dimensional slice of the cube as a table and enabling OLAP operations (roll-

up, drill-down, pivot, dimension reduction etc.)

OLAP Browser code is available at:

https://github.com/LOSD-Data/qb-olap-browser

Requirements:

● Webserver

The simplest way to deploy and test the code is by using Python SimpleHTTPServer (simple HTTP server

that provides standard GET and HEAD request handlers):

python -m SimpleHTTPServer [port]

Configuration options:

The configuration file is located at ‘resources/config.js’. Before the deployment,

‘jsonqbAPIuri’ needs to be specified. OLAP Browser requires a JSON-stat API instance in order

to process the Data Cubes.

4.3.7 OLAP Browser Docker Image

A Docker image of OLAP Browser (running on Alpine Linux) is available on GitHub: https://github.com/LOSD-Data/docker-olap-browser

Usage

Update the configuration file config.js and provide JSON-QB API endpoint URL - jsonqbAPIuri. See more at: https://github.com/LOSD-Data/docker-json-qb-api

Build an image from a Dockerfile:

docker build -t olap-browser .

Run Docker container:

docker run -d -p 8000:8000 olap-browser

4.3.8 JSON-QB API

JSON-qb API aims to provide an easy to use API for developers that reuse statistical data stored in the

form of RDF Data Cubes. The API implementation can be installed on top of any RDF repository and

offer basic and advanced operations on RDF Data Cubes.

Installation:

1. Clone the project

2. Add the following to pom.xml

<plugin>

<groupId>org.eclipse.jetty</groupId>

Page 15: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

15

<artifactId>jetty-maven-plugin</artifactId>

<version>9.2.11.v20150529</version>

</plugin>

3. move resources to src/main/resources

4. run the following command

mvn clean install -Dmaven.clean.failOnError=false

(in case pulling new updates)

5. Configure the address of SPARQLservice in config.prop

6. finally, run

sudo mvn -Djetty.http.port=8001 jetty:run-war

Availability:

The source code can be downloaded from:

https://github.com/LOSD-Data/json-qb-api-implementation

4.3.9 JSON-QB API Docker Image

A Docker image of JSON-QB API (running on Alpine Linux) is available on GitHub: https://github.com/LOSD-Data/docker-json-qb-api

Usage

Update the configuration file config.prop and provide SPARQL endpoint URL - SPARQLservice.

See more at: https://github.com/LOSD-Data/json-qb-api-implementation and Section 3.11 Virtuoso

Build an image from a Dockerfile:

docker build -t json-qb-api .

Run Docker container:

docker run -d -p 8000:8000 json-qb-api

4.3.10 Virtuoso

Virtuoso is an efficient, reliable and high-performance database. It provides a SPARQL endpoint with

SPARQL 1.1 capabilities, to which queries are posed against in order to retrieve data. It was used as

the database to store the LOSD for the tools described above. The download instruction is available

at http://vos.openlinksw.com/owiki/wiki/VOS/VOSDownload

Server/System Information:

● Windows / Linux / Unix: AIX, HP-UX, Solaris, etc. / Mac OS X

● The Virtuoso database requires a minimum of 64Mb of system memory for each instance to

operate in.

● A typical installation will require a minimum of 400Mb of hard disk space to install the code,

samples, documentation and sample database. The database will need additional space for

data inserted, backups, logs and reports, web pages, etc.

Page 16: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

16

4.3.11 Moodle

Moodle is a free and open-source learning management system. It is primarily developed in Linux

using Apache, PostgreSQL/MySQL/MariaDB and PHP.

Server/System Information:

● Ubuntu 16.04LTS

● Disk space: 200MB for the Moodle code, plus additional storage for the learning resources.

● Processor: 1GHz (min), 2GHz dual core or more recommended.

● Memory: 512MB (min), 1GB or more is recommended.

Detailed installation details are available at https://docs.moodle.org/36/en/Installing_Moodle

Page 17: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

17

5 JSON-STAT to RDF Export A new addition to the LOSD Publishing Pipeline is the capability to export JSON-stat data automatically

as RDF. This functionality is added to the LOSD Data Hub (CKAN). The following steps show how a user

can convert a JSON-stat dataset via the LOSD Data Hub UI.

Add a JSON-stat dataset to the LOSD portal using the ‘Add Dataset’ button. A user has to be logged in

to be able to add datasets https://losd-data.staging.derilinx.com/dataset

Open the new JSON-stat dataset that you have just added. There will be a ‘Convert this resource to

RDF’ section. Complete the following fields:

● Dataset ID (A human-readable name for the new RDF dataset. Do not include any white spaces

or special characters)

● Vocabulary Namespace ()

● Dataset Namespace

Click on the ‘Convert to RDF’ button.

Page 18: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

18

You will be redirected back to the dataset page with a message that the ‘RDF file being created. Please

visit the dataset page after a few moments.’ The time it takes to generate the RDF file depends on the

size of the JSON-stat dataset.

When you refresh the page, you will see the newly created RDF file. Click on the dataset to review it.

Page 19: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

19

To view the data, click on the ‘Download’ button or on the URL link.

Page 20: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

20

To push the RDF data to the Virtuoso datastore, complete the ‘Push the resource to RDF Store’ section.

Page 21: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

21

6 JSON-STAT to RDF API A new addition to the LOSD Publishing Pipeline is the capability to convert JSON-stat data to RDF. This

functionality is independent of the LOSD Data Hub (CKAN). The following steps show how a user can

convert a JSON-stat dataset via the API. The usage instruction is available in https://RDFapi.losd-

dck.staging.derilinx.com/convert_to_RDF

This API allows users to give input either as JSON-stat content or as a URL to a JSON-stat file. In

addition, users can also define the desired output format (either as text or as JSON-LD standard).

Parameter Description:

• OutputFormat: Either text or JSON-LD

• DataNmSpace: This is the data name space that will be used in the converted RDF. It should

be a valid http:// URL

• DatasetId: It defines the name of the RDF dataset. Any valid name without any special

characters

• FileURL: Valid HTTP link to the JSON-stat file (optional). If not provided, the ‘Content’

parameter should be given.

• Content: Content of the JSON-stat (optional). If not provided, the ‘FileURL’ should be given.

• VocabNmSpace: This the vocab namespace that will be used in RDF file should be a valid

http:// URL.

Page 22: LOSD Publication and Capacity Building...4.3 Juma API 11 4.4 Juma Docker Image 12 4.5 Cube Visualizer 13 4.6 Cube Visualizer Docker Image 13 4.7 OLAP Browser 14 4.8 OLAP Browser Docker

LOSD D4.1

22

7 LOSD Example Application

https://losd-dataviz.staging.derilinx.com/

In order to demonstrate the capability of the LOSD Pipeline, we developed an example application -

the Unemployment Rate Visualiser. The Unemployment Rate Visualiser utilises the Linked

unemployment rate data from the four pilot countries to demonstrate comparisons across the

regions. As the data is available as Linked Data, the application can easily access the necessary

information, map the data according to the NUTs regions, and display corresponding graphs

dynamically. The application is interactive, allowing a user to browse around different countries and

regions, and to zoom in on the graphs.

Some of the challenges that we faced when creating the Unemployment Rate Visualiser is that the

data was not completely standardised across countries. For example, some data referred to NUTS2

regions and some to NUTS3 regions, the time periods varied per country, and some countries

expressed the unemployment rate by quarter and some by qualification code.

Nevertheless, the Unemployment Rate Visualiser clearly demonstrates the querying and

interoperability benefits that publishing as Linked Open Statistical Data brings.

8 Conclusion This deliverable has presented a complete documentation of the LOSD Publishing Pipeline, including

how to install and setup each of the tools within the LOSD publication pipeline, and an overview of

the JSON-stat to RDF exporter. From this document, the reader should be able to replicate the LOSD

publication pipeline for their own use.