SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans

1

SDMX Reference Infrastructure(SDMX-RI)

Work in progress, status and plans

Bengt-Åke Lindblad, Adam Wroński

Eurostat

Eurostat Unit B3 – IT and standards for data and metadata exchange

Content

What is SDMX-RI Why use SDMX-RI Usage status Development strategy Release calendar

Eurostat Unit B3 – IT and standards for data and metadata exchange2

What is SDMX-RI

Universal framework for modern data provision Set of pick-and-choose reusable building blocks

allowing a statistical office to expose data to the external world based on access rights

Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse

Uses SDMX standards incl. one for Web Services



Why SDMX-RI

Developed to simplify the exchange of data Provides standard software and components,

allowing individual statistical organisations to interact and exchange their data using the same software and methodology

Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes

Developed in both Java and .NET


Usage

As of Sept 2012 deployed in 22 EU countries Tested across EU Member States (2011

population census) Running in Mexico Expression of interest

Latin America, the Caribbean, OECD and Russia July 2013:

expected to run in EU27 member states in Eurostat dissemination Web Service upgrade


SDMX-RI versions

StreamingSDMX-RI

Initial Solution

Common API SDMX

v2.1

SDMX-RIIntermediate

Solution

SDMX-RIUltimate Solution


Development strategy 2012-2013

Architectural changes are needed (two step approach) Intermediate solution to solve performance & out of

memory errors Ultimate solution to provide a common API and

implement SDMX 2.1 Implement new user requests and correct defects Widen the scope and usage of SDMX-RI among data

providers ESSVIP project such as Web INfrastructure for Statistics Reuse for other statistical data collections DSWS (Eurostat dissemination web service)


Why an intermediate solution

Problems needed to be solved within short time for existing users Performance

High memory consumption and response times constrained organizations from putting it in production

Out Of Memory errors Impossible to serve large data requests.


What changed?

Streaming of data in the service Migration to JAX-WS in Java

IO revised with Streaming Writers Data Retriever API changed (streaming)


Benefits

Better performance Improvement up to 4 times faster for .Net Improvement up to 7 times faster for Java

Solution to Out Of Memory problems for large datasets No memory constrains Data are streamed to the client


Impact

Organizations that have installed initial solution Only re-install the Web Service Package Existing users of Web Service package not affected

Same v2.0 interface remain Organizations using the components API

API has changed due to streaming support Migration required

Organizations has done modifications to the source code Have to make the changes again if they want to use the

intermediate version.


Why an ultimate solution?

Use of common API (that supports various versions and flavors of SDMX) Interfaces which allow implementation using the

components and also the interfaces for component-to-component communication

Better inter-organization component reusability Ability to support new versions of SDMX without

changing our applications Support of SDMX 2.1

New messages (data representation, queries) New Interfaces (SOAP/REST)


What will be changed?

All modules migrated to the common API To use of new SDMX Model/IO Components APIs will be changed

Support of SDMX 2.1 messages and new query features Extend to support 2.1 standardized SOAP and RESTful APIs

New Web Service end points Co-existence with v2.0 service Support of v2.1 error codes


Impact

Mapping Assistant users will be able to reuse their Mapping Stores Mapping Store upgrade will be supported Will be required to re-install MA only

Organizations that have installed initial solution Only to re-install the Web Service Package Existing users of Web Service package will not be affected

Same v2.0 interface will remain Organizations using components API

API has changed due to streaming support Migration will be required Migration guidelines will be provided

Organizations done modifications to the source code Will have to make the changes again if they want to use the

new version.


Status - release Common API development (jar/dll expected end 2012)

Java: ongoing .NET: not started yet (API should be ported to C# and then

implemented) SDMX 2.1 support (expected Q1 2013)

Not yet started (dependent on API) Enhancements and bug corrections (independent)

Mapping Assistant (Sept 2012) Improved transcoding, error handling, QueryEditor, etc

Mapping Assistant (end 2012) Implementation of 11 request mainly linked to the user interface

and the usage Test Client

using encrypted user id and password


[email protected]

[email protected]

SDMX-RI contacts


mailto:[email protected]

mailto:[email protected]

Web Service Package extension to SDMX 2.1 and new interfaces

ControllerController

Data Retriever (streaming)

Data Retriever (streaming)

Web

S

ervi

ce

Pro

vide

r

Common API

Common API

Structure RetrieverStructure Retriever

NSI_Service_2.0NSI_Service_2.0 NSI_Service_2.1NSI_Service_2.1

SOAP Request v2.0

SOAP Request v2.1

NsiRestServiceNsiRestService

REST Request v2.1

API Implementation

API Implementation


Documents

SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans