Francesco Rizzo (ISTAT - Italy)

SDMX ISTAT FRAMEWORK

GENEVE 10 -11 May 2007 OECD SDMX Expert Group

SummarySDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Why Istat is starting to use SDMX

SDMX Istat Framework – an overview

SDMX data Web Service (demo)

SDMX Web Navigator (demo)

Why Istat is starting to use SDMXSDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

The Istat Information System is made up of several production Directorates sub-systems that, independently one from another, cover the full life cycle of statistical data, from collection to dissemination.

Data Collection

Data Editing

Data Aggregation

Production Directorates

metadata

web navigator web navigator

thematicDB

validatedmicrodata

Geneva 08-10 May 2007 Meeting on the Management of Statistical Information Systems

Data Collection

Data Editing

Data Aggregation

Production Directorate

thematicDB

Data Collection

Data Editing

Data Aggregation

Production Directorate

thematicDB

metadata

web navigatorweb navigator

web navigator

Integrated OutputManagement System

web navigator

Integrated Output Management System project

Why Istat is starting to use SDMXSDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

For this reason Istat is following the evolution of the SDMX initiative with interest :

Tactically:

through the participation in the EUROSTAT SODI task force

Strategically:through the building of a special working group whose main objective is to analyze and verify the use of SDMX in the internal architecture of ISTAT Information System

Why Istat is starting to use SDMXSDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

The participation in the EUROSTAT SODI initiative has involved Istat to gain experience in the use of XML and Web Services technologies for integration purpose inside Statistical Information Systems.

In order to facilitate this objective, we are developing several software modules organized in a framework (current version is 1.0)

SDMX Istat Framework version 1.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Characteristics:

• compatible to SODI needs

• useful to introduce SDMX for both internal purposes and external information exchange

• can be used entirely from the reporting phase to the dissemination phase, or alternatively using modules separately, integrating them into one information system

• in the future it might be possible to distribute the framework under Public Licence

time series database

.dat Loader

ConIstat modules

.ist

ConIstatweb navigator

Check

fix formatted record file

GENEVE 10-11 May 2007 OECD SDMX Expert Group

mapping

SDMX web navigatorESA

STS

STS-ESA

SODI RSS

SDMX Queries

new modules

SDMX data web service

RSS builder&

query builder

under construction modules

Referencemetadata manager

Referencemetadatadatabase

Reference metadata SDMX Query Referencemetadata

web navigator

Referencemetadata web service

Reference metadata RSS

Word extractorIMF

Eurostat

gesmes

SDMX Istat Framework version 1.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Overview of changes in the Loader module:

New dissemination functions were added in order to satisfy some SODI requirements:

• to publish an RSS;

• to publish on or more SDMX Query file(s);

• to publish one or more SDMX Compact file(s) (optional)

Response-time test:

an entire Dataflow containing 685 time series, 120 months long (developing 82.200 observation) < 20 seconds

demo

SDMX Istat Framework version 1.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Overview of the SDMX Data Web Service module:

SDMX Compactbuilder

XML validate

query Parser

sql query builder

Database

SDMX Query

SDMX Compact

SDMX Istat Framework version 1.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Overview of SDMX Web Navigator module:

This module acts as a client towards the web service and represents a web graphic interface that allows the following functions:

• building a SDMX Query using DSDs as analysis dimensions

• sending the SDMX Query to the web service for querying the database

• Testing the SDMX Queries that a user can build starting from SDMX Queries created with the graphic interface

demo

SDMX Istat Framework version 1.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

Overview of Manager and Navigator SDMX reference metadata module:

ProductionDirectorates

Database

Referencemetadatamanager

Wordextractor

Eurostat

IMF

SDMXReferencemetadata

web service

XML

Referencemetadata

web navigator

XSLTHTML

SDMX Istat Framework version 2.0SDMX Istat Framework

Geneva 10-11 May 2007 OECD SDMX Expert Group

A generalized SDMX Query parse module (beta version):

This module accepts only queries with DataWhere section but also accept queries regarding:• different Dataflows• subsets of different DataFlows

A SDMX Query is processed by the following steps:• rewrites the XML stream using Boolean logic rules, and adding to each node an attribute that acts as a “unique key”. The scope of the unique key is to set a hierarchy between “parent nodes” and “child nodes”;• converts the XML stream in a memory tabular data structure whose columns represent all types of nodes (Time, Dimension, Attribute, Dataflow) except for And and Or;• re-organizes the in-memory tabular data structure so that all columns have an And relation, and all rows have an Or relation;• converts the in-memory tabular structure in a SQL query