SDMX framework 2 - OECD · SDMX global conference of 2009 SDMX framework 2.0 SDMX framework 2.0 SDMX Metadata framework (Mauro Bianchi –Capacity building C) SDMX Data framework

SDMXglobal conference

of 2009


of 2009

Laura Vignola

Italian National Institute of Statistics

SDMX framework 2.0


of 2009


of 2009 SDMX framework 2.0

SDMX framework 2.0

SDMX Metadata framework

(Mauro Bianchi – Capacity building C)

SDMX Data framework

(Dario Camol – Capacity building B)


of 2009

Paris, 19-21/01/2009


of 2009


of 2009 SDMX Metadata Framework

Istat production of questonnaires

•FMI (document)

•EUROSTAT – STS (document)

•EUROSTAT – ESMS (xml)


of 2009

Paris, 19-21/01/2009


of 2009


of 2009 SDMX Metadata framework

Store the reference metadata into a database

Create a single access point to insert metadata for all the questionnaires

Reuse the common information among various questionnaires


of 2009

Paris, 19-21/01/2009


of 2009



•Manager web application (creates the environment using an xml file)

•Web application to insert data

•Desktop application to produce output

•Web service application

•Rss provider


of 2009

Paris, 19-21/01/2009


of 2009


of 2009


of 2009

Paris, 19-21/01/2009

<SDDS>

<CONFIG>

<CONNECTION>

<STATUS>1</STATUS><VALUE> connection string;</VALUE>

</CONNECTION>

<CONNECTION>.......</CONNECTION>

</CONFIG>

<TEMPLATES>

<TEMPLATE>

<VALUE>1</VALUE>

<AGENCY_NL>DQAF</AGENCY_NL><AGENCY_EN>DQAF</AGENCY_EN>

<SDMX_STRUCTURE>

<PK_SDMX>02.01.04</PK_SDMX>

<DESCRIPTION>Source data definitions, scope, …..</DESCRIPTION>

<FK_SDMX>01.04</FK_SDMX>

<FK_TEMPLATE>1</FK_TEMPLATE>

<INTERFACE>1</INTERFACE>

<COMMENTNL>…...</COMMENTNL>

<COMMENTEN>…….</COMMENTEN>

</SDMX_STRUCTURE>

<SDMX_STRUCTURE> .......</SDMX_STRUCTURE>

</TEMPLATE>

<TEMPLATE>……</TEMPLATE>

</TEMPLATES>

<DECODE>

<SDMX_DECODE>

<PK_SDMX>03</PK_SDMX>

<LEVEL>02.01.04</LEVEL>

</SDMX_DECODE>

<SDMX_DECODE>……</SDMX_DECODE>

</DECODE>

<DOMAIN>

<DATA_FLOW>

<PK_DATAFLOW>SSTSIND_ORDR_MS</PK_DATAFLOW>

<DATAFLOW_NL>New order received …</DATAFLOW_NL>

<DATAFLOW_EN>New order received ...</DATAFLOW_EN>

<RSS_DESC />

</DATA_FLOW>

<DATA_FLOW>……</DATA_FLOW>

</DOMAIN>

</SDDS>


of 2009



Manager

Loader

FMI

STS

ESMS

SDDS.xml

DQAF.doc

STS.doc

EUROSTAT

SDMX query

ESMS

Output

rss

WebService


of 2009

Paris, 19-21/01/2009

ESMS.xml

SDDS.xml

SDDS.xml

SDDS.xml

SDDS.xml


of 2009


of 2009 SDMX DATA framework

Partecipation in several Eurostat projects(SODI, Demography, Census Hub)

Istat scenario

Time (Long terms solution, Short terms solution)


of 2009

Paris, 19-21/01/2009


of 2009



Create an environment structured with SDMX elements

Follow data from loading to dissemination


of 2009

Paris, 19-21/01/2009


of 2009



Manager desktop application

Loader desktop Application

Web service

Rss provider


of 2009

Paris, 19-21/01/2009


of 2009



Manager

Creates the environment

Creates the tables for loading data


of 2009

Paris, 19-21/01/2009


of 2009



Loader

Loads datafile (csv, fixed length record, gesmes TS)

Creates SDMX datafile and rss (pull method without WS)

Paris, 19-21/01/2009


of 2009


of 2009



Web Service

Query parser

SQL generator

SDMX Generator


of 2009

Paris, 19-21/01/2009


of 2009


of 2009 Query parser

•Time

•And

•Or

•Dimension

•Attribute

•Dataflow

•Data Provider

Meeting of the OECD

Expert Group on

Statistical Data and

Metadata Exchange

Basic types of SDMX query nodes

Complex nodes

Simple nodes


of 2009

Paris, 19-21/01/2009


of 2009



<Time><StartTime> Date 1</StartTime><EndTime> Date 2</EndTime>

</Time>

Meeting of the OECD

Expert Group on


Metadata Exchange

<And><And> Cond 1, Cond 2 </And>

</And>

<And> Cond 1Cond 2

</And>

<And><StartTime> Date 1</StartTime><EndTime> Date 2</EndTime>

</And>


of 2009

Paris, 19-21/01/2009


of 2009



<Or> <Or> Cond 4, Cond 5 </Or>

</Or>

Meeting of the OECD

Expert Group on


Metadata Exchange

<And>Cond 1<Or> Cond 2, Cond 3</Or>

</And>

<Or> Cond 1Cond 2

</Or>

<Or><And> Cond 1, Cond 2</And><And> Cond 1, Cond 3 </And>

</Or>


of 2009

Paris, 19-21/01/2009


of 2009


of 2009


of 2009

Paris, 19-21/01/2009

<Query>

<query:DataWhere>

<query:And>

<query:DataFlow>STSIND_PROD_M</query:DataFlow>

<query:Dimension name="FREQUENCY">M</query:Dimension>

<query:Dimension name="INDICATOR">PROD</query:Dimension>

<query:Or>

<query:And>

<query:Dimension name="ADJUSTMENT">S</query:Dimension>

<query:Or>

<query:Dimension name="STS_ACTIVITY">N100DA</query:Dimension>

<query:Dimension name="STS_ACTIVITY">N100C0</query:Dimension>

<query:Dimension name="STS_ACTIVITY">N100D0</query:Dimension>

</query:Or>

<query:Time>

<query:StartTime>2000-01</query:StartTime>

<query:EndTime>2006-01</query:EndTime>

</query:Time>

</query:And>

<query:And><query:Dimension name="STS_ACTIVITY">N100C0</query:Dimension><query:Time>

<query:StartTime>1995-01</query:StartTime><query:EndTime>2003-01</query:EndTime>

</query:Time></query:And>

</query:Or>

</query:And>

</query:DataWhere>

</Query>


of 2009



Meeting of the OECD

Expert Group on


Metadata Exchange

START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY

2000-01 2006-01 SSTSIND_PROD_M S M PROD N100DA

2000-01 2006-01 SSTSIND_PROD_M S M PROD N100D0

2000-01 2006-01 SSTSIND_PROD_M S M PROD N100C0

1995-03 2003-02 SSTSIND_PROD_M M PROD N100C0

ANDOR


of 2009

Paris, 19-21/01/2009

Problems with long query

Italy-Long-Query.xml


of 2009



<And>Cond 1<Or> Cond 2, Cond 3</Or><Or> Cond 4, Cond 5</Or>

</And>

<And>Cond 1<Or> Cond 2-3</Or><Or> Cond 4-5</Or>

</And>

AGE CAS SEX GEO

TOT - 001 - 002 - 003 - 004 -005 - 006 - 007 - 008 - 009 -010 - 011 - 012 - 013 - 014 -015 - 016 - 017 - 018 - 019 -020 - 021

TOT - 001 - 002 - 003 - 004 -005 - NST - NAP

TOT – 001 - 002 IT - ITC - ITC1 - ITC11 - ITC11_001001 -ITC11_001002 - ITC12 - ITC12_002002 -ITC12_002003 - ITC3 - ITC31 - ITC31_008001 -ITC31_008002 - ITC32

- ITC32_009001 - ITC32_009002


of 2009


of 2009


of 2009 SQL generator

Meeting of the OECD

Expert Group on


Metadata Exchange



2000-01 2006-01 SSTSIND_PROD_M S M PROD N100D0

2000-01 2006-01 SSTSIND_PROD_M S M PROD N100C0

1995-03 2003-02 SSTSIND_PROD_M M PROD N100C0

SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT=‘S’ AND FREQUENCY=‘M’ AND INDICATOR=‘PROD’ AND STS_ACTIVITY=N100DA AND TIME_PERIOD BETWEEN 2000-01 AND 2006-01

UNION

SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT=‘S’ AND FREQUENCY=‘M’ AND INDICATOR=‘PROD’ AND STS_ACTIVITY=N100D0 AND TIME_PERIOD BETWEEN 2000-01 AND 2006-01

………………………………………………


of 2009

Paris, 19-21/01/2009


of 2009


of 2009The architecture

Meeting of the OECD

Expert Group on


Metadata Exchange

ESA

DEM

STS

EUROSTATManager

csv

GES

FLR

SDMX compact

SDMX cross

sectional

Loader

rss

Query parser

SQL generator

SDMX generator

rss

SDMX compact

SDMX cross

sectional

SDMX query


of 2009

Paris, 19-21/01/2009


of 2009


of 2009 Sql Mapping generator

Meeting of the OECD

Expert Group on


Metadata Exchange



NOME_TABLE

NOME_FIELD

CATEGORIA

TIPO

VS

ATECO

U_MIS

DATAFLOW

ADJUSTMENT

FREQUENCY

INDICATOR

STS_ACTIVITY

OBS_STATUS

OBS_CONF

STS_MAPPING_TABLE


of 2009

Paris, 19-21/01/2009

CONISTAT


of 2009


of 2009Sql Mapping generator

Meeting of the OECD

Expert Group on


Metadata Exchange

ESA

DEM

STSEUROSTATManager

csv

GES

FLR

SDMX compact

SDMX cross

sectional

Loader

rss

SDMX compact

SDMX cross

sectional

SDMX query

Query parser

SQ

L g

en

erato

r

SDMX generator

SQ

L M

ap

pin

g

gen

erato

r

Query parser

SQL generator

SDMX generator

Sdmx db

CONISTAT


of 2009

Paris, 19-21/01/2009


of 2009


of 2009 Links for download

Meeting of the OECD

Expert Group on


Metadata Exchange


of 2009

Paris, 19-21/01/2009

http://circa.europa.eu

http://forge.osor.eu/projects/sdmxframework/

http://circa.europa.eu/

http://forge.osor.eu/projects/sdmxframework/

Documents

SDMX framework 2 - OECD · SDMX global conference of 2009 SDMX framework 2.0 SDMX framework 2.0 SDMX Metadata framework (Mauro Bianchi –Capacity building C) SDMX Data framework