View
48
Download
0
Category
Preview:
Citation preview
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
Laura Vignola
Italian National Institute of Statistics
SDMX framework 2.0
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX framework 2.0
SDMX framework 2.0
SDMX Metadata framework
(Mauro Bianchi – Capacity building C)
SDMX Data framework
(Dario Camol – Capacity building B)
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX Metadata Framework
Istat production of questonnaires
•FMI (document)
•EUROSTAT – STS (document)
•EUROSTAT – ESMS (xml)
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX Metadata framework
Store the reference metadata into a database
Create a single access point to insert metadata for all the questionnaires
Reuse the common information among various questionnaires
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX Metadata framework
•Manager web application (creates the environment using an xml file)
•Web application to insert data
•Desktop application to produce output
•Web service application
•Rss provider
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
<SDDS>
<CONFIG>
<CONNECTION>
<STATUS>1</STATUS><VALUE> connection string;</VALUE>
</CONNECTION>
<CONNECTION>.......</CONNECTION>
</CONFIG>
<TEMPLATES>
<TEMPLATE>
<VALUE>1</VALUE>
<AGENCY_NL>DQAF</AGENCY_NL><AGENCY_EN>DQAF</AGENCY_EN>
<SDMX_STRUCTURE>
<PK_SDMX>02.01.04</PK_SDMX>
<DESCRIPTION>Source data definitions, scope, …..</DESCRIPTION>
<FK_SDMX>01.04</FK_SDMX>
<FK_TEMPLATE>1</FK_TEMPLATE>
<INTERFACE>1</INTERFACE>
<COMMENTNL>…...</COMMENTNL>
<COMMENTEN>…….</COMMENTEN>
</SDMX_STRUCTURE>
<SDMX_STRUCTURE> .......</SDMX_STRUCTURE>
</TEMPLATE>
<TEMPLATE>……</TEMPLATE>
</TEMPLATES>
<DECODE>
<SDMX_DECODE>
<PK_SDMX>03</PK_SDMX>
<LEVEL>02.01.04</LEVEL>
</SDMX_DECODE>
<SDMX_DECODE>……</SDMX_DECODE>
</DECODE>
<DOMAIN>
<DATA_FLOW>
<PK_DATAFLOW>SSTSIND_ORDR_MS</PK_DATAFLOW>
<DATAFLOW_NL>New order received …</DATAFLOW_NL>
<DATAFLOW_EN>New order received ...</DATAFLOW_EN>
<RSS_DESC />
</DATA_FLOW>
<DATA_FLOW>……</DATA_FLOW>
</DOMAIN>
</SDDS>
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX Metadata framework
Manager
Loader
FMI
STS
ESMS
SDDS.xml
DQAF.doc
STS.doc
EUROSTAT
SDMX query
ESMS
Output
rss
WebService
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
ESMS.xml
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Partecipation in several Eurostat projects(SODI, Demography, Census Hub)
Istat scenario
Time (Long terms solution, Short terms solution)
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Create an environment structured with SDMX elements
Follow data from loading to dissemination
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Manager desktop application
Loader desktop Application
Web service
Rss provider
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Manager
Creates the environment
Creates the tables for loading data
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Loader
Loads datafile (csv, fixed length record, gesmes TS)
Creates SDMX datafile and rss (pull method without WS)
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SDMX DATA framework
Web Service
Query parser
SQL generator
SDMX Generator
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Query parser
•Time
•And
•Or
•Dimension
•Attribute
•Dataflow
•Data Provider
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
Basic types of SDMX query nodes
Complex nodes
Simple nodes
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Query parser
<Time><StartTime> Date 1</StartTime><EndTime> Date 2</EndTime>
</Time>
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
<And><And> Cond 1, Cond 2 </And>
</And>
<And> Cond 1Cond 2
</And>
<And><StartTime> Date 1</StartTime><EndTime> Date 2</EndTime>
</And>
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Query parser
<Or> <Or> Cond 4, Cond 5 </Or>
</Or>
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
<And>Cond 1<Or> Cond 2, Cond 3</Or>
</And>
<Or> Cond 1Cond 2
</Or>
<Or><And> Cond 1, Cond 2</And><And> Cond 1, Cond 3 </And>
</Or>
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
<Query>
<query:DataWhere>
<query:And>
<query:DataFlow>STSIND_PROD_M</query:DataFlow>
<query:Dimension name="FREQUENCY">M</query:Dimension>
<query:Dimension name="INDICATOR">PROD</query:Dimension>
<query:Or>
<query:And>
<query:Dimension name="ADJUSTMENT">S</query:Dimension>
<query:Or>
<query:Dimension name="STS_ACTIVITY">N100DA</query:Dimension>
<query:Dimension name="STS_ACTIVITY">N100C0</query:Dimension>
<query:Dimension name="STS_ACTIVITY">N100D0</query:Dimension>
</query:Or>
<query:Time>
<query:StartTime>2000-01</query:StartTime>
<query:EndTime>2006-01</query:EndTime>
</query:Time>
</query:And>
<query:And><query:Dimension name="STS_ACTIVITY">N100C0</query:Dimension><query:Time>
<query:StartTime>1995-01</query:StartTime><query:EndTime>2003-01</query:EndTime>
</query:Time></query:And>
</query:Or>
</query:And>
</query:DataWhere>
</Query>
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Query parser
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100DA
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100D0
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100C0
1995-03 2003-02 SSTSIND_PROD_M M PROD N100C0
ANDOR
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
Problems with long query
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Query parser
<And>Cond 1<Or> Cond 2, Cond 3</Or><Or> Cond 4, Cond 5</Or>
</And>
<And>Cond 1<Or> Cond 2-3</Or><Or> Cond 4-5</Or>
</And>
AGE CAS SEX GEO
TOT - 001 - 002 - 003 - 004 -005 - 006 - 007 - 008 - 009 -010 - 011 - 012 - 013 - 014 -015 - 016 - 017 - 018 - 019 -020 - 021
TOT - 001 - 002 - 003 - 004 -005 - NST - NAP
TOT – 001 - 002 IT - ITC - ITC1 - ITC11 - ITC11_001001 -ITC11_001002 - ITC12 - ITC12_002002 -ITC12_002003 - ITC3 - ITC31 - ITC31_008001 -ITC31_008002 - ITC32
- ITC32_009001 - ITC32_009002
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 SQL generator
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100DA
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100D0
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100C0
1995-03 2003-02 SSTSIND_PROD_M M PROD N100C0
SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT=‘S’ AND FREQUENCY=‘M’ AND INDICATOR=‘PROD’ AND STS_ACTIVITY=N100DA AND TIME_PERIOD BETWEEN 2000-01 AND 2006-01
UNION
SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT=‘S’ AND FREQUENCY=‘M’ AND INDICATOR=‘PROD’ AND STS_ACTIVITY=N100D0 AND TIME_PERIOD BETWEEN 2000-01 AND 2006-01
………………………………………………
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009The architecture
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
ESA
DEM
STS
EUROSTATManager
csv
GES
FLR
SDMX compact
SDMX cross
sectional
Loader
rss
Query parser
SQL generator
SDMX generator
rss
SDMX compact
SDMX cross
sectional
SDMX query
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Sql Mapping generator
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY
2000-01 2006-01 SSTSIND_PROD_M S M PROD N100DA
NOME_TABLE
NOME_FIELD
CATEGORIA
TIPO
VS
ATECO
U_MIS
DATAFLOW
ADJUSTMENT
FREQUENCY
INDICATOR
STS_ACTIVITY
OBS_STATUS
OBS_CONF
STS_MAPPING_TABLE
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
CONISTAT
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009Sql Mapping generator
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
ESA
DEM
STSEUROSTATManager
csv
GES
FLR
SDMX compact
SDMX cross
sectional
Loader
rss
SDMX compact
SDMX cross
sectional
SDMX query
Query parser
SQ
L g
en
erato
r
SDMX generator
SQ
L M
ap
pin
g
gen
erato
r
Query parser
SQL generator
SDMX generator
Sdmx db
CONISTAT
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
SDMXglobal conference
of 2009
SDMXglobal conference
of 2009 Links for download
Meeting of the OECD
Expert Group on
Statistical Data and
Metadata Exchange
SDMXglobal conference
of 2009
Paris, 19-21/01/2009
http://circa.europa.eu
http://forge.osor.eu/projects/sdmxframework/
Recommended