Essnet for SDMX Phase II WP2 PC-Axis SDMX Integration

Essnet for SDMX Phase II WP2 PC-Axis SDMX

IntegrationPC-Axis International Reference Group

meeting 31 May 2012Lars Nordbäck and Ulf Sandberg

Statistics Sweden

23-04-21 1

Results from Essnet for SDMX II WP2 PC-Axis SDMX Integration 2012-05-31

Objectives • Routine for reporting of statistics to international

organisations in SDMX format• To set up routines to send the data in accordance

with the Eurostat Reference Infrastructure• Use the dissemination databases and PC-Axis

environment as sources at the NSI

February 2012:

Now restricted to cover only CensusHub project domain. No other domain at Statistics Sweden has DSD and code lists available within the duration of this project. No encryption facility like in eWA portal available in the RI.

23-04-21 2


23-04-21 3


Implementation Status of Domains

Balance of Payments

National Accounts

Research & Development

Pesticides

Waste Census Aviation

Fisheries

Local Coordinator meeting October 2011

Activities Only for CensusHub

• Continue the work made in Phase I, for example test the full routines coordinated with Eurostat.

• Manuals for users and administrators at the NSI in the PC-Axis community

• More use cases added to the users manual for the Process, deleted

• Continue the verifying of new versions of the Reference Infrastructure

23-04-21 4


Links to visualisation tools based on Generic SDMX from PC-Axis

23-04-21 5


1. Data Express working

2. eXplorer test is going on, with some resulting bug fixing in the conversion software

Deliverables Only for CensusHub

• Documentation. Documents to be delivered is use cases with different type of materials (2012).

• Workshops. The purpose with the workshops is to transfer the methods and software for use of the same platform based on PC-axis and the Eurostat SDMX Reference Infrastructure achievements in Phase 1 and Phase 2

23-04-21 6


Deliverables (cont.)

• SDMX functionality integrated in the web-software• Implementing of attributes on cell level in PX-Web• Implementation of the Generic Default SDMX-component

in PX-Web dotnet

23-04-21 7


The links with work packages defined in ESSnet-1

• The Phase II is based on the work in Phase I in order to use the Reference Infrastructure in an NSI in the PC-Axis community

• WP 3 Data Express

23-04-21 8


Re-usability and links to SDMX

• Other NSI in the PC-Axis community can use the results• Nordic data model, SQL

• MS-SQL• Oracle• Sybase • MySQL• ODBC

• PC-Axis files

• For international reporting focusing CensusHub• Links to visualisation tools based on SDMX

23-04-21 9


The re-usability within ESS• The different activities within the WP 2 will be

documented according to the Eurostat templates and made available within the ESS.

• The deliveries will be available according to the present time plans that is stated for the activities respectively.

• Software within the PC-Axis family will be available on the PC-Axis portal: http://www.scb.se/Pages/StandardNoLeftMeny____314045.aspx

• Software that can be used outside the PC-Axis family will be put on the Eurostat Portal for software downloads together with installation instructions

23-04-21 10


http://www.scb.se/Pages/StandardNoLeftMeny____314045.aspx

http://www.scb.se/Pages/StandardNoLeftMeny____314045.aspx

Actions in the developing work grouped in areas

23-04-21 11


5. Eurostat SDMX infrastructure – test of routines and further development of

SDMX-RI

1. Process descriptions inclusive user manuals at Statistics Sweden

2. Transfer of processes

4. Confidential materials

3. The Data Model and applications

6. Extension of material

Explanation of signs:Activity restricted to cover the CensusHub related processActivity deleted

1. Process descriptions inclusive user manuals at Statistics Sweden• a) Design proposal on the work flow for different

central organizational units and subject matter divisions for a first delivery and work flow for continuous deliveries using Eurostat-RI.

• b) Proposal for the distribution of responsibility between different Objects of Maintenance when it comes to applications in the Infrastructure at Statistics Sweden and the Eurostat-RI. It includes an up date of the Maintenance Specifications.

23-04-21 12


• c) User Manual for statistics producers and others. • d) Process Support System• e) Adjustments in the User Manuals and the Process

Support System from Feed Back from the users that has entered new materials etc.

23-04-21 13


1. Cont. Process descriptions inclusive user manuals at Statistics Sweden

2. Transfer of Processes

Internally at Statistics Sweden • a) Preparations of tests of the new way of working

for selected test products.

• b) Seminars, information, time table and test together with the producers.

23-04-21 14


2. cont. Transfer of Processes To other countries• c) Preparation of User Manuals for the PC-Axis

family of users. As in 1 c but adjusted for the PC-Axis environment and translated into English.

• d) Presentation and information for other countries and Swedish authorities that will use the process developed in Phase IBefore this activity more tests has to be done. The workshop will be carried out in cooperation with ISTAT.

23-04-21 15


2. Cont. Transfer of Processes

For other organizations in Sweden responsible for official Statistics – low priority• e) Study of how other governmental bodies in

Sweden will do the reporting to Eurostat in the SDMX era and information concerning the changesStatistics Sweden is local coordinator for Sweden, not only for Statistics Sweden but other organizations.

23-04-21 16


3. The Data model and applications

a) Test of PX-Edit concerning Attributes on Cell level and capacity test of PX-Edit, PC-Axis and PX-Web using large census files PX-Edit must be tested at Statistics Sweden when Finland has finalized the development and test. Also including assessment of the User Manual and give feed back to Statistics Finland.

23-04-21 17


3. Cont. The Data model and applicationsb) Implementing of attributes on cell level in PX-Web

(dot net version)• Do a study on the amount of work to Develop

Presentation of attributes on cell level in tables and files. The study is based on the Data model and the file format for attributes on cell level. It is desired that the result can be used for the database system on the Statistics Sweden website.

23-04-21 18


3. Cont. The Data model and applicationsc) Test and adjustment of the Generic Default

SDMX-format from PC-Axis and check that the same SDMX format is used by other agencies.

• Test of the result from the conversion to Generic Default SDMX format from PC-Axis technical platform. The produced files will be tested in applications that is said to be compatible with the generic format. For instance the Diagram package from Phase I Portuguese and also the eXplorer.

23-04-21 19


3. Contin. The Data model and applicationsd) Implementation of the Generic Default SDMX-

component in PX-Web dot netWill be coordinated with the development of the Webb GUI for the database system at Statistics Sweden Website. The system documentation will be updated. Is in the Beta version of PX-Web released in May 2012.

e) Documentation of data model

23-04-21 20


3. Cont. The Data model and applications

f) Adjustments for the census Adjustment of publishing software for the CensusHub data. NEW

g) Access to Census conflicts Access to Census server. Can Website access and Web service access the same server without conflicts affecting performance? NEW and deleted since the CensusHub data will reside in a separate server for the Eurostat access.

23-04-21 21


4. Confidential Statistics• a) Permission issues

Concerns permission inside Statistics Sweden on servers and/or applications. Only those persons working with a certain statistics shall have the permission to watch the statistics. One of the suggestions from Statistics Sweden for the update of the Eurostat RI.

23-04-21 22


4. Cont. Confidential Statisticsb) Security issues for confidential statistics• Investigate what recommendation from the

Security organization at Statistics Sweden that is valid for international reporting over a Web Server and Web services.

• Server and Catalogue structure proposals• Encryption issues for Eurostat• Await exercises from the Swedish Defence tests of

Statistics Sweden Website in a related project• Relation to the project for “study of Generated files

and simple web services”.

23-04-21 23


4. Cont. Confidential Statisticsc) Further deliveries to other International

Organisations and Confidential Statistics• How is the statistics handled by Eurostat that is

made available for other international organizations? Some data can be confidential.

• Those using encryption today and also those that are sending confidential data today without encryptions? Who are doing what? Check with Eurostat what rules that are valid for third party for encrypted data and Confidential data respectively.

23-04-21 24


5. Eurostat Reference Infrastructure – test of routines and further development of the Eurostat RI

a) Complete tests of (the Standard routine and) the Census routine for both the SQL and PC-Axis files in collaboration with Eurostat to get the whole chain showed working. Up date of DSD:s in the Registry are required.

b) Installation and test of new versions of the SDMX-RI; feed-back to Eurostat

c) Meeting in Luxembourg with the developers of the Eurostat RI New

23-04-21 25


6. More statistics – low priority

a) Test of different types of statistics for delivery to Eurostat. Will also be included in the maintenance plan

b) b) Possible further development caused by results of test for different statistics

23-04-21 26


6. Cont. More statistics – low priority

c) More use cases shall be added to the User’s Manual for the Process- up date, Swedish users Manual and the Process Support System The Unit for National Accounts are interested to get information. They have complex statistics. It is needed that the relevant DSD are made available from Eurostat to make it possible to do the Mapping work. Hitherto very few DSD has been made available. A more Generic User Manual in English is needed for the PC-Axis Family of users.

23-04-21 27


The project plan after Kick off meeting 6 September 2011

23-04-21 28


ESSnet for SDMX Phase 2, WP 2 PC-Axis SDMX IntegrationProject/activity 2011 2012

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep1 a Tomas, Ulf, Bodil 3*20 hours1 b Henrik, Ulf, Bodil 3*10 hours1 c Tomas, Ulf, Bodil 35, 35, 10 hours1 d Tomas, Bodil, (Doris) 20, 10 hours1 e Tomas, Ulf, Bodil, (Doris) 10 ,10, 10 hours

2 a Henrik,Ulf, Bodil 3, 3, 3 hours2 b Henrik, Ulf, Bodil 10, 25, 10 hours2 c Ulf, Lars, Thomas 30, 30, 30 hours2 d Lars + ? 56 + 56 hours2 e Lars, Ulf, Bodil, Henrik 20, 20, 10, 10 hours

3 a Lars, Ulf 40, 40 hours3 b Mikael, Petros, Lars, ?, (Doris) 3, 3, 3, 3+7, 7 hours 200 hours???3 c Lars, Petros 10, 10 hours3 d Petros, Mikael 10, 10 hours3 e Petros, Doris, Else-Marie 10, 10, 10 hours

4 a Ulf, Mikael, Petros,(IT4), Lars 10, 10, 10, 10+20, 20 hours4 b Ulf, Mikael, Petros,(IT4), Lars 15, 15, 15, 15 hours4 c Ulf, Bodil 40, 10 hours

5 a Ulf, Lars, Mikael, Henrik 85, 15, 10, 10 hours5 b Ulf, Lars, Henrik 70, 20, 10 hours

6 a Ulf 50 hours6 b Mikael, Petros, Doris, Else-Marie 20, 20, 20, 20 hours6 c Ulf, Lars, Thomas M, (Doris) 25, 25, 25 hours

The project plan April 2012

23-04-21 29


Test of the Eurostat-RI in ESSnet for SDMX Phase 2, WP 2 PC-Axis SDMX Integration, revised April 2012Project/activity 2011 2012

2012-04-26 Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep1 Process descriptions inclusive user manuals at Statistics Sweden1 a Design workflow Ulf and Lars 2x251 b Responsibillity planHenrik, Ulf, Bodil 3*10 hours2 Transfer of 2 c English manuals Ulf, Lars 20, 20 hourspart of 2 d Workshop in Rome 25-27 June2 d Pres for other NSI:sLars + Ulf? 2x10 56 + 56 + 20 hours ?>3 The Data model and applications3 a PX-Edit Attr. Lars, Ulf 40, 40 hours3 b Attr. In PX-Web Mikael, Petros, Lars, ?, (Doris) 3, 3, 3, 3+7, 7 hours 160 hours3 c Test. Generic defaultLars, Petros 10, 10 hours3 d Gen. Def. In PX-WebPetros, Mikael 10, 10 hours3 e Doc. Data models Petros, Doris, Else-Marie 10, 10, 10 hours3 f Adj. For the Census Else-Marie/konsult 50 hours New3 g Access to cens. ConflictsHakim, Ulf, Lars 10+3+3 hours New4 Confidential Statistics4 a Authority issues Ulf, Mikael, Petros,(IT4), Lars 10, 10, 10, 10+20, 20 hours5 Eurostat Reference Infrastructure - test of routines and further developoment of the Eurostat RI5 a Test with Eurostat Ulf, Lars, Mikael, Henrik 50, 15, 10, 10 hours ?>Census fictive dataCensus real data5 b Install. New vers. Ulf, Lars, Henrik 70, 20, 10 hours5 c Meeting RI LuxembourgUlf, Lars 20+2 hours New5 d Test of ISTAT Excel RI webserviceLars 10 hours NewActivities that has been dropped after meeting with Eurostat 26 of January 2012

1 c User manuals Thomas, Ulf, Bodil 35, 35, 10 hours1 d Process supp.syst. Thomas, Bodil, (Doris) 20, 10 hours1 e Adj. Man. Pr.supp. Thomas, Ulf, Bodil, (Doris) 10 ,10, 10 hours2 a Tests products Henrik,Ulf, Bodil 3, 3, 3 hours2 b Intern. Seminars Henrik, Ulf, Bodil 10, 30, 10 hours2 e Study other Sw. Org.Lars, Ulf, Bodil, Henrik 20, 20, 10, 10 hours4 b Security confid. Ulf, Mikael, Petros,(IT4), Lars 15, 15, 15, 15 hours4 c Other int. Org. Ulf, Bodil 40, 10 hours

Process based on PC-Axis files

23-04-21

Results from Essnet for SDMX II WP2 PC-Axis SDMX Integration 2012-05-31 30

Process for CensusHub reporting from PC-Axis files

2012-05-26

PC-Axis Directory

4. Mapping Assistant

1. Create file for

loading of Dissemination

data base

Producer

SDMX-ML

Statistical product

production database

Loading

OtherInternational Organisations

NSI

2.PX-Edit

Mapping StoreData Structure

Definitions, codes

mappings etc

External PC-Axis Directory

Fil

Mapping of PX-table to DSD

Listing of Mapping

information for check

6. Publishing to external PC-Axis directory

Copying

Sending querry

3.PC-Axis Main

Module or PX- Web

Check of entered data

5.SDMX

converter/Test Client

SDMX-ML

For Check

2. Loading data and entering metadata

Publishingspecialist

Firewall Firewall

Web hotel

Eurostat91.194.86.67

7a. Pull Requestor

7b.Data Retriever

m.fl.

http://censusweb.scb.se/NSIservice.asmx

Mapping tables in PC-Axis directory With DSD

Snap shots from software in the process based on PC-Axis files

23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21



23-04-21


Snap shots Test Client

23-04-21


Snap shots Web Client, main window

23-04-21


Snap shots Web Client - Selections

23-04-21


Snap shots Web Client – View Result

23-04-21


Snap shots Web Client – Show result as a table

23-04-21


Snap shots Web Client – View Result as Download as a HTML page

23-04-21


23-04-21 48

Training sesion for PC-Axis SDMX Integration 2011/2012

Process in the CensusHub at Statistics Sweden SQL database

23-04-21


Process for reporting SQL from SCB the CensusHub without RSS

2012-05-24

Census Statistical SQL

database

4. Mapping Assistant

1. Creating file to load into SuperCross

Producer

SDMX-ML

Census production-

database

Otherinternational organisations

SCB

2.. Load SuperCross for checking and

internal outputs

7b.Data Retriever

etc.


Definitions, codes

mappings etc

CensusStatistical SQL

database

File

Mapping Tables with DSD

Mapping list for check

6. Table

transportCopy

Sending query

5.Test Client/Web Client

SDMX-ML

To check

Eurostat91.194.86.67

Firewall

Q01\A

http://censusweb.scb.se/NSIservice.asmx

Web hotelWith web services

Firewall

7a.Pull Requestor

CensusSuperCross

Loading data

3a. Load SQL

Database

File

3b..Setting attributes


Definitions, codes

mappings etc

Possible internal

Mapping Store??

Mapping Tables with DSD

Copy