SDMX Converter Application User Manual Version 4

SDMX Converter Application User Manual

Page 1

EUROPEAN COMMISSION EUROSTAT Directorate B: Statistical methodologies and tools Unit B-3: Statistical information technologies


Version 4.11

April 2016


Page 2

Document Change Record

Issue/Revision

Date Change

2.4 16.04.2009 Revision of this document according to comments by ESTAT on the 18/03/2009 and to OSS packaging guidelines.

2.5 26.05.2009 Revision of this document depicting the SDMX Converter v.2.4.5 release. Changes include:

• Correction of a conversion sample

• Enrichment of CLI with some missing parameter values.

2.6 10.07.2009 Revision of this document depicting the SDMX Converter v.2.4.7 development release. Changes include:

• Defining the encoding for reading and writing flat files

• Enhance mapping with accepting non mapped components

2.7 23.07.2009 Revision of this document depicting the SDMX Converter v.2.4.7 release. Changes include:

• Insertions of new error messages

• Defining flat file’s limitations

2.8 04.09.2009 Revision of this document according to comments by ESTAT on the 11-08-2009.

2.9 21.09.2009 Revision of this document according to testing team observations and the new release of the SDMX Converter version 2.5.0.

2.10 23.09.2009 Update of this document according to change in the WSDL of the SDMX Converter WS.

2.11 20.10.2009 Update of this document depicting the SDMX Converter release version 2.5.1. Changes include:

• Update of the table of error messages

• Update of the screenshots

• Update of the mapping subsection

2.12 03.11.2009 Update of this document depicting the SDMX Converter version 2.5.5 and according to ESTAT comments provided on the 4-11-2009

2.13 17.12.2009 Update of this document depicting the SDMX Converter version 2.5.6.

2.14 31.03.2010 Update of this document depicting the SDMX Converter version 2.5.7. Sections 2, 5 and 7 have been updated.

2.15 04.05.2010 Update of this document depicting the SDMX Converter version 2.5.8 (development version). Sections 1 and 4 have been updated.


Page 3

2.16 28.05.2010 Resubmitted in the context of the QTM4 on the SDMX tools maintenance.

Update of this document depicting the SDMX Converter version 2.5.8. Sections 1, 2, 3, 4, 5, 6 and 8 have been updated.

2.17 21.10.2010 Resubmitted in the context of the QTM5 on the SDMX tools maintenance. Update of this document depicting the SDMX Converter version 2.5.9. Sections 1, 3, and 5 have been updated.

2.18 17.11.2010 Resubmitted in the context of the QTM8 on the SDMX tools maintenance. Update of this document depicting the SDMX Converter version 2.6.0. Sections 1, 3, and 5 have been updated.

2.19 14.02.2011 Resubmitted in the context of the QTM-2 on the SDMX tools maintenance. Update of this document depicting the SDMX Converter version 2.6.1. Sections 1, 3, and 5 have been updated.

2.20 15.03.2011 Resubmitted in the context of the QTM-1 on the SDMX tools maintenance. Update of this document depicting the SDMX Converter version 2.7.0. Sections 1, 2, 3, 5 and 8 have been updated.

2.21 18.03.2011 Update of this document to depict a conversion from DSPL to SDMX-ML and backwards using the API of the SDMX Converter. Section 3 has been updated.

2.22 15.04.2011 Resubmitted in the context of the QTM-1 on the SDMX tools maintenance under Subtask 4.3. Sections 1, 2, 3, 5 and 8 have been updated.

2.23 12.07.2011 Resubmitted in the context of the QTM-4 on the SDMX tools maintenance under Subtask 5. Sections 1, 3 and 5 have been updated.

2.24 05.05.2012 Resubmitted in the context of the QTM-1 on the SDMX tools maintenance under Task 4. Sections 1, 2, 3, 4, and 5 have been updated.

2.25 19.12.2012 Resubmitted in the context of the QTM-10 on the SDMX tools maintenance under SubTask 5. Sections 1, 3, and 5 have been updated.

2.26 08.03.2012 Resubmitted in the context of the QTM-10 on the SDMX tools maintenance under SubTask 5. Sections 1, 2, 3, and 5 have been updated.

3.0 14.11.2013 Added QTM-1 Excel to SDMX-ML conversion functionality.

3.1 27.01.2014 Validation is done also for SDMX output files. Header is needed also for Excel input files.

3.2 30.01.2014 Excel external parameter file is expected to be text file. OBS_LEVEL counted both when independent or in MIXED position type. In section 2 – Excel chapters


Page 4

3.3 11.02.2014 MIXED position type is used for each observation. In section 2 – Excel chapter

3.4 26.02.2014 List of jar libraries modified in Chapter 3 - “Interacting with the Converter programmatically”

3.5 31.03.2014 Add the SDMX 2.1 formats

3.6 14.05.2014 Canonical concepts for DSPL conversion – Chapter 1 and 2

4.0 31.07.2014 Update of document in the context of QTM-1 Task 1 on 2014. Sections 1 and 2 - CSV and FLR limitations, Header Information and Header Row in CSV/FLR parameters have been updated

4.1 7.10.2014 Update of document to depict: • a conversion from GESMES/TS using different names for

GESMES observation flags.( Chapter 2 ).

• The limitations of MessageGroup to CSV conversion

4.2 16.10.2014 • Update Figure 3 and exist_Row parameter to header_row in section 4. Calling converter application from a command prompt

• Added Appendix E explaining Excel Decimal Number handling

• Added Appendix F explaining the rounding mechanism implemented in Converter

• Added details about RoundingPrecision parameter in the Excel parameters section

4.3 06.02.2015 Update Apendix A section 1.4 – Error messages concerning Excel conversion.

4.4 02.03.2015 Update Chapter 4 - Calling converter application from a command prompt. Add all the formats supported by Converter CLI.

4.5 20.04.2015 Update Chapter 2 – More details about validation and namespaces added

4.6 22.05.2015 Update Chapter 2 - Converting from GESMES/TS

4.7 29.05.2015 Update Appenix A – Section 1.4 Error messages concerning Excel conversion

4.8 01.07.2015 Add Appendix G – CSV Input/Output escaping mechanism Updated Chapter 2 , Chapter4, Chapter 5 with more parameters for CSV input/output formats in accordance with the SDMXCONV-230

4.9 25.09.2015 The document refers to SDMX Converter 5.0.x and contains the following changes: • General improvements

• Changes related to STRUVAL

4.10 16.10.2015 New section 5.4 created - Using the “customValidation” method of the web service.


Page 5

4.11 1.04.2015 Section 4 “Calling converter application from a command prompt” - updated in accordance with SDMXCONV-351, 352, 353

4.12 09.05.2016 Update Excel Parameters in accordance with SDMXCONV-355. Update Converter WSDL in accordance with SDMXCONV-304, 337

Table of Contents

SDMX Converter Application User Manual .............................. 1

Document Change Record ............................................................ 2

1 Introduction ............................................................................... 10

1.1 Structure of this document ................................................................................ 10 1.1 Supported conversions ...................................................................................... 10 1.2 CSV and FLR limitations .................................................................................. 12 1.3 Conversion between DSPL and SDMX-ML messages .................................... 13 1.4 Limitations when converting Message Group SDMX-ML messages .............. 13

2 Converter Graphical User Interface (GUI) ........................... 15

2.1 Input / Output Files and Formats ...................................................................... 15 2.2 DSD ................................................................................................................... 16 2.3 Excel Parameter ................................................................................................ 17 2.4 CSV / FLR ........................................................................................................ 18 2.5 Other ................................................................................................................. 20 2.6 Namespace ........................................................................................................ 20 2.7 Performing a conversion ................................................................................... 20 2.8 Input & Output files and Formats ..................................................................... 21 2.9 Data Structure Definition selection ................................................................... 28 2.10 Excel Parameter ................................................................................................ 32

2.10.1 External Parameter File ............................................................................. 33 2.10.2 Map Parameters ........................................................................................ 35

2.11 CSV/FLR Parameters ........................................................................................ 37 2.11.1 Header Information ................................................................................... 38 2.11.2 Mapping .................................................................................................... 44 2.11.3 Mapping for CSV and FLR formats ......................................................... 45 2.11.4 Mapping for DSPL format ........................................................................ 53 2.11.5 Transcoding ............................................................................................... 55 2.11.6 Transcoding for CSV and FLR formats .................................................... 56 2.11.7 Transcoding for DSPL format .................................................................. 60 2.11.8 CSV Delimiter Character .......................................................................... 61 2.11.9 CSV/FLR Date Format ............................................................................. 62 2.11.10 CSV/FLR Ordered input ........................................................................... 62 2.11.11 Header Row .............................................................................................. 62 2.11.12 Write Header ............................................................................................. 64

2.12 Other Parameters ............................................................................................... 65 2.12.1 Gesmes/TS Technique .............................................................................. 65 2.12.2 SDMX Validation ..................................................................................... 65

2.13 Namespace ........................................................................................................ 67


Page 6

2.14 Specifying the encoding .................................................................................... 67 2.15 Performing a conversion ................................................................................... 68

3 Interacting with Converter programmatically ...................... 72

4 Calling converter application from a command prompt ..... 84

5 Using the Converter Web Service ........................................... 89

5.1 SOAP Web services ......................................................................................... 89 5.2 Web Service Descriptor for Converter (WSDL) .............................................. 89 5.1 Using the the “convert” method of the web service ......................................... 90 5.2 A sample C# web service client for “convert” .................................................. 93 5.3 Using the “validate” method of the web service ............................................... 93

5.3.1 “validate” method parameters ....................................................................... 94 5.3.2 “validate” method result ............................................................................... 94 5.3.3 A sample request to “validate” using SOAPUI tool ..................................... 96

5.4 Using the “customValidation” method of the web service ............................... 99 5.4.1 “customValidation” method parameters ....................................................... 99 5.4.2 “customValidate” method result ................................................................. 100

5.5 Using an implemented SDMX Converter WS Client ..................................... 100

6 APPENDIX A: Tables of error messages ............................. 104

6.1 Error messages for all usages of the SDMX Converter .................................. 104 6.2 Error messages concerning SDMX Converter GUI ........................................ 108 6.3 Error messages concerning SDMX Converter CLI ........................................ 110 6.4 Error messages concerning Excel conversion ................................................. 110

7 APPENDIX B – Structure of a multilevel CSV/FLR .......... 112

7.1 Multilevel files ................................................................................................ 112 7.2 Sample multilevel CSV file ............................................................................ 113 7.3 Sample CSV file with cross sectional measures that appear in one record .... 113

8 APPENDIX C – Sample template ......................................... 115

9 APPENDIX D - Examples of Excel to SDMX conversion .. 119

10 APPENDIX E – How Excel handles decimal numbers .... 120

10.1 Excel adheres to the IEEE 754 standard ......................................................... 120 10.2 What Excel displays is NOT what is stored internally ................................... 120

11 APPENDIX F – The rounding mechanism implemented in Converter .................................................................................... 121

12 APPENDIX G – CSV Input/Output escaping mechanism122

12.1 CSV as Input ................................................................................................... 122

13 APPENDIX H – Converter Web Service WSDL .............. 124


Page 7

14 Annex K: Checks done by the “validate” method of Converter Web Service ............................................................. 130

14.1 Syntax Validation ............................................................................................ 130 14.2 Structural Validation ....................................................................................... 130 14.3 Data Validation ............................................................................................... 131


Page 8

1. Table of figuresFigure 1 - Sample csv file ..............................................................................................................................12 Figure 2 - The csv file after suppressing the new line character ...................................................................12 Figure 3 - Main GUI Window .......................................................................................................................15 Figure 4: CSV/FLR Parameters ....................................................................................................................19 Figure 5 - Input and Output files browsing ...................................................................................................22 Figure 6 - Input / Output files choosing ........................................................................................................23 Figure 7 - Text fields with selected files ........................................................................................................24 Figure 8 - Batch Conversion .........................................................................................................................25 Figure 9 - Format selection ...........................................................................................................................26 Figure 10 - Output file field population ........................................................................................................27 Figure 11 - Warning message for overwriting output file .............................................................................28 Figure 12 - DSD Selection from file ..............................................................................................................29 Figure 13 - DSD Selection from Registry by specifying DSD .......................................................................30 Figure 14 - DSD Selection from Registry by specifying Dataflow ................................................................31 Figure 15 - DSD selection is disabled when converting from DSPL dataset ................................................32 Figure 16 - Excel Parameter section enabled for Excel Input Format .........................................................33 Figure 17 - Mapping parameters window .....................................................................................................36 Figure 18 - Map Parameters is pressed but no parameter sheet was found inside Excel .............................37 Figure 19 - ‘sample_header.prop’ file ..........................................................................................................39 Figure 20 - selecting the header file ..............................................................................................................40 Figure 21 - Edit SDMX Header .....................................................................................................................41 Figure 22 - Edit SDMX Header Dialog .........................................................................................................42 Figure 23 - Contact Information dialog for Sender and Receiver .................................................................43 Figure 24 - Warning message for overwriting header file ............................................................................43 Figure 25 - Select a value from the combo box .............................................................................................44 Figure 26 - Mapping GUI .............................................................................................................................45 Figure 27 - Map cross sectional measures ....................................................................................................46 Figure 28 - Assigned values for mapping ......................................................................................................47 Figure 29 - Warning message for overwriting mapping file .........................................................................48 Figure 30 - Non-mapped column ...................................................................................................................49 Figure 31 - Error message ............................................................................................................................49 Figure 32 - Setting fixed value ......................................................................................................................50 Figure 33 - Number of levels .........................................................................................................................52 Figure 34 - Mapping for a multilevel CSV file ..............................................................................................53 Figure 35 - Mapping GUI for DSPL .............................................................................................................54 Figure 36 - DSPL concepts in mapping dialog .............................................................................................55 Figure 37 - Transcoding GUI ........................................................................................................................56 Figure 38 - Adding transcoding rules ...........................................................................................................57 Figure 39 - A transcoding rule ......................................................................................................................57 Figure 40 - Dimensions with transcoding .....................................................................................................58 Figure 41 - Warning message for losing transcoding rules ..........................................................................58 Figure 42 - Warning message for overwriting transcoding file ....................................................................59 Figure 43 - Transcoding GUI ........................................................................................................................60 Figure 44 - Transcoding Rule with DSPL codes ...........................................................................................61 Figure 45 - Header Row parameter for CSV/FLR formats ...........................................................................64 Figure 46 - Missing field message .................................................................................................................68 Figure 47 - Conversion completed ................................................................................................................69 Figure 48 - Load a Template .........................................................................................................................69 Figure 49 - Mapping Loaded.........................................................................................................................70 Figure 50 - Invalid template file has been loaded .........................................................................................70 Figure 51 - Warning message for overwriting a template file .......................................................................71 Figure 52 – Java project properties in Eclipse .............................................................................................72 Figure 53 - Project location ..........................................................................................................................73 Figure 54 - SDMX Converter Platform Independent Package ......................................................................84 Figure 55 - SDMX Converter Windows Installer ..........................................................................................84 Figure 56 - New SoapUI project for Converter Web Service ........................................................................97 Figure 57 - Converter WS configured in SoapUI ..........................................................................................97 Figure 58 - Adding attachments to the "validate" request ............................................................................98 Figure 59 - the SOAP response of the validate method .................................................................................99 Figure 60 - Test Client in the SDMX Converter Web Service Package ......................................................101


Page 9

Figure 61 - Testing the SDMX Converter WS .............................................................................................101 Figure 62 - Converter WS Client .................................................................................................................101 Figure 63 - Instance of a Converter WS Client ...........................................................................................103


Page 10

1 Introduction The SDMX converter application offers to the user the ability to convert between all the existing formats of the SDMX version 2.0 standard (generic, compact, utility and cross-sectional), GESMES (TS, 2.1, DSIS), CSV and FLR formats (please see CSV and FLR limitations) and DSPL - Dataset Publishing Language (see Conversion between DSPL and SDMX-ML messages).

1.1 Structure of this document

This document is structured along the following chapters.

Chapter 2 of the document explains the Converter User Interface and operation.

Chapter 3 of this document outlines the interaction with the Converter application via its programming API.

Chapter 4 of this document outlines the interaction with the Converter using a command line interface (CLI).

Chapter 5 of this document outlines the interaction with the Converter web service.

Several appendixes are included at the end of the documents as Chapter 6. They contain sections related to the possible errors generated by the application, how the application is handling the CSV input/output and some details about the Converter web service.

1.1 Supported conversions

The table below presents all transformations supported by SDMX Converter:


Page 11

Input

Output

CSV GESMES

TS GESMES

2.1 GESMES

DSIS SDMX

GENERIC SDMX

COMPACT SMDX

UTILITY SDMX CROSS SECTIONAL FLR DSPL EXCEL

SDMX 2.1 (*****)

Message Group (******)

CSV (**) NO YES YES YES YES YES YES YES(*) NO NO YES YES YES

GESMES/TS

YES NO YES YES YES YES YES YES(*) YES NO YES YES NO

GESMES/2.1

YES YES NO YES YES YES YES YES(*) YES NO YES YES NO

GESMES/DSIS

YES YES YES NO YES YES YES YES(*) YES NO YES YES NO

SDMX GENERIC

YES YES YES YES NO YES YES YES(*) YES YES YES YES NO

SDMX COMPACT

YES YES YES YES YES NO YES YES(*) YES YES YES YES NO

SMDX UTILITY

YES YES YES YES YES YES NO YES(*) YES YES YES YES NO

SDMX CROSS-SECTIONAL

YES

(*)

YES

(*)

YES

(*)

YES

(*)

YES

(*)

YES

(*)

YES

(*) NO

YES

(*) YES NO YES NO

FLR(**) NO YES YES YES YES YES YES YES

(*) NO NO NO YES NO

DSPL NO NO NO NO YES

(***)

YES

(***)

YES

(***)

YES

(***) NO NO

NO

YES NO

SDMX 2.1 YES YES YES YES YES YES YES YES

(****) NO YES YES NO NO


Page 12

(*) The selected DSD for these files must support cross-sectional messages i.e. having defined cross-sectional measures and cross-sectional attachment levels for its components (see SDMX standard www.sdmx.org)

(**)When the output format (CSV, FLR) is a flat file the attributes, attached in higher level than observations in the input message are omitted.

(****)The provided DSD has to be valid with Time dimension or cross sectional measures.

(*****)With exception of SDMX V2.1 formats no other format supports multiple datasets in a single data message. Consequently Converter only supports conversions of SDMX V2.1 files having a single dataset per message. Also conversion from SDMX 2.1 to SDMX 2.0 formats is possible only when the DSD is compatible with both formats. The compatibility with V2.0 formats is checked in SDMX Source API before conversion begins.

(******)The Message Group input file should have only one dataset reported in the message.

1.2 CSV and FLR limitations

In order for a flat CSV or FLR file to be converted successfully some limitations exist. Each line in such a file represents one record. This implies that no new line characters should be present inside the data in that record. However there exist some cases where new line characters can be present in data. For example attributes may represent free text data that contain new line characters. Below is given a screenshot that illustrates this case:

Figure 1 - Sample csv file

In such a case user should suppress all these new line characters from the file.

Figure 2 - The csv file after suppressing the new line character

For flat CSV files also the attributes at a higher level than Observation are processed. The user has to repeat the respective attribute on every related observation if not the first encountered value is used and the rest are ignored. In case the values of the attributes other than the Observation level are not the same for the same group or time series then a warning is printed in the log. The conversion will be successful using the first encountered value.


Page 13

In the mapping file all the dimensions, attributes, primary measure, time reference etc must be provided according to the DSD. For default mapping the implicit order is: dimensions, time dimension, primary measure, observation level attributes, dataset attributes, group attributes and series level attributes. For multilevel files and those containing groups is mandatory to have the data ordered - rows for the same series or group are consecutive (Input Ordered). Input has to be ordered also when converting to cross sectional format. Please see section about Header Row in section CSV/FLR parameters for details on how to handle the first row of a CSV/FLR file.

1.3 Conversion between DSPL and SDMX-ML messages

DSPL is a representation language for data and metadata of datasets. An analysis document has been prepared for the feasibility of the conversion between DSPL and SDMX-ML messages (please consult Analysis document for conversions between SDMXML and DSXML v1.1). Furthermore, elaborated information on the DSPL is available at http://code.google.com/apis/publicdata/. Please note as general guidelines the following:

When converting from data in DSPL format to SDMX-ML message no other parameter need to be defined. A DSD is constructed by the application according to the metadata of the DSPL input. Furthermore the output file is a zip file as it might contain more than one SDMX-ML messages. This depends on the number of DSPL slices that exist in the input.

When converting from SDMX-ML messages to DSPL dataset a DSD should be defined and a mapping as well as a transcoding file could be provided. The mapping will map the DSD components to DSPL concepts. The application retrieves the DSPL concepts to be mapped via an http accessible repository from Google. In case such a connection is not available then the conversion can continue without a specific mapping.

The output file of a DSPL dataset is a zip file as it contains data and metadata (csv files and an xml file).

1.4 Limitations when converting Message Group SDMX-ML messages

The MessageGroup is used to allow more than one data message to be included in a single transmission. Currently, Converter supports Message Group as input sdmx-ml file for all four different representations of reporting datasets: Generic data message, Compact data Message, Utility data Message and Cross Sectional Data Message. This support has the following limitations when converting Message Group sdmx-ml messages:

The single output format supported for Message Group messages is CSV

The input sdmx-ml file must have only ONE dataset reported in the message


Page 14


Page 15

2 Converter Graphical User Interface (GUI) In Error! Reference source not found. below, the main GUI of the application is presented:

Figure 3 - Main GUI Window

The main GUI is divided in seven main sections:

2.1 Input / Output Files and Formats

In this section the following parameters are provided:

Input File: The path of the input file (file to be converted). For example D:\Test\Input.txt could be used as an input file, where D is the drive, \Test\ is the path on the user’s machine and Input.txt the input file to be converted.

Output File: The path of the output files (the converted / generated file). For example D:\Test\Output.txt could be used as an output file, where D is the drive, \Test\ is the path on the user’s machine and Output.txt the output converted file. Alternatively the name of the converted file (for example Output.txt) could be used as the Output File parameter, without defining a destination folder. In that case the application creates a


Page 16

folder named “Output Files” in the local repository folder (Converter_Data/) of the SDMX Converter. The output-converted file is placed in that folder.

Input Format: The input file format (SDMX Generic, SDMX Compact, SDMX Utility, GESMES/TS, GESMES/2.1, GESMES/DSIS, SDMX Cross-Sectional, FLR, CSV, DSPL). If the input file is of SDMX 2.0 format then the Input Format is automatically set to the to the respective kind i.e. GENERIC_SDMX for a Generic Data message, COMPACT_SDMX for a Compact Data message, UTILITY_SDMX for a Utility Data message, CROSS_SDMX for a Cross-sectional Data message, EXCEL for an Excel file, GENERIC_DATA_2_1 and GENERIC_TS_DATA_2_1 (generic time series data) for SDMX 2.1 generic messages, STRUCTURE_SPECIFIC_DATA_2_1 and STRUCTURE_SPECIFIC_TS_DATA_2_1 for SDMX 2.1 structure specific data messages – the equivalent of compact data in SDMX 2.0).

Output Format: The output file format (SDMX Generic, SDMX Compact, SDMX Utility, GESMES/TS, GESMES/2.1, GESMES/DSIS, SDMX Cross-Sectional, FLR, CSV, DSPL, GENERIC_DATA_2_1, GENERIC_TS_DATA_2_1, STRUCTURE_SPECIFIC_DATA_2_1, STRUCTURE_SPECIFIC_TS_DATA_2_1).

The fields of this section are mandatory for every possible combination of input and output file formats, except for when converting from DSPL to SDMX-ML message.

2.2 DSD

In this section the user provides parameters about the DSD (Data Structure Definition). In general, the DSD to be used may be a structure file that is stored locally, or may be acquired through the Registry. Moreover a structure file might have multiple DSDs. In this case the user can define which DSD to be used in the conversion by specifying the DSD agency, the DSD id and the DSD version. Otherwise the SDMX Converter display an error message to ask the user to specify the DSD and the conversion will not be performed.

Use Registry: If this field is set to “False”, the user must provide the DSD through a locally stored file. If this field is set to “True”, the user must provide the information that is needed to access the DSD file through the Registry. In that case, the Registry that will be used will be the one specified in the config.txt file (please consult the SDMX Converter Installation Guide Section 4). Moreover when using the registry to retrieve the DSD, the user can alternatively select to specify the dataflow by providing the necessary information for it i.e. Dataflow Id, Dataflow Version and Agency Id; then the DSD that this dataflow refers to can be retrieved by Registry.

DSD File: The path of the DSD file. This field is mandatory only if the Use Registry field is set to “False”.

DSD ID, DSD Agency and DSD Version: the fields with the ID, Agency and Version of the DSD. These fields can be set with appropriate values in the following cases

• If the Use Registry field is set to “True” and the Specify Dataflow is set to “False”.


Page 17

• If the structure xml file has multiple DSDs and the one to be used for the conversion has to be defined.

Dataflow ID, Dataflow Agency and Dataflow Version: the fields with the ID, Agency and Version of the Dataflow. These fields are mandatory only if the Specify Dataflow is set to “True”.

This section is also mandatory for every possible selection combination between input and output file format.

2.3 Excel Parameter

This section is enabled only when the input format is Excel.

Parameter File: The path of the external parameter file (file that contains the description layout of the Excel parameter to be converted). The external parameter file is expected to be a text file. For example D:\Test\ExternalParameter.txt could be used as an input file, where D is the drive; \Test\ is the path on the user’s machine and ExternalParameter.txt the parameter file to be used for processing the data inside the Excel input file.

Map Parameters: In case the description of the data sheets layout resides inside the Excel input file and there is more than one parameter sheet a mapping can be provided to indicate for each data sheet what parameter sheet has to be used for converting. For example if the Excel input file contains the following sheets with data: 1500S, 1600S, 1750S, 1850S and sheets: Parameters1, Parameters2 and Parameters3 as description of the layout. A possible mapping can be:

1500S – Parameters1 1600S – Parameters1 1750S – Parameters2 1850S – Parameters3

Meaning that 1500S and 1600S will be processed according to the description in sheet Parameters1, 1750S will be processed according to the description provided in Parameters2 and 1800S according to Parameters3 sheet content.


Page 18

2.4 CSV / FLR

This section concerns parameters of CSV and FLR input and output files. These parameters are:

Edit Header: If a header is to be loaded through the Edit SDMX Header button. This field is mandatory only for CSV, FLR and EXCEL input files. When it is selected the Header parameter is disabled.

Header: The path to the file containing the CSV, FLR or Excel header. This field is mandatory only for CSV and FLR input files. When it is selected the Edit Header parameter is not selected.

Levels of CSV/FLR file: The number of levels that a CSV or FLR multilevel file contains. The default value for this option is 1 i.e. flat file. Applicable only in conversions from and to CSV/FLR files.

Output Date Format: Through this field the user may specify whether the date format to be used in the output CSV or FLR file must be SDMX compatible (i.e. YYYY-MM) or Gesmes compatible (YYYYMM).

Default Mapping: This field may refer either to the input or the output file. Depending on whether the input is CSV or FLR or the output is CSV or FLR or DSPL (only one may be CSV or FLR or DSPL), the semantics of this field change.

If the input file is CSV or FLR, this field is used to specify whether the input file is written with the default mapping (dimensions and attributes appear in the same order as in the DSD). If the field is unchecked, the Change Mapping button is enabled. The combo box next to ‘Default Mapping’ field is enabled only if the selected DSD has multiple cross-sectional measures. Through this combo box the user can decide what components to map. The options are two, map the measure dimension or the cross sectional measures. Through the Change Mapping button the user may specify the mapping that is used.

If the output file is CSV or FLR or DSPL, this field is used to specify whether the output file will be written with the default mapping (dimensions and attributes appear in the same order as in the DSD). If the field is unchecked, the Change Mapping button is enabled. The combo box next to ‘Default Mapping’ field is enabled only if the selected DSD has multiple cross-sectional measures. Through this combo box the user can decide what components to map. The options are two, map the measure dimension or the cross sectional measures. Through the Change Mapping button the user may specify the mapping that is used.

Input ordered: This field is used to specify whether the CSV or FLR input file is ordered or not.

Transcoding: This field may refer either to the input or the output file. In case the input file is CSV or FLR and the output is SDMX as well as in case the input file is SDMX and the output file is CSV or FLR the user can specify transcoding rules (rules concerning the codes that each coded component will have in the output file) through the Transcoding button.

CSV delimiter: If the input file is CSV, this field is used to specify the delimiter that is used. If the output file is CSV, this field is used to specify the delimiter to be used.


Page 19

Header Row: When converting from CSV/FLR file this field is used to specify whether the first line of the input file to be ignored in the conversion procedure. This depicts the case where the input file contains a row with arbitrary data that should not be processed and if mapping is defined based on the column headers from the first row. When converting to CSV/FLR file this field is used to specify whether in the first line of the output file the name fields of the reporting values should be written.

Please see Header Row section from CSV/FLR parameters for exact details related to header row.

Write Header: This field is used to specify whether a properties file containing header information (please see section: CSV and FLR limitations) should be written when converting to CSV/FLR file.

Unescape CSV Inputs Fields: Specify whether input CSV file contains quoted values or escaped special characters (please see Appendix G – CSV Input/Output Escaping Mechanism).

Escape CSV Output Fields: If checked Converter will enclose all outputted values in double quotes (please see Appendix G – CSV Input/Output Escaping Mechanism).

Figure 4: CSV/FLR Parameters


Page 20

2.5 Other

Gesmes/TS Technique: This field can be used only when the output file is Gesmes/TS, to determine whether the technique to be used will be “Time-Range” or “Single-Observation”1.

SDMX Validation: This field should be used only when the input or output file is SDMX (SDMX 2.0 formats: Generic, Compact, Utility or Cross-Sectional or a SDMX 2.1 format: GENERIC_DATA_2_1, GENERIC_TS_DATA_2_1, STRUCTURE_SPECIFIC_DATA_2_1, STRUCTURE_SPECIFIC_TS_DATA_2_1). If checked, validation is performed in the input or output file using the SDMX validator. The set of rules concern the following:

o syntax validation to examine if the input message is syntactically correct

o code validation against the DSD to examine if codes belong to specified codelists

2.6 Namespace

This section contains fields about the namespace that will be used in output SDMX files.

Default Namespace: This field is used to determine if the namespace to be used for the output SDMX file will be generated from information in the DSD (Agency Id, XSD url etc.) and from the default package name of the Registry as described in the SDMX Registry specification section 5 of the standard (i.e. “urn:sdmx:org.sdmx.infomodel”).

Namespace: If the “Default Namespace” field is unchecked, this field is enabled. It is used to specify the namespace to be used.

Prefix: If the “Default Namespace” field is unchecked, this field is enabled. It is used to specify the prefix to be used.

2.7 Performing a conversion

Load Template: This field is used to load a saved template for a conversion that will populate the parameters in the gui dialog

Save Template: This field is used to save in an xml file the template for a conversion to be performed

Convert: This field is used to perform the conversion.

1 http://www.sdmx.org/docs/1_0/SDMX%201_0%20SECTION_04_SDMX-EDI.pdf


Page 21

A detailed description of the operations that can be performed using the GUI, as well as an explanation of the fields that are included in it, will be presented in the following paragraphs.

Converting from GESMES/TS

In GESMES/TS (SDMX-EDI) there are 2 kinds of observation attributes:

• the observation flags which are part of the ARR segments

• the normal attributes that are included in the attribute section.

The observation flags are fixed. They are the observation status, observation confidentiality and observation pre-break. In SDMX-ML the concept of "observation flags" does not exist. In order to map between SDMX-ML DSD observation attributes and SDMX-EDI observation flags the OBS_CONF or CONF_STATUS are considered to be the concept for “observation confidentiality”. Also OBS_PRE_BREAK or PRE_BREAK_VALUE are considered to be the concept for "observation pre-break". These attribute names are hard-coded in SDMX Converter and, at conversion time, checked which one exists in DSD to be used in the output. If any of these names was not found in DSD the OBS_CONF/OBS_PRE_BREAK pair is used.

Notes:

1) If in the future other attributes names will be used in DSD for observation flags in GESMES/TS the Converter needs to be updated as the above names are hard coded.

2) Parsing of ARR segment in the input Gesmes file is dependent by the value of the frequency code. As the frequency code lists could evolve over time Converter needs to be updated in order to support new codes.

Currently Converter supports A, H, Q, M, W, B or D as values for frequency in the Gesmes file. B and D are interpreted in the same way.

2.8 Input & Output files and Formats

Use the [BROWSE] button in order to select the input and the output files as Error! Reference source not found. presents:


Page 22

Figure 5 - Input and Output files browsing


Page 23

Provide an input file (the file that will be converted) and an output file (the generated/converted file). Use the [BROWSE] buttons of the “Input / Output Files and Formats” section of the user interface. When these buttons are clicked, a file browser will appear as Error! Reference source not found. presents:

Figure 6 - Input / Output files choosing


Page 24

By selecting input and output files the respective text fields will be filled with the path of the input and output file at “Input File” and “Output File” fields respectively, as Error! Reference source not found. presents:

Figure 7 - Text fields with selected files

Alternatively more than one file may be selected for conversion, by pressing the control key. Then the “Input File” field is filled with all the paths of the files that have been selected while the “Output File” field is filled with the destination folder and not the converted file as in the case that one conversion is performed. In this destination folder all the output files of the conversion will be stored. The name for each output file will be constructed by the name of the input file, the output format and a number that is incremented. For example if the following two SDMX Generic files are to be converted “2_IN_Generic_Sts2.2.xml” and “4.3_IN_Generic.xml” and the selected output format is SDMX Compact, then in the output destination folder the two converted SDMX Compact files will be named us “2_IN_Generic_Sts2.2_COMPACT_SDMX_OUT_0.xml” and “4.3_IN_Generic_COMPACT_SDMX_OUT_1.xml”


Page 25

The following picture depicts the batch conversion. When the cursor is over the “Input File” text field a tool tip appears with all the files that have been selected. The “Output File” field is filled with a destination and not a file.

Figure 8 - Batch Conversion

For the batch conversion to be performed successfully all the selected input files should have the same format.

For each input and output file, select its respective format in the combo boxes at the right side of the “Input / Output Files and Formats” section as presented in Error! Reference source not found. below.


Page 26

Figure 9 - Format selection

Whenever the user selects an output format and the output file field does not contain a path or a file name the field is populated by default with an extension. The extensions according to the selected output format are the following:

• Extension “.csv” for a CSV output format

• Extension “.txt” for an FLR output format

• Extension “.ges” for a GESMES/TS, GESMES/2_1 or GESMES/DSIS output format

• Extension “.xml” for a GENERIC_SDMX, COMPACT_SDMX, UTILITY_SDMX or CROSS_SDMX output format

• Extension “.zip” for a DSPL format.

The following picture depicts the situation where the user selects the output format COMPACT_SDMX and the output field is populated with the “.xml” extension


Page 27

Figure 10 - Output file field population

If a user selects to convert a file and has not provided a name for the output file then the application by default provides a name constructed by the “Output_” word and the selected output format. If a user selects to convert more than one files and has not provided a path for the output file to be placed, then the application by default populates the Output file field with the name of the folder in the local repository folder (Converter_Data/Output Files) of the SDMX Converter (“Output Files”).

In two consecutive conversions if the Output file field has not changed, then a warning message appears informing the user that he is going to overwrite the output file.


Page 28

Figure 11 - Warning message for overwriting output file

Important note: The selected input and output formats must be in agreement with the selected input and output files. For instance, if the selected Input Format is a GENERIC_SDMX, then the input file should be a valid SDMX Generic file.

2.9 Data Structure Definition selection

After the file and the format selection, provide all the necessary input parameters in order to select a DSD.

The DSD can be retrieved with two ways:

From a local file: In that case the “Use Registry” combo box should have a false value and the path of the DSD file should be provided as Error! Reference source not found. presents.


Page 29

Figure 12 - DSD Selection from file

From Registry: In that case the “Use Registry” combo box should have a true value and the DSD ID, DSD Agency, DSD version should be provided as following presents:


Page 30

Figure 13 - DSD Selection from Registry by specifying DSD

Alternatively the dataflow can be specified and the DSD that this dataflow refers to is retrieved from Registry. In this case the Dataflow ID, Dataflow Agency, Dataflow Version should be provided as following Error! Reference source not found. presents:


Page 31

Figure 14 - DSD Selection from Registry by specifying Dataflow

When converting from DSPL to SDMX-ML all sections apart from “Input/Output Files and Formats” and “Namespace” are disabled. A DSD is constructed by the application and exploited when writing the output message.


Page 32

Figure 15 - DSD selection is disabled when converting from DSPL dataset

Important note: Registry URL and Registry Action should be set in the config.txt file (please consult the SDMX Converter Installation Guide Section 4). In case that the config.txt file is empty or it is not configured properly, the application will prompt the user with the respective message (see Appendix A Error messages message 8)

2.10 Excel Parameter

The input parameters are:

• the external parameter file

• the mapping parameters

This section is enabled only when Excel is selected as Input Format as seen in Figure 15. The input file can only be xls or xlsx. Inside the Excel file the sheets starting with “VAL” will be ignored, those starting with “Parameter” will be considered parameter sheets which contain description of the layout of the data sheets. All the rest are considered data sheets containing the actual data to be processed.


Page 33

Figure 16 - Excel Parameter section enabled for Excel Input Format

2.10.1 External Parameter File

Parameter File is used as layout description of the Excel input file. If the external parameter file is provided and mapping parameter is not provided then the external parameter file will be used for all data sheets even if the Excel file contains other parameter sheets.

Element Type PosType Position DataStart B30 STS_INDICATOR DIM ROW 26 NumColumns 17 STS_ACTIVITY DIM ROW 27 DefaultValue NaN PRICES DIM CELL B6 RoundingPrecision 4 REF_AREA DIM CELL B2 FREQ DIM CELL I16 ADJUSTMENT DIM CELL B4 REF_YEAR DIM CELL D12 UNIT_MEASURE DIM CELL B21 STS_BASE_YEAR DIM FIX 2000 STS_INSTITUTION DIM CELL B10 TIME_PERIOD DIM MIXED ROW 25 COLUMN 1 OBS_STATUS ATT CELL D4 OBS_CONF ATT CELL D5 UNIT ATT CELL D2 UNIT_MULT ATT CELL D3 TITLE ATT FIX ESA95 Questionnaire 0101


Page 34

UNIT_INDEX_BASE ATT SKIP TITLE_COMPL ATT CELL D11 NAT_TITLE ATT FIX nan COMPILATION ATT SKIP BREAKS ATT CELL D12 COVERAGE ATT SKIP SOURCE_PUB ATT SKIP OBS_COM ATT SKIP SOURCE_AGENCY ATT SKIP TIME_FORMAT ATT FIX P3M OBS_STATUS ATT OBS_VALUE 1 OBS_CONF ATT CELL D5 OBS_PRE_BREAK ATT SKIP

The external parameter file must follow the rules below:

• All the contained data must be separated by space. Only the value after FIX position type can have a value containing space character.

• The parameter file must contain on the first row the column headers.

• The Element is the name of the dimension or attribute exactly as it is named in the DSD file.

• The Type can only be DIM (dimension) or ATT (attribute).

• PosType is CELL, ROW, COLUMN, FIX, SKIP, OBS_LEVEL or MIXED.

• DataStart is the position where the actual data starts.

• Either NumColumns, the number of columns containing data, or MaxEmptyRows, the maximum row allowed to be empty before finishing reading data, must be provided.

• DataStart and NumColumns (or MaxEmptyRow) can be defined to the left of the Elements or at the end of the file.

• MaxEmptyColumns, the maximum number of empty cells allowed in the current row before start reading the next row. If not present the default is 1000.

• When MIXED is used as PosType not all the combinations are allowed. For example VALUATION DIM MIXED COLUMNS 5 ROW 20 is a valid description if COLUMN 5 is found empty ROW 20 will be used. The same for VALUATION DIM MIXED CELL B5 FIX 23 if CELL B5 is empty the FIX value ‘23’ will be used. Unaccepted combinations are between FIX, SKIP, OBS_VALUE. e.g. MIXED OBS_VALUE 1 FIX 25 will fail.

• When OBS_LEVEL appears the column corresponding to the Observation values is expected to exist even if OBS_LEVEL is part of MIXED position type. e.g. For the declaration below, if H14 cell is found non empty the converter application will skip one column to the right which represents the observation level attributes that can be empty or non empty.

OBS_STATUS ATT MIXED CELL H14 OBS_LEVEL 1

• For MIXED the values are checked for each Observation. So for the example below for each Observation the observation level is checked and it the cell at that corresponding position is empty the value from the cell H14 will be used.

OBS_STATUS ATT MIXED OBS_LEVEL 1 CELL H14

• If an Observation value is left empty intentionally, the data sender might to transmit further attributes (e.g. OBS_STATUS), giving reasoning on the fact that the observation is blank. For this a new parameter is added called “DefaultValue”. This


Page 35

parameter is optional. If present and all the mandatory dimensions can be resolved when parsing the empty observation cell, the Converter will create an observation with the observation value equals with the value specified in the DefaultValue parameter (this could be either a 0, a " " or a "NaN" for example, the converter will only put the value specified in the output file). This parameter can be defined to the left of the Elements in the second group of parameters (DataStart, MaxEmptyRows).

DefaultValue “NaN”

• For all cells storing NUMBERs, Excel uses internally a rounding mechanism in order to avoid displaying floating point approximations induced by the IEEE 754 standard (see appendix E and appendix F for more details). In order to control the rounding performed by Converter for Excel decimal numbers, a new parameter (RoundingPrecision) has been introduced and will be applied for converting the values from all cells storing NUMBERs as well as from cells having a formula which evaluates to a NUMBER. The RoundingPrecision parameter is optional, if not present Converter uses 6 as default precision number. The parameter value should be an integer between 0 and 15. If it is 0 the numbers will be rounded without decimals. If the parameter value is greater than 15 then the value will be ignored and 15 will be used. If the parameter value is a negative integer then the default value (6) is used.

RoundingPrecision 4 The same rules apply to Excel Parameter Sheets with the difference that each piece of information is stored in its own cell instead of being separated by space.

2.10.2 Map Parameters

When Excel input format is selected and the Excel file contains more than one parameter sheet describing the data layout then a mapping is needed to know which parameter sheet to use for processing the each data sheet. If no mapping is provided and no external parameter is given then the first parameter sheet will be used for all data sheets. If the mapping exists then it has priority even an external file was uploaded.


Page 36

Figure 17 - Mapping parameters window

As seen in Figure 17 for each data sheet found in the Excel input file a parameter sheet has to be selected. In this example the Excel file has two data sheets names Data1 and Data2 and two parameter sheets named Parameters_Data1 and Parameter_Data2.


Page 37

Figure 18 - Map Parameters is pressed but no parameter sheet was found inside Excel

When the Excel input file does not contain any parameter sheets then a mapping is not possible and accesing Map Parameters button will fail with the message as in Figure 18. In this case an external parameter file is mandatory for the converstion to be succesful. Please check APPENDIX D for Excel input and enternal parameter file examples.

2.11 CSV/FLR Parameters

The input parameters are: • The header information • The default/manual mapping • The CSV delimiter character • The ordered input checkbox • The output date format • The header row dropdown box • The write header checkbox • The Unescape CSV Input Fields checkbox • The Escape CSV Output Fields checkbox


Page 38

The following chapters explain in detail all the parameters.

2.11.1 Header Information

Use the [BROWSE] button to select the Header file. This file should contain the following variables, according to the SDMX_2_0_SECTION_03A_SDMX_ML standard2:

• header.id=JD014 • header.test=true • header.truncated=false • header.name=Trans46302 • header.prepared=2001-03-11T09:30:47-05:00 • header.senderid=BIS • header.sendername=Bank for International Settlements • header.sendercontactname=G.B. Smith • header.sendercontactdepartment=Statistics • header.sendercontactrole=Director • header.sendercontacttelephone=210 1111111 • header.sendercontactfax=210 1110999 • header.sendercontactx400= • header.sendercontacturi=www.sdmx.org • [email protected] • header.receiverid=ECB • header.receivername=European Central Bank • header.receivercontactname=John Smith • header.receivercontactdepartment=Statistics • header.receivercontactrole=Director • header.receivercontacttelephone=210 1234567 • header.receivercontactfax=210 123456999 • header.receivercontactx400= • header.receivercontacturi=www.sdmx.org • [email protected] • header.datasetagency=BIS • header.datasetid=BIS_JD_237 • header.datasetaction=Append • header.extracted=2001-03-11T09:30:47-05:00 • header.reportingbegin=2000-01-01T00:00:00 • header.reportingend=2000-01-01T00:00:00 • header.source= • header.lang=en • header.keyfamilyref=KFR • header.keyfamilyagency=KFA

In the above variables, sample values have been assigned for presentation purposes only.

Header files should have suffix ‘.prop’. A sample of such a file named ‘sample_header.prop’ can be found in the ‘(package)/Converter_data/params/’ directory structure. The following picture depicts the ‘sample_header.prop’ file.

2 http://sdmx.org/docs/2_0/SDMX_2_0_SECTION_03A_SDMX_ML.pdf


Page 39

Figure 19 - ‘sample_header.prop’ file

By selecting a header file, the SDMX Converter GUI, should look like Error! Reference source not found. below:


Page 40

Figure 20 - selecting the header file


Page 41

Alternatively check the Edit Header checkbox to enable the [Edit SDMX Header] button and disable the [BROWSE] button to select the Header file, as the following picture depicts.

Figure 21 - Edit SDMX Header

Use the [Edit SDMX Header] button to load a header file and view it in the edit header dialog as it is shown in the following picture.


Page 42

Figure 22 - Edit SDMX Header Dialog

Change the values by editing them in the text fields.

For the Sender and Receiver press the [Provide Sender Info] and [Provide Receiver Info] buttons respectively to edit the values. The following picture presents the contact information dialog that appears when pressing the above buttons.


Page 43

Figure 23 - Contact Information dialog for Sender and Receiver

Use the [Save] button to save the loaded or edited header properties in a prop file. Use the [OK] button for the header to be used in the conversion.

If the header prop file to be saved has the same name with the previously saved header prop file, then a warning message appears informing the user that he is going to overwrite the header prop file.

Figure 24 - Warning message for overwriting header file


Page 44

2.11.2 Mapping

The mapping applies only for CSV and FLR Input Formats and DSPL output formats. In the mapping, the user assigns the order of every dimension, primary measure and observation attribute concerning the CSV and FLR formats. When the conversion concerns a DSPL output format in the mapping the user assigns a DSPL concept that is mapped to an SDMX dimension and primary measure.

By checking the “Default Mapping” check box, the application will use the default mapping, as it comes from the selected DSD.

By un-checking the “Default Mapping” check box, the [CHANGE MAPPING] button will be enabled. The combo box next to “Default Mapping check box” refers to whether the user wants to map the measure dimension or the cross sectional measures, in case the DSD used in the conversion is a cross sectional DSD with multiple crossX measures (please see Error! Reference source not found.). The above selection defines the components that will appear in the mapping GUI dialog. If the DSD is not a cross sectional one, or it is a cross-sectional without crossX measures, then this combo box becomes disabled.

Figure 25 - Select a value from the combo box


Page 45

2.11.3 Mapping for CSV and FLR formats

Click on the [CHANGE MAPPING] button and the following screen presented in Error! Reference source not found., will appear:

Figure 26 - Mapping GUI

This screen presents all the dimensions, the primary measure and the observation attributes of the selected DSD.

In case the user has selected to map cross sectional measures instead of measure dimension then in the mapping GUI dialog the primary measure as well as the measure dimension will not appear. Only the cross sectional measures will be available to be mapped (Error! Reference source not found.). Please see in APPENDIX B a sample CSV file that illustrates the above case.


Page 46

Figure 27 - Map cross sectional measures

The user may change mapping by providing other value(s) to each field.

In order to concatenate two or more fields use the “+” character (plus symbol). In order to assign a range of columns for a field use the “-“character (dash symbol).


Page 47

Error! Reference source not found. below presents such an assignment:

Figure 28 - Assigned values for mapping

The above assignment, declares that the columns 4 and 5 of the CSV Input File will concatenate into the DEMO dimension. Click on [OK] button to confirm the mapping change or click on [CANCEL] button to cancel it.

There are also four other buttons providing helpful functionalities:

• [Load from file] button allows the loading of a predefined mapping in XML format.

• [Save to file] button allows to save the mapping in XML format

• [Default Mapping] button fills all the fields with the default mapping

• [Clear Fields] button clears the values from all the fields.

If the mapping file to be saved has the same name with the previously saved mapping file, then a warning message appears informing the user that he is going to overwrite the mapping file.


Page 48

Figure 29 - Warning message for overwriting mapping file

A sample of the above-mentioned XML format for mapping is shown below, for another DSD: <?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="FREQ" value="1" level="" fixed="false"/> <Concept name="JD_TYPE" value="2" level="" fixed="false"/> <Concept name="JD_CATEGORY" value="3" level="" fixed="false"/> <Concept name="VIS_CTY" value="4" level="" fixed="false"/> <Concept name="TIME" value="5" level="" fixed="false"/> <Concept name="OBS_VALUE" value="6" level="" fixed="false"/> <Concept name="OBS_CONF" value="7" level="" fixed="false"/> <Concept name="OBS_STATUS" value="8" level="" fixed="false"/> <Concept name="OBS_PRE_BREAK" value="9" level="" fixed="false"/> </Mapping>

Another sample of an XML mapping file, applicable for an FLR input file is shown below: <?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="FREQ" value="1-1" level="" fixed="false"/> <Concept name="JD_TYPE" value="2-2" level="" fixed="false"/> <Concept name="JD_CATEGORY" value="3-3" level="" fixed="false"/> <Concept name="VIS_CTY" value="4-5" level="" fixed="false"/> <Concept name="TIME" value="6-12" level="" fixed="false"/> <Concept name="OBS_VALUE" value="13-14+15-17" level="" fixed="false"/> <Concept name="OBS_CONF" value="18-18" level="" fixed="false"/> <Concept name="OBS_STATUS" value="19-19" level="" fixed="false"/> <Concept name="OBS_PRE_BREAK" value="20-20" level="" fixed="false"/> </Mapping>

The above assignment, declares that:

• FREQ correlates with the value of column 1 • JD_TYPE correlates with the value of column 2 • JD_CATEGORY correlates with the value of column 3 • VIS_CTY correlates with the value of column 4, concatenated with the value

of column 5 • TIME correlates with the concatenated values of columns 6,7,8,9,10,11,12 • OBS_VALUE correlates with the value of column 13, concatenated with the

value of column 14, concatenated with the values of columns 15,16,17 • OBS_CONF correlates with the value of column 18 • OBS_STATUS correlates with the value of column 19 • OBS_PRE_BREAK correlates with the value of column 20

A user can also define a mapping where components are not mapped to any column. This can be indicated


Page 49

• By providing a mapping file like the following (assuming that the obs_value does not correlate with a column):

<?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="COM_DATA_ID" value="1" level="" fixed="false"/> <Concept name="OBS_VALUE" value="" level="" fixed="false"/> <Concept name="COM_DATA_SET_ID" value="2" level="" fixed="false"/> <Concept name="TR_LEU_ID" value="3" level="" fixed="false"/> <Concept name="EU_LEU_ID" value="4" level="" fixed="false"/> <Concept name="BVD_ID" value="5" level="" fixed="false"/> <Concept name="CITY_NAME" value="6" level="" fixed="false"/> <Concept name="CONS_PERSON_EMPLOYED" value="7" level="" fixed="false"/> <Concept name="COUNTRY_ACCESS_CODE" value="8" level="" fixed="false"/> <Concept name="COUNTRY_CODE_BOP" value="9" level="" fixed="false"/> ... </Mapping>

• By not assigning any column to a component in the mapping dialog (please see the picture below)

Figure 30 - Non-mapped column

For components that are dimensions a user must assign a column for them. If the user does not, the following message appears:

Figure 31 - Error message

The user can also set a fixed value for a component. When trying to convert a message of CSV or FLR format to an SDMX_ML some columns may not be present because the data is implied. For example the column for the ‘REF_COUNTRY’ concept may be omitted .In that case user can set a fixed value in the mapping dialog for the absent column (see Error! Reference source not found.):


Page 50

Figure 32 - Setting fixed value


Page 51

A sample of the above-mentioned XML format for mapping is shown below: <?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="FREQ" value="ANNUAL" level="" fixed="true"/> <Concept name="JD_TYPE" value="1-1" level="" fixed="false"/> <Concept name="JD_CATEGORY" value="2-2" level="" fixed="false"/> <Concept name="VIS_CTY" value="3-4" level="" fixed="false"/> <Concept name="TIME" value="5-11" level="" fixed="false"/> <Concept name="OBS_VALUE" value="12-16" level="" fixed="false"/> <Concept name="OBS_CONF" value="17-17" level="" fixed="false"/> <Concept name="OBS_STATUS" value="18-18" level="" fixed="false"/> <Concept name="OBS_PRE_BREAK" value="19-19" level="" fixed="false"/> </Mapping>

A user can also specify a fixed value in case he/she wants to replace an existing one and cannot use the transcoding procedure. For example a user may want to replace an existing value for an observation attribute which might not have a codelist thus cannot be transcoded.


Page 52

For multilevel input or output CSV/FLR files a user should define the following:

• The number of levels of the input or output CSV/FLR file in the “Levels of CSV /FLR file” text field. In the following picture the number is set to 3 because the number of the levels is 3 (please see also Error! Reference source not found.)

Figure 33 - Number of levels

• In the mapping in which level each component appears.


Page 53

Figure 34 - Mapping for a multilevel CSV file

The respective mapping file is the following:

<?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="TAB_NUM" value="1" level="1" fixed="false"/> <Concept name="REV_NUM" value="2" level="1" fixed="false"/> <Concept name="FREQ" value="3" level="1" fixed="false"/> <Concept name="COUNTRY" value="4" level="2" fixed="false"/> <Concept name="SEX" value="5" level="2" fixed="false"/> <Concept name="DEMO" value="6" level="2" fixed="false"/> <Concept name="UNIT_MULT" value="7" level="2" fixed="false"/> <Concept name="DECI" value="8" level="2" fixed="false"/> <Concept name="TIME_FORMAT" value="9" level="2" fixed="false"/> <Concept name="UNIT" value="10" level="2" fixed="false"/> <Concept name="TIME" value="11" level="3" fixed="false"/> <Concept name="OBS_VALUE" value="12" level="3" fixed="false"/> <Concept name="OBS_STATUS" value="13" level="3" fixed="false"/> </Mapping>

In APPENDIX B a sample multilevel CSV file for which the above mapping is applicable is presented.

2.11.4 Mapping for DSPL format

Click on the [CHANGE MAPPING] button and the following screen presented in Error! Reference source not found. will appear:


Page 54

Figure 35 - Mapping GUI for DSPL

This screen presents all the dimensions and the primary measure of the selected DSD. The DSPL concepts are retrieved from the DSPL repository (http://dspl.googlecode.com/hg/datasets/google/canonical). This repository includes already defined datasets by Google. A user might select to map the ‘REF_AREA’ component to the ‘country’ DSPL concept and the ‘TIME PERIOD’ component to the ‘quarter’ DSPL concept.

If the DSPL repository is not available then when trying to access mapping functionality a message will appear preventing the user from using the mapping functionality.


Page 55

Figure 36 - DSPL concepts in mapping dialog

In case the user has selected to map cross sectional measures instead of measure dimension then in the mapping GUI dialog the primary measure as well as the measure dimension will not appear. Only the cross sectional measures will be available to be mapped.

A sample of the above-mentioned XML format for mapping is shown below: <?xml version="1.0" encoding="UTF-8"?> <Mapping> <Concept name="FREQ" targetconcept="<none>" level="" fixed="false" /> <Concept name="REF_AREA" targetconcept="country" level="" fixed="false" /> <Concept name="ADJUSTMENT" targetconcept="<none>" level="" fixed="false" /> <Concept name="STS_INDICATOR" targetconcept="<none>" level="" fixed="false" /> <Concept name="STS_ACTIVITY" targetconcept="<none>" level="" fixed="false" /> <Concept name="STS_INSTITUTION" targetconcept="<none>" level="" fixed="false" /> <Concept name="STS_BASE_YEAR" targetconcept="<none>" level="" fixed="false" /> <Concept name="TIME_PERIOD" targetconcept="quarter" level="" fixed="false" /> <Concept name="OBS_VALUE" targetconcept="<none>" level="" fixed="false" /> </Mapping>

2.11.5 Transcoding

Transcoding applies for conversions with input and output format CSV or FLR as well as for DSPL output format. Transcoding is applied when some coded columns in the input file have different coding than the given DSD (in case of conversion where the output file is SDMX or GESMES) or when the user wants in the output file a


Page 56

different coding from the one that is used in the input file (in case of conversion where the output file is FLR or CSV or DSPL).

2.11.6 Transcoding for CSV and FLR formats

The user presses the [Transcoding] button in the Main Gui Window and the following screen presented in Error! Reference source not found. will appear:

Figure 37 - Transcoding GUI

This screen presents all the components that are coded according to the DSD File. Next to each name of the coded components there is a button that the user can press in order to add transcoding rules for the specific component. Finally, the checkboxes at the right side of the screen are clicked when there exist transcoding rules for a specific component and are not clicked otherwise. The first time the user presses the [Transcoding] button the screen will look like the one in Error! Reference source not found.

The user can add transcoding rules manually or by loading an xml file with transcoding rules. For adding the rules manually the button next to the component that the user wants to add the transcoding rules must be pressed. The following screen presented Error! Reference source not found. appears when the user wants to add transcoding rules for the concept COUNTRY.


Page 57

Figure 38 - Adding transcoding rules

The user can add a transcoding rule by pressing the [Add] button. For removing a transcoding rule the user can select a row in the screen and then press the [Remove] button. In order to confirm the transcoding rules added (or removed) for this dimension press the [OK] button otherwise close the window or press the [Cancel] button.

For adding a transcoding rule write in the ‘From’ field the code in the input file and in the ‘To’ field the corresponding code used in the DSD. In the ‘To’ field the user can select the code by a drop-down-list that shows all the codes (with their descriptions) that the coded component can have.

Figure 39 - A transcoding rule

In the above figure the coding value ALB for the concept COUNTRY will be transcoded to the value AL. When the user has added (or removed) the transcoding rules and wants to confirm them for the specific dimension, then the [OK] button must be pressed. In the same way the user can add transcoding rules for every dimension appearing in the screen presented in Error! Reference source not found.. To confirm ALL the transcoding rules added, then the [OK] button in this


Page 58

screen (Error! Reference source not found.) must be pressed. The next time the user presses the [Transcoding] button in the Main GUI Window the dimensions with the transcoded rules will have a clicked checkbox next to them (Error! Reference source not found..

Figure 40 - Dimensions with transcoding

This screen shows that the user has added transcoding rules for the dimensions FREQ, COUNTRY and DECI.

To load from an xml file the transcoding rules then the [Load from file] button (Error! Reference source not found.) can be pressed. If there are already specified transcoding rules a warning message will be shown that the user will lose them

Figure 41 - Warning message for losing transcoding rules

The user can press the [Save to file] button in order to save to an xml file the transcoding rules added.

If the transcoding file to be saved has the same name with the previously saved transcoding file, then a warning message appears informing the user that he is going to overwrite the transcoding file.


Page 59

Figure 42 - Warning message for overwriting transcoding file

Press the [OK] button after loading rules from an xml file. For discarding them press [Cancel] or close the window.

A sample of the above-mentioned xml format for transcoding is shown below. The following file contains transcoding rules for the dimensions FREQ, COUNTRY and DECI (Error! Reference source not found.) <?xml version="1.0" encoding="UTF-8"?> <Structure xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message" xmlns:structure="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/structure" xmlns:common="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/structure SDMXMessage.xsd"> <Header> <ID>REGISTRY_RESPONSE</ID> <Test>false</Test> <Prepared>2011-04-06T15:03:32+03:00</Prepared> <Sender id="ESTAT"/> </Header> <StructureSets> <structure:StructureSet id="Id_for_StructureSet" version="1.0" isFinal="false"> <structure:Name xml:lang="en">Name of StructureSet</structure:Name> <structure:CodelistMap id="FREQ"> <structure:Name xml:lang="en">Name of CodelistMap</structure:Name> <structure:CodelistRef/> <structure:TargetCodelistRef/> <structure:CodeMap> <structure:MapCodeRef>ANN</structure:MapCodeRef> <structure:MapTargetCodeRef>A</structure:MapTargetCodeRef> </structure:CodeMap> </structure:CodelistMap> <structure:CodelistMap id="COUNTRY"> <structure:Name xml:lang="en">Name of CodelistMap</structure:Name> <structure:CodelistRef/> <structure:TargetCodelistRef/> <structure:CodeMap> <structure:MapCodeRef>ALB</structure:MapCodeRef> <structure:MapTargetCodeRef>AL</structure:MapTargetCodeRef> </structure:CodeMap> </structure:CodelistMap> <structure:CodelistMap id="DECI"> <structure:Name xml:lang="en">Name of CodelistMap</structure:Name> <structure:CodelistRef/> <structure:TargetCodelistRef/> <structure:CodeMap> <structure:MapCodeRef>ONE</structure:MapCodeRef> <structure:MapTargetCodeRef>1</structure:MapTargetCodeRef> </structure:CodeMap> </structure:CodelistMap> </structure:StructureSet> </StructureSets> </Structure>

For the dimension FREQ there exists a transcoding rule that will transcode the code ‘ANN’ to the code ‘A’. Also there exists a transcoding rule for the dimension COUNTRY that will transcode the value ALB to the value AL. Finally there exists a


Page 60

transcoding rule for the dimension DECI that will transcode the code ‘ONE’ to the code ‘1’.

2.11.7 Transcoding for DSPL format

The user presses the [Transcoding] button in the Main Gui Window and the following screen presented in Error! Reference source not found. will appear:

Figure 43 - Transcoding GUI

This screen presents all the components that are coded according to the DSD File. Next to each name of the coded components there is a button that the user can press in order to add transcoding rules for the specific component. Finally, the checkboxes at the right side of the screen are clicked when there exist transcoding rules for a specific component and are not clicked otherwise.

The user can add transcoding rules manually or by loading an xml file with transcoding rules. For adding the rules manually the button next to the component that the user wants to add the transcoding rules must be pressed. The following screen appears when the user wants to add transcoding rules for the concept REF_AREA. If this component has not been mapped to a DSPL component, then the transcoding rule dialog is as in Error! Reference source not found.. Otherwise the transcoding rule dialog alters and in the column “From (Native Format)” all the codes of the DSPL concept that are presented. These codes are defined in a csv file that resides in the Google repository.


Page 61

Figure 44 - Transcoding Rule with DSPL codes

A sample xml file for the afore-mentioned transcoding is shown below. The following file contains transcoding rules for the dimensions REF_AREA. <?xml version="1.0" encoding="UTF-8"?>

<Structure xmlns="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message" xmlns:structure="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/structure" xmlns:common="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.SDMX.org/resources/SDMXML/schemas/v2_0/structure SDMXMessage.xsd">

<Header> <ID>REGISTRY_RESPONSE</ID> <Test>false</Test> <Prepared>2011-03-14T14:55:56+02:00</Prepared> <Sender id="ESTAT" /> </Header> <StructureSets> <structure:StructureSet id="Id_for_StructureSet" version="1.0" isFinal="false"> <structure:Name xml:lang="en">Name of StructureSet</structure:Name> <structure:CodelistMap id="REF_AREA"> <structure:Name xml:lang="en">Name of CodelistMap</structure:Name> <structure:CodelistRef /> <structure:TargetCodelistRef /> <structure:CodeMap> <structure:MapCodeRef>SE</structure:MapCodeRef> <structure:MapTargetCodeRef>SE</structure:MapTargetCodeRef> </structure:CodeMap> </structure:CodelistMap> </structure:StructureSet> </StructureSets> </Structure>

2.11.8 CSV Delimiter Character

In the CSV Delimiter Character field, the user must provide the character used as column separator in the CSV input file. Any combination of characters is allowed for this field. There is a default semicolon (;) value for this field in Converter’s UI.


Page 62

2.11.9 CSV/FLR Date Format

In this field the user specifies the date format to be used in the output CSV or FLR file. Two formats are available, the SDMX format (“-” separated fields, i.e. 2008-09) and the Gesmes format (non separated fields, i.e. 200809).

2.11.10 CSV/FLR Ordered input

In this field the user specifies whether the CSV or FLR input file is ordered.

2.11.11 Header Row

This field implies that a row with data appears in the first line of the CSV/FLR file. If the CSV/FLR is the input file, then using this field the user specifies whether the first line of the CSV/FLR input file is to be ignored. If the CSV/FLR is the output file, then using this field the user specifies whether a row with the name fields of the reporting values should be written.

The header row parameter is applicable only to CSV and FLR input and output formats and only for flat files. In other cases even if it is provided it will be ignored.

The possible values for the header row parameter are:

• USE_COLUMN_HEADERS

• DISREGARD_COLUMN_HEADERS

• NO_COLUMN_HEADERS

If no value is provided then the default considered value is NO_COLUMN_HEADERS.

The same parameter is used for both input and output having different effects depending on the type CSV or FLR. See below each case:

CSV Input

If the value of the header row is USE_COLUMN_HEADERS then it is expected on the first row of the input file to have the column headers named accordingly to the DSD and separated by the provided delimiter in the same order as the input data.

In this case the mapping is defined based on the order of the column headers from the first row. The USE_COLUMN_HEADERS for header row has priority over provided mapping file or over default mapping. Meaning that even if another mapping is given as parameter or filled in GUI the mapping obtained from the column headers of the input file will be used for processing the data.

To see the defined mapping from column headers press change mapping button. If this mapping should be modified this can be done in change mapping then it has to be saved and reloaded when another option than USE_COLUMN_HEADERS is used.

If header row has value DISREGARD_COLUMN_HEADERS then the first row of the input file will be skipped from processing. It is assumed it contains metadata and not further actions besides ignoring the first row are made.


Page 63

If header row has value NO_COLUMN_HEADERS then no action is made and the first row of the input file is expected to contain data that should be processed.

CSV Output

If the value of the header row is USE_COLUMN_HEADERS then the first row of the output file will contain the name of the fields separated by the provided delimiter in the same order as the data.

If the value of the header row is DISREGARD_COLUMN_HEADERS or NO_COLUMN_HEADERS then no action is done and no header row is added to the output file.

FLR Input

Providing mapping for FLR input is always mandatory. Because of the special mapping the header row alone cannot define the mapping hence the USE_COLUMN_HEADERS does not build a mapping for the input file.

If for a FLR input file header row has value USE_COLUMN_HEADERS or DISREGRD_COLUMN_HEADERS then it is assumed that the first row of the input file is not data and it is skipped from processing.

If header row has value NO_COLUMN_HEADERS for a FLR input file then it is assumed that no header row exists and all the rows of the file are processed as data.

FLR Output

If the value of the header row is USE_COLUMN_HEADERS then the first row of the output file will contain the name of the fields in the same order they were defined in the mapping.

If the value of the header row is DISREGARD_COLUMN_HEADERS or NO_COLUMN_HEADERS then no action is done and no header row is added to the output file.


Page 64

Figure 45 - Header Row parameter for CSV/FLR formats

When handling the Header Row the application assumes the input and the parameters are correct.

2.11.12 Write Header

In this field the user specifies if a properties file with header information is to be written. The file is written in the folder where the output converted file will be written and its name is the name of the output file + the word “Header”. For example if the output file name is “CompactOut” the header properties file name will be “CompactOutHeader.prop”. In case the user interacts with the web service of the SDMX Converter, the header properties file is written by default in the folder ‘params’ that can be found in ‘(package)/Converter_Data/params/’ directory structure and its name is ‘Header.prop’.

Unescape CSV Input Fields

This field is available when the input format is CSV. If it is checked Converter will be able to read values enclosed in double quotes or having the delimiter escaped (please see Appendix G – CSV Input/Output Escaping Mechanism). By default it is un-checked.

Escape CSV Output Fields


Page 65

This field is available when the output format is CSV. If it is checked Converter will enclose all outputted values in double quotes (please see Appendix G – CSV Input/Output Escaping Mechanism). By default this field is checked.

2.12 Other Parameters

2.12.1 Gesmes/TS Technique

This field is used to specify the writing technique to be used for Gesmes/TS output files. Two options are available, the Single-Observation technique, and the Time-Range technique3. A Gesmes_TS file written with a time range technique, will contain the '+' sign for every observation that is not reported for a specific period in that time range. Please note that the the monthly period cannot be represented as 'YYYYMM' in an input file, and thus the Gesmes_TS writer will assume that no observation value exists for the specific period and place a '+' in the Gesmes_TS output file. The user should alter the time value in the input file to be represented with 'YYYY-MM' format.

2.12.2 SDMX Validation

By checking the “SDMX Validation” check box, the application will perform a validation on the SDMX Input File or SDMX Output file or both, if both input and output are of type SDMX, in accordance with the selected DSD. If this validation of input file fails, the application aborts the conversion and the user receives an error message. If the validation of the output file fails the user will receive a message but the converted file can be checked.

The SDMX Validation is applicable ONLY for SDMX Generic, SDMX Compact, SDMX Utility and SMDX Cross-Sectional formats.

Important note: When “SDMX Validation” is checked, Converter verifies the XML syntax of both input and output messages and then, checks the code validation against the DSD to examine if all the code values belong to the specified code lists.

Even if “SDMX Validation” is not checked, some validations are still performed and, in case any of them fail, the conversion is stopped with a message for the user. The list of these validations is:

• Converter checks if the Data Structure Definition is compliant with the SDMX standard. This means that the DSD should be built according to the SDMX XSD schemas. If this check does not pass, the conversion will be stopped.

• Another check is to verify if there are any dimensions requested in the DSD but missing from the message

Example:

In the DSD there is PENSION_FUNDTYPE dimension definition: <mes:KeyFamilies> <str:KeyFamily agencyID="ESTAT" id="NA_PENS" ..."> <str:Name xml:lang="en">NA Pensions</str:Name>

3 http://www.sdmx.org/docs/1_0/SDMX%201_0%20SECTION_04_SDMX-EDI.pdf


Page 66

<str:Components> ............ <str:Dimension codelist="CL_PENS_FUNDTYPE" conceptRef="PENSION_FUNDTYPE" ... > <str:TextFormat textType="String" maxLength="6".../> </str:Dimension> ............. </mes:KeyFamilies>

In the input file the Series does not have values for the above dimension: <na_:DataSet COMMENT_DSET="Excel Template V1.2b"> <na_:Series FREQ="A" REF_AREA="LU" COUNTERPART_AREA="W2" REF_SECTOR="S13" COUNTERPART_SECTOR="S14" STO="K5" INSTR_ASSET="_Z" ACCOUNTING_ENTRY="_Z" UNIT_MEASURE="XDC" REF_PERIOD_DETAIL="C" TIME_FORMAT="P1Y" DECIMALS="0" TABLE_IDENTIFIER="T2900" UNIT_MULT="6" COMMENT_TS="xxx">

<na_:Obs TIME_PERIOD="2010" OBS_VALUE="896" OBS_STATUS="A" CONF_STATUS="F"/>

... </na_:DataSet>

In this case the Converter will stop the conversion with the following message regardless of checking or not the SDMX Validation checkbox:


Page 67

2.13 Namespace

In conversions involving Compact, Utility and Cross Sectional SDMX formats as input and/or output files, a namespace and a prefix are mandatory. For this, the section “Namespace” of the Converter’s user interface provides the respective fields.

The Namespace field must have the complete URN path required for the format under conversion as described in the SDMX ML 2.0 standard documentation for Namespace Modules.

The prefix field holds the name of the above namespace declaration for the format under conversion as described in the SDMX ML 2.0 standard documentation for Namespace Modules.

For example for an SDMX Compact message, the URN and the prefix values are marked with red and yellow colour respectively, in the following message sample:

<CompactData xmlns=”http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message” xmlns:bisc=”urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=BIS:EXT_DEBT:compact” xmlns:compact=”http://www.SDMX.org/resources/SDMXML/schemas/v2_0/compact” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://www.SDMX.org/resources/SDMXML/schemas/v2_0/message SDMXMessage.xsd urn:sdmx:org.sdmx.infomodel.keyfamily.KeyFamily=BIS:EXT_DEBT:compact BIS_JOINT_DEBT_Compact.xsd http://www.SDMX.org/resources/SDMXML/schemas/v2_0/compact SDMXCompactData.xsd”>

This also applies for an SDMX Utility message and an SDMX Cross Sectional message. For CSV, GESMES and SDMX Generic formats, the namespace and prefix fields are disabled and ignored by the conversion process. Important note: It is mandatory for the namespace definition, following the ‘=’sign to adopt the naming ‘KeyFamilyAgency:KeyFamilyID’ (e.g. ‘ESTAT:STS’). Converter adds the last part of the namespace automatically (i.e. ‘compact’ ‘utility’ and ‘cross’ depending on the output type). If the “Default Namespace” is checked the Namespace text box will be disabled. The complete list of output formats where Converter allows changing the namespace is:

• COMPACT_SDMX • UTILITY_SDMX • CROSS_SDMX • DSPL • STRUCTURE_SPECIFIC_DATA_2_1 • STRUCTURE_SPECIFIC_TS_DATA_2_1

2.14 Specifying the encoding

The ‘encoding.txt’ file is a configuration file, which is included in the SDMX Converter package. The directory structure where the ‘encoding.txt’ file can be found is ‘(package)/Converter_Data/params/’. Specifically for each package please consult the Installation guide Section 4.


Page 68

This file is used in order for right conversions to be performed from a CSV (or FLR) file or conversions to a CSV (or FLR) file. The content of encoding.txt file is the following:

# Use one of the following encodings: # UTF-8 # ISO-8859-1 encoding=UTF-8 # For more encodings here is the complete list: # http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html

In a conversion when reading from a flat file or writing to a flat file the encoding.txt is read. By default is assumed that the encoding of the file is UTF-8. As it cannot be predefined which encoding is used in the file the user can change the property ‘encoding’ in the encoding.txt file and set the proper encoding. In order to configure this file with the appropriate encoding the user

• can access the url given in the ‘encoding.txt file’ (http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html)

• select the right encoding to be used

• in the ‘encoding.txt’ file set to the value ‘encoding’ the encoding selected in the previous step

For example to convert a CSV file which uses the ‘Cp1252’ (Windows Latin-1) encoding then the user has to open the encoding.txt file and set the encoding to Cp1252. In that way the encoding.txt file will have the following content:

# Use one of the following encodings: # UTF-8 # ISO-8859-1 encoding= Cp1252 # For more encodings here is the complete list: # http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html

2.15 Performing a conversion

Having provided all the necessary parameters, click on the [CONVERT] button in order to make a conversion. If one or more of the mandatory fields is missing, an information message will appear denoting the error. In the log file (‘(package)/Converter_Data/logs/’) a more specific error message, in this case the fields that are missing, are written.

Figure 46 - Missing field message


Page 69

In the previous example the sender parameter field was empty. Provide a value for the field that failed and click on the [CONVERT] button again. When a conversion completes successfully the following message appears:

Figure 47 - Conversion completed

Use the [Load Template] button to load an xml file that stores a conversion template. When the respective file is selected the converter GUI dialog’s parameters are populated.

For the Template.xml file, which is presented in the Error! Reference source not found., the parameters of the Converter that are populated in the SDMX Converter GUI dialog are presented in the following picture.

Figure 48 - Load a Template


Page 70

The “Default Mapping” checkbox is not selected and the [Change Mapping] button is enabled because there exists a mapping in the Template.xml file (please see Error! Reference source not found.).

If the user presses the [Change Mapping] button the following mapping dialog appears:

Figure 49 - Mapping Loaded

Click the [CONVERT] button in order to make the conversion.

When loading a template file the converter validates it against the “Template.xsd” file, which is presented in the Error! Reference source not found.. If the loaded xml file is not valid then an errror message is presented to the user.

For example if the element ‘Mapping’ is missing from the loaded xml file then the following error appears:

Figure 50 - Invalid template file has been loaded

When all the necessary parameters in the SDMX Converter GUI dialog have been provided, use the [Save Template] button to save for later use in a template file the conversion that is to be performed.

If the template file to be saved has the same name with the previously saved template file, then a warning message appears informing the user that he is going to overwrite the template file.


Page 71

Figure 51 - Warning message for overwriting a template file

In order to make the conversion, one should click on the CONVERT button.


Page 72

3 Interacting with Converter programmatically This section presents the SDMX converter API and an example on how a developer should interact with it programmatically by using the Java programming language.

The following steps concern building a new project in Eclipse platform that will use the SDMX Converter API.

1. Create a new project named ‘Converter’.

2. Open the properties of the project and choose ‘Properties’. The ‘Properties’ window opens:

Figure 52 – Java project properties in Eclipse

Click on ‘Java Build Path’ on the left pane of the properties window.

3. Click on ‘Libraries’ tab on the right pane.

4. Click ‘Add External Jars’ on the right pane. A file system selection window opens.

5. Add the following jars: commons-beanutils-1.7.0.jar, commons-collections-3.2.jar, commons-lang-2.1.jar, commons-logging-1.1.jar, commons-digester-1.7.jar, validator-1.1.2.jar, service-1.8.6.jar, core-1.7.4, jdom-1.0.jar, joda-time-1.6.2, sdmxsource-0.9.25b.jar, sdmx_converter-4.0.0.jar,. The above jars can be found in the package of the SDXM Converter Platform Independent and in the folder Libraries (‘package/Libraries’).

6. Copy from the above directory the folders params(‘package/params') and logs(‘package/logs') to the workspace where the project has been saved. To view the location of the workspace open the properties of the project and click


Page 73

on ‘Resource’ on the left pane of the properties window. In the red rounded rectangle the location of the project is depicted.

Figure 53 - Project location

7. Create a new Class named ‘ConverterTester’

In this class a new instance of the Converter class has to be created as in the following statement:

Converter converter = new Converter();

Having the Converter instance, we invoke its convert() method:

converter.convert(inputFormat, outputFormat, inputStream, outputStream, params);

The arguments of the convert() method are:

• Formats inputFormat the input format

• Formats outputFormat the output format

• InputStream inputStream stream for reading the dataset to be converted

• OutputStream outputStream stream for writing the conversion output

• InputParams params auxiliary params needed for the conversion


Page 74

For the first two arguments, use the Formats enumeration Class.

For instance, when submitting a conversion from CSV to SDMX Compact format the values are Formats.CSV and Formats.COMPACT_SDMX respectively.

The inputStream and outputStream objects can be created by giving the paths of the input and output files respectively. For instance:

FileInputStream inputStream = new FileInputStream(new File(“D:/ selectedInputFile”));

and FileOutputStream outputStream = new FileOutputStream(new File"(“D:/selectedOutputFile”));

Finally create an InputParams object with the following statement: InputParams params = new InputParams();

Having the InputParams object, give it state according to the mandatory and optional parameters that Chapter 2 describes.

Specifically, the parameters should be set:

keyFamilyBean (KeyFamilyBean)

i.e. params.setKeyFamilyBean(keyFamilyBean);

headerBean (HeaderBean)

i.e. params.setHeaderBean(headerBean);

mappingMap (Map)

i.e. params.setMappingMap (map);

The above-mentioned Map should have the following structure:

Concept (key) CSV Column Number (value)

Concept1 1

Concept2 4

Concept3 2+3 (case of concatenating columns)

Or if a fixed value is to be set the following:

Concept (key) CSV Column Number (value)

Is Fixed Value

Concept1 GREECE True

Concept2 1 False

Concept3 2+3 (case of concatenating columns)

False


Page 75

The following statement could be used, in order to receive the necessary Map from an existing mapping file. Instead of creating it explicitly: FileInputStream mappingIs = new FileInputStream(mapping_file);

params.setMappingMap(IoUtils.getMappingMap(mappingIs));

StructureSet(Map)

i.e. params.SetStructureSet(map);

The above-mentioned Map should have the following structure:

Concept (key) Map of Transcoding Rules for this concept (value)

Concept1 Map of rules for concept 1

Concept2 Map of rules for concept 2

The Map of Transcoding Rules for a concept should have the following structure:

Coded ‘From’ Value (key) Coded ‘To’ Value (value)

Code1 in the input file Code1 in the output file taken from transcoding

Code2 in the input file Code2 in the output file taken from transcoding

delimiter (String)

i.e. params.setDelimeter (“;”);

namespaceUri (String)

i.e. params.setNamespaceUri (“a URI”);

namespacePrefix (String)

i.e. params.setNamespacePrefix (“a prefix”);

dateCompatibility (String)

i.e. params.setDateCompatibility(“SDMX”);

params.setDateCompatibility(“GESMES”);

generatedFileComment (String)

i.e. params.setGeneratedFileComment(“This is a comment”);

orderedFlatInput (boolean)

i.e. params.setOrderedFlatInput(true);

gesmesWritingTechnique (String)

i.e. params.setGesmesWritingTechnique(“Time Range”);

params.setGesmesWritingTechnique(“Single Observation”);

encoding(String)

i.e. params.setFlatFileEncoding(“UTF-8”);

headerRow(String)


Page 76

i.e params.setHeaderRow(Defs.USE_COLUMN_HEADERS);

codelist(ArrayList<CodeListBean>)

i.e params.setCodeList(codelist);

levelNumber(String)

i.e. params. SetLevelNumber(“3”);

Below the complete source code of the ‘ConverterTester’ class for conversion from CSV to SDMX Compact format with header, “;” for CSV delimiter and DSD file.

package main.Test_Converter; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.InputStream; import com.agilis.sdmx.converter.Converter; import com.agilis.sdmx.io.data.Formats; import com.agilis.sdmx.io.data.InputParams; import com.agilis.sdmx.io.data.IoUtils; import com.agilis.sdmx.io.metadata.StructureReader; import com.agilis.sdmx.model.base.HeaderBean; import com.agilis.sdmx.model.structure.KeyFamilyBean; import com.agilis.sdmx.model.structure.StructureBean; public class ConverterTester { public static void main(String[] args) throws Exception { try { String baseDir, headerFilename, structureFileName, dataFilename, outFilename, mappingfilename; StructureReader strreader; StructureBean structure; KeyFamilyBean kf; InputParams params; Converter conv = new Converter(); params = new InputParams(); // formats Formats inputFormat = Formats.CSV; Formats outputFormat = Formats.COMPACT_SDMX; // Input-Output file FileInputStream inputStream; inputStream = new FileInputStream(new File( "D:/Documents and Settings/fch/Desktop/Mantis/5031/MILK_TABLEA_M_template_sdmx_data no obs empty.csv")); FileOutputStream outputStream; outputStream = new FileOutputStream(new File("D:/Documents and Settings/fch/Desktop/Mantis/5031/output.xml")); // header InputStream headerIs = new FileInputStream("D:/Documents and Settings/fch/Desktop/Mantis/5031/1.header.prop"); HeaderBean header = IoUtils.parseHeaderPropertyFile(headerIs); params.setHeaderBean(header);


Page 77

// delimiter params.setDelimiter(";"); // encoding params.setFlatFileEncoding("UTF-8"); // DSD structureFileName = new String("D:/Documents and Settings/fch/Desktop/Mantis/5031/MILK_TABLEA_M+ESTAT+2.2.xml"); strreader = new StructureReader(); structure = strreader.doparse(new FileInputStream(structureFileName)); kf = (KeyFamilyBean) structure.getKeyfamilies().get(0); params.setKeyFamilyBean(kf); // make the conversion conv.convert(inputFormat, outputFormat, inputStream, outputStream, params); } catch (Exception ex) { ex.printStackTrace(); } } }

Another sample of a complete source code of the ‘ConverterTester’ class for a conversion from an SDMX Compact message to FLR is the following:

package main.Test_Converter; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.InputStream; import java.util.Map; import com.agilis.sdmx.converter.Converter; import com.agilis.sdmx.converter.ui.exceptions.BaseException; import com.agilis.sdmx.io.data.Formats; import com.agilis.sdmx.io.data.InputParams; import com.agilis.sdmx.io.data.IoUtils; import com.agilis.sdmx.io.metadata.StructureReader; import com.agilis.sdmx.model.structure.KeyFamilyBean; import com.agilis.sdmx.model.structure.StructureBean; public class ConverterTester { public static void main(String[] args) throws Exception { try { String baseDir, headerFilename, structureFileName, dataFilename, outFilename, mappingfilename; StructureReader strreader; StructureBean structure; KeyFamilyBean kf; InputParams params; Converter conv = new Converter(); params = new InputParams(); //DSD structureFileName = new String("D:/ MILK_TABLEA_M+ESTAT+2.2.xml"); strreader = new StructureReader();


Page 78

structure = strreader.doparse(new FileInputStream(structureFileName)); kf = (KeyFamilyBean) structure.getKeyfamilies().get(0); params.setKeyFamilyBean(kf); Formats inputFormat = Formats.COMPACT_SDMX; Formats outputFormat = Formats.FLR; FileInputStream inputStream = new FileInputStream(new File("D:/14.2_IN_Compact_Bis1.0.xml")); FileOutputStream outputStream = new FileOutputStream(new File("D:/selectedOutputFile.xml")); structureFileName = "D:/BIS_JOINT_DEBT_v1.0.xml"; strreader = new StructureReader(); structure = strreader.doparse(new FileInputStream(structureFileName)); kf = (KeyFamilyBean) structure.getKeyfamilies().get(0); // mapping mappingfilename = "D:/" + "14.2_mapping.xml"; InputStream mappingIs = (InputStream) new FileInputStream(mappingfilename); Map<String, String[]> mappingMap = IoUtils.getMappingMap(mappingIs); params = new InputParams(); params.setKeyFamilyBean(kf); params.setFlatFileEncoding("UTF-8"); params.setMappingMap(mappingMap); //delimiter params.setDelimiter(";"); //levels params.setLevelNumber("1"); conv.convert(inputFormat, outputFormat, inputStream, outputStream, params); } catch (Exception ex) { ex.printStackTrace(); } } }

Only the parameters that are given in the main method change in the above examples.

The following sample depicts a complete source code of the ‘ConverterTester’ class for a conversion from an SDMX Compact message to DSPL:

package Converter; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.Map; import java.util.Properties; import java.util.logging.Level; import java.util.logging.Logger; import com.agilis.sdmx.converter.Converter;


Page 79

import com.agilis.sdmx.converter.Resources; import com.agilis.sdmx.converter.ui.exceptions.ApplicationException; import com.agilis.sdmx.io.data.Formats; import com.agilis.sdmx.io.data.InputParams; import com.agilis.sdmx.io.data.IoUtils; import com.agilis.sdmx.io.metadata.StructureReader; import com.agilis.sdmx.model.structure.CodeListBean; import com.agilis.sdmx.model.structure.ConceptSchemeBean; import com.agilis.sdmx.model.structure.KeyFamilyBean; import com.agilis.sdmx.model.structure.StructureBean; public class ConverterTester { public static void main(String[] args) { try { String structureFileName, mappingfilename; StructureReader strreader; StructureBean structure; KeyFamilyBean kf; InputParams params; Converter conv = new Converter(); //initialize the configuration properties Resources.configProperties = new Properties(); try { Resources.configProperties.load(new FileInputStream(Resources.CONFIG_FILE_PATH)); } catch (IOException e) { Resources.logger.log(Level.SEVERE, "Could not read/find config file: " + Resources.CONFIG_FILE_PATH, e); throw (new ApplicationException(101, e.getMessage())); } // initialize the logger Resources.logger = Logger.getLogger("com.agilis.sdmx.io"); Resources.logger.setLevel(Level.ALL); // formats Formats inputFormat = Formats.COMPACT_SDMX; Formats outputFormat = Formats.DSPL;

// Input-Output file FileInputStream inputStream; inputStream = new FileInputStream(new File( "/Users/fch/Desktop/39.1_IN_Compact_Bjt1.0.xml")); FileOutputStream outputStream; outputStream = new FileOutputStream(new File("/Users/fch/Desktop/Output_DSPL.zip")); // initialize the Input Params params = new InputParams(); // DSD structureFileName = new String("/Users/fch/Desktop/BIS_JOINT_DEBT_v1.0.xml"); strreader = new StructureReader(); structure = strreader.doparse(new FileInputStream(structureFileName)); kf = (KeyFamilyBean) structure.getKeyfamilies().get(0); params.setKeyFamilyBean(kf);


Page 80

//encoding params.setFlatFileEncoding("UTF-8"); //mapping mappingfilename = "/Users/fch/Desktop/39.1_mapping.xml"; InputStream mappingIs = (InputStream) new FileInputStream(mappingfilename); Map<String, String[]> mappingMap = IoUtils.getMappingMap(mappingIs); params.setMappingMap(mappingMap); //codelist ArrayList<CodeListBean> codelist; codelist = (ArrayList<CodeListBean>) structure.getCodelists(); params.setCodeList(codelist); //concepts ArrayList<ConceptSchemeBean> concepts; concepts = (ArrayList<ConceptSchemeBean>) structure.getConcepts(); params.setConceptScheme(concepts); // make the conversion conv.convert(inputFormat, outputFormat, inputStream, outputStream, params); } catch (Exception ex) { ex.printStackTrace(); } } }

The following sample depicts a complete source code of the ‘ConverterTester’ class for a conversion from DSPL message to an SDMX-ML:

package Converter; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.util.Properties; import java.util.logging.Level; import java.util.logging.Logger; import com.agilis.sdmx.converter.Converter; import com.agilis.sdmx.converter.Resources; import com.agilis.sdmx.converter.ui.exceptions.ApplicationException; import com.agilis.sdmx.io.data.Formats; import com.agilis.sdmx.io.data.InputParams; public class ConverterTester { public static void main(String[] args) { try { InputParams params; Converter conv = new Converter(); //initialize the configuration properties


Page 81

Resources.configProperties = new Properties(); try { Resources.configProperties.load(new FileInputStream(Resources.CONFIG_FILE_PATH)); } catch (IOException e) { Resources.logger.log(Level.SEVERE, "Could not read/find config file: " + Resources.CONFIG_FILE_PATH, e); throw (new ApplicationException(101, e.getMessage())); } // initialize the logger Resources.logger = Logger.getLogger("com.agilis.sdmx.io"); Resources.logger.setLevel(Level.ALL); // formats Formats inputFormat = Formats.DSPL; Formats outputFormat = Formats.COMPACT_SDMX; // Input-Output file FileInputStream inputStream; inputStream = new FileInputStream(new File( "/Users/fch/Desktop/40.1_IN_DSPL_Tutorial.zip")); FileOutputStream outputStream; outputStream = new FileOutputStream(new File("/Users/fch/Desktop/40.1_OUT_COMPACT.zip")); // initialize the Input Params params = new InputParams(); //encoding params.setFlatFileEncoding("UTF-8"); // make the conversion conv.convert(inputFormat, outputFormat, inputStream, outputStream, params); } catch (Exception ex) { ex.printStackTrace(); } }

}

The following example depicts the usage of API for converting EXCEL to SDMX-ML:

package Converter; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.util.Properties; import java.util.logging.Level; import java.util.logging.Logger; import com.agilis.sdmx.converter.Converter; import com.agilis.sdmx.converter.Resources; import com.agilis.sdmx.converter.ui.exceptions.ApplicationException; import com.agilis.sdmx.io.data.Formats;


Page 82

import com.agilis.sdmx.io.data.InputParams; public class ConverterTester { public static void main(String[] args) { try { InputParams params; Converter conv = new Converter(); try { InputStream repositoryIs = new FileInputStream("repository.properties"); properties.load(repositoryIs); } catch (FileNotFoundException e1) { e1.printStackTrace(); throw (new ApplicationException(109, e1.getMessage())); } catch (IOException e) { e.printStackTrace(); throw (new ApplicationException(109, e.getMessage())); } // get language used in the header Resources.repositoryPath = properties.getProperty("path.file"); //initialize the configuration properties Resources.configProperties = new Properties(); try { Resources.configProperties.load(new FileInputStream(Resources.repositoryPath + File.separator + Resources.CONFIG_FILE_PATH)); } catch (IOException e) { Resources.logger.log(Level.SEVERE, "Could not read/find config file: " + Resources.CONFIG_FILE_PATH, e); throw (new ApplicationException(101, e.getMessage())); } // initialize the logger Resources.logger = Logger.getLogger("com.agilis.sdmx.io"); Resources.logger.setLevel(Level.ALL); // formats Formats inputFormat = Formats.EXCEL; Formats outputFormat = Formats.COMPACT_SDMX; // Input-Output file FileInputStream inputStream; inputStream = new FileInputStream(new File( "src/test/resources/Test Procedure 48/TC 48.3/48.3_IN_Excel.xlsx")); InputStream excelInputStream = new FileInputStream(new File( "src/test/resources/Test Procedure 48/TC 48.3/48.3_IN_Excel.xlsx")); FileOutputStream outputStream; outputStream = new FileOutputStream(new File("src/test/resources/Test Procedure 48/TC 48.3/OUT_COMPACT.xml")); InputStream mappingIs = new FileInputStream(new File("src/test/resources/Test Procedure 48/TC 48.3/MappinsXlsParameters.txt")); // initialize the Input Params params = new InputParams(); //encoding params.setFlatFileEncoding("UTF-8"); //keyFamily


Page 83

StructureReader strreader = new StructureReader();; StructureBean structure = new StructureBean(); structure = strreader.doparse(new FileInputStream("src/test/resources/Test Procedure 48/TC 48.3/ESTAT_STS_v2.2.xml")); KeyFamilyBean kf = (KeyFamilyBean) structure.getKeyfamilies().get(0); params.setKeyFamilyBean(kf); ArrayList<ConceptSchemeBean> concepts = (ArrayList<ConceptSchemeBean>) structure.getConcepts(); params.setConceptScheme(concepts); ArrayList<CodeListBean> codelist = (ArrayList<CodeListBean>) structure.getCodelists(); params.setCodeList(codelist); //set header InputStream headerIs = new FileInputStream("src/test/resources/Test Procedure 48/TC 48.3/header.prop"); HeaderBean header = IoUtils.parseHeaderPropertyFile(headerIs); params.setHeaderBean(header); //excel parameters params.setExcelType(".xls"); //set all the Parameter Sheets that are contained in the Excel File params.setAllExcelParameters(ExcelIOUtils.getExcelParameters(excelInputStream, false, params.getExcelType())); params.setExcelParameterMap(ExcelIOUtils.getParametersMap(mappingIs)); // make the conversion conv.convert(inputFormat, outputFormat, inputStream, outputStream, params); } catch (Exception ex) { ex.printStackTrace(); } }

}

The advantage of the converter API is that it provides the ability for the Converter functionality to be integrated directly from other applications, which are developed in Java. The Converter API encapsulates the whole of the conversion functionality. It is the preferred way of integrating the Converter since it is faster, native (directly with Java), and provides better error handling.


Page 84

4 Calling converter application from a command prompt It is possible for the Converter application to be called using a command line interface (CLI). CLI interface has the following characteristics:

• Offers an alternative way for integrating Converter functionality to other applications. This pertains to applications that are not implemented in Java. Therefore they call CLI commands from the system shell.

• CLI can be utilised to perform conversions in a batch-processing mode without user interaction. It can be coupled with shell scripting of the target platform for tailor-made batch processing.

• CLI requires less system resources, such as processor time and memory, to run than GUIs.

• It is not targeted to human user interaction. The GUI is intended for this purpose.

Among the different packages that the SDMX Converter is being released, the SDMX Converter application package for every platform and the SDMX Converter application package for win32 platform using an installer can be called using CLI. For each of the above packages the content is presented in the pictures below:

Figure 54 - SDMX Converter Platform Independent Package

Figure 55 - SDMX Converter Windows Installer

The commands described in this section can appear in the above packages. Navigate to the directories where the packages have been put and give the following statement: Converter [Options] InputFile OutputFile InputFormat OutputFormat


Page 85

• For Windows OS (32-bit edition or 64-bit edition): The converter command is converter.bat

• For Unix OS: The converter command is converter.sh Arguments: Argument Description InputFile The absolute path of the Input file OutputFile The absolute path of the Output file InputFormat The Input format. One of the following values:

CSV, FLR, GESMES_TS, GESMES_2_1, GESMES_DSIS, GENERIX_SDMX, COMPACT_SDMX, UTILITY_SDMX, CROSS_SDMX, DSPL, GENERIC_DATA_2_1, GENERIC_TS_DATA_2_1, STRUCTURE_SPECIFIC_DATA_2_1, STRUCTURE_SPECIFIC_TS_DATA_2_1, EXCEL, MESSAGE_GROUP

OutputFormat The desired output format. One of the following values: CSV, FLR , GESMES_TS, GESMES_2_1, GESMES_DSIS, GENERIX_SDMX, COMPACT_SDMX, UTILITY_SDMX, CROSS_SDMX, DSPL, GENERIC_DATA_2_1, GENERIC_TS_DATA_2_1, STRUCTURE_SPECIFIC_DATA_2_1, STRUCTURE_SPECIFIC_TS_DATA_2_1

Options: Option Description -reg true|false Boolean value, default: false.

Use –reg true if Registry should be used for retrieving the structure files (whose identifiers should be provided using the –dsd_id, -dsd_version and –dsd_agency arguments so if –reg true the dsd_id, dsd_version and dsd_agency become mandatory arguments). If –reg false or –reg argument does not exist at all, the dsd_file argument becomes mandatory.

-dsd_file path The absolute path of the DSD file (mandatory if –reg parameter is missing or set to false)

-dsd_id id The DSD ID to be used in Registry -dsd_agency agency The DSD Agency for Registry -dsd_version version The DSD Version for Registry -df true|false Must be true if dataflow information is to be used to retrieve

DSD information -df_id id The Dataflow ID for Registry -df_version version The Dataflow Version for Registry -df_agency agency The Dataflow Agency for Registry -header_file path The absolute path of the Header file, applicable ONLY for

CSV input -date_format format The date format to be used for output CSV/FLR. The value of

the format can be either SDMX or GESMES. By default the option is SDMX

-level number Must be a number if the file is a multilevel csv file. Applicable only for CSV/FLR input/output.


Page 86

-mapping_file path The absolute path of the Mapping file, applicable ONLY for CSV/FLR input file

-ordered_input true|false

Boolean value. True or false to denote if input is ordered or not. Applicable for CSV/FLR input.

-trans_file path The absolute path of the Transcoding file, applicable ONLY for CSV-FLR input-output file

-delimiter “delim” The CSV delimiter of the input file in double quotes. This parameter is applicable only to the CSV input/output type. Examples: -delimiter “;”(for semicolon) -delimiter “ ” (for blank-space) A special case is the TAB character which should be used like -delimiter “Tab”

-unescapeCSVInput true|false

Unescape CSV Input Option, applicable ONLY for CSV input file. Default value: false

-escapeCSVOutput true|false

Escape CSV Output Option, applicable ONLY for CSV output file. Default value: false

-header-row value The possible values for the header row parameter are: • USE_COLUMN_HEADERS if in the first row of the output CSV/FLR the

field names that correspond to each column should be written or the

input CSV/FLR file has a first row containing the column names and it

should be used,

• DISREGARD_COLUMN_HEADERS if the first row of the input

CSV/FLR file should be ignored,

• NO_COLUMN_HEADERS if there is no extra first row in the input

CSV/FLR file or no first header row should be written in the output

CSV/FLR file – see CSV/FLR parameters

-header_Writing true|false

Must be true if a header.prop file is to be written

-nuri uri The namespace URI, applicable ONLY for SDMX Compact, SDMX Utility and SDMX Cross Sectional output files.

-nprefix prefix The namespace prefix, applicable ONLY for SDMX Compact, SDMX Utility and SDMX Cross Sectional output files.

-ges_technique value The Gesmes/TS writing technique. The value can be either TIME_RANGE or SINGLE_OBS

-validation true|false Applicable for SDMX input and output files. Set true to perform SDMX validation.

-mapping_param_file path

Applicable for EXCEL input files. Path to a file containing mapping between data sheets and parameter sheets - please see APPENDIX D for examples

-parameter_file path Applicable for EXCEL input files. Path to an external parameter file - please see APPENDIX D for examples

The above arguments and options analysis may appear in a command prompt window by typing:

• converter.bat –help for windows • converter.sh –help for UNIX


Page 87

Note: The options may have arbitrary order

Three examples are presented in the following table

Source file

Output file Command Description/Details/Comments

Example 1

4.3_IN_Flr_Bi

OutCompact.x

converter.bat -dsd_file "D:\Test Case 8\BIS_JOINT_DEBT_v1.0.xml" -header_file "D:\Test Case 8\header.prop" -mapping_file "D:\Test Case 8\14.3_mapping.xml" -trans_file "D:\Test Case 8\14.3_Transcoding.xml" "D:\Test Case 8\14.3_IN_Flr_Bis1.0.txt" "D:\Test Case 8\OutCompact.xml" FLR COMPACT_SDMX.

A conversion from FLR to SDMX compact, without usingRegistry, with mappingfile, transcoding file and header file.

Example 2

Q;SE;N;EMQ;SE;N;EMQ;SE;N;EMQ;SE;N;EMQ;SE;N;EMQ;SE;N;EMQ;SE;N;EMQ;SE;N;EM

8_OUT_Generic.x

converter.bat -header_file "D:\Test Case 8\header.prop" -reg true -dsd_id "STS" -dsd_version "2.2" -dsd_agency "ESTAT" -delimiter ; -ordered_input true "D:\Test Case 8\8_IN_Csv_Sts2.1.csv" "D:\Test Case 8\8_OUT_Generic.xml" CSV GENERIC_SDMX

A conversion from CSV to SDMX generic, with using Registry to retrieve the DSD (DSD ID: STS, DSD Version: 2.2, DSD Agency: ESTAT), with header file: header.prop and CSV delimiter: “;”,

Example 3

1;P2;MON;Am3;2000-013;2000-012;ANN;Am3;2000-043;2000-091;A2;MON;Am3;2000-04

30_OUT_Compac

converter.bat -header_file "D:\Test Case 8\header.prop" -dsd_file "D:\Test Case 8\BIS_JOINT_DEBT_v1.0.xml" -mapping_file "D:\Test Case 8\30.1_mapping.xml" -trans_file "D:\Test Case 8\30_Transcoding.xml" -ordered_input true -level 3 -delimiter “;” "D:\Test Case 8\30.1_IN_CSV_Bis1.0.csv" "D:\Test Case 8\30_OUT_Compact.xml" CSV COMPACT_SDMX

A conversion from a multilevel CSV file to an SDMX Compact file with header file, mapping file, transcoding file, delimiter = “;” and number of levels=3.

Example 4

48.1_OUT_Comxml

converter.bat -header_file "C:\devel\Applications\GITRepo\converter.java\src\test\resources\Test Procedure 47\header.prop" -dsd_file "C:\devel\Applications\GITRepo\converter.java\src\test\resources\Test Procedure 47\ECB+ECB_EXR1+1.0.xml" -delimiter

A conversion from an Excel file (.xls) with few data sheets and a single parameter sheet to SDMX_Compact .


Page 88

" " "C:\devel\Applications\GITRepo\converter.java\src\test\resources\Test Procedure 47\SDMX-NA_T1600_T1950_V20.xlsx" "C:\devel\Applications\GITRepo\converter.java\src\test\resources\Test Procedure 47\47_OUT_Compact_CLI.xml" EXCEL COMPACT_SDMX

In the above statements the ‘converter.bat’ calls the Converter application while all the necessary arguments are following.


Page 89

5 Using the Converter Web Service This section presents the SDMX Converter Web Service. Specifically is described how you can build a client to invoke the web service and make conversions. Also instructions for using the test client that is included in the package of the SDMX Converter WS are provided.

5.1 SOAP Web services

The Web Service interface offers the possibility to expose Converter functionalities on over the World Wide Web. The clients can call the application server where the Converter service is deployed. The advantage of calling the converter through the web service is the interoperability obtained. One client can interact with the converter regardless of the client application platform and the programming language used. However, SOAP is slower than interacting with the other interfaces since datasets are sent over the network and the CPU time is shared with other conversion requests. Moreover, the overhead of the SOAP and HTTP protocol is added.

5.2 Web Service Descriptor for Converter (WSDL)

To invoke the converter web service a client should be developed that will reach the WSDL of the service. The WSDL of the converter web service can be viewed at http://url:port/ConverterWebService/services/ConverterWebService?wsdl. For example for the Eurostat environment the URL is 158.167.200.146 and the port is 7002. Therefore the WSDL can be viewed at: http://158.167.200.146:7002/ConverterWebService-x.y.z/services/ConverterWebService?wsdl

NOTE: The packaged war in the release has the version as part of the context root, therefore in the above URL it should be changed to “ConverterWebService-5.0.0” or any other version that you have deployed in your premises. A shortened version of the WSDL of the SDMX Converter Web service would look like below: <?xml version="1.0" encoding="UTF-8"?> <wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:ns1="http://org.apache.axis2/xsd" xmlns:ns="http://webservice.converter.sdmx.agilis.com" xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl" xmlns:http="http://schemas.xmlsoap.org/wsdl/http/" xmlns:ax26=http://errorreport.webservice.converter.sdmx.agilis.com/xsd xmlns:ax24="http://webservice.converter.sdmx.agilis.com/xsd" xmlns:ax21="http://exceptions.ui.converter.sdmx.agilis.com/xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" targetNamespace="http://webservice.converter.sdmx.agilis.com"> <wsdl:documentation>ConverterWebService</wsdl:documentation> <wsdl:types> <xs:schema attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://errorreport.webservice.converter.sdmx.agilis.com/xsd"> … </xs:schema>


Page 90

<xs:schema xmlns:ax27=http://errorreport.webservice.converter.sdmx.agilis.com/xsd xmlns:ax25="http://webservice.converter.sdmx.agilis.com/xsd" xmlns:ax23="http://exceptions.ui.converter.sdmx.agilis.com/xsd" attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://webservice.converter.sdmx.agilis.com"> … </xs:schema> <xs:schema xmlns:ax22="http://webservice.converter.sdmx.agilis.com" attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://exceptions.ui.converter.sdmx.agilis.com/xsd"> … </xs:schema> <xs:schema attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://webservice.converter.sdmx.agilis.com/xsd"> … </xs:schema> </wsdl:types> <wsdl:message name="convertRequest"> <wsdl:message name="convertResponse" <wsdl:message name="BaseException"> <wsdl:message name="Exception"> <wsdl:message name="validateRequest"> <wsdl:message name="validateResponse"> <wsdl:portType name="ConverterWebServicePortType"> <wsdl:binding name="ConverterWebServiceSoap11Binding" type="ns:ConverterWebServicePortType"> <wsdl:binding name="ConverterWebServiceSoap12Binding" type="ns:ConverterWebServicePortType"> <wsdl:binding name="ConverterWebServiceHttpBinding" type="ns:ConverterWebServicePortType"> <wsdl:service name="ConverterWebService"> </wsdl:definitions>

For a full WSDL please consult APPENDIX H – Converter Web Service WSDL

5.1 Using the “convert” method of the web service

The parameters that could be given for the invocation of a conversion are the following: Parameter Type Description

from String the input format

to String the output format

input Byte Array the input of the dataset to be converted

params ServiceInputParams auxiliary params needed for the conversion

These are indicated in the WSDL as following: <xs:element name="convert"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="from" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="to" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="input" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="params" nillable="true" type="ax24:ServiceInputParams"/> </xs:sequence> </xs:complexType> </xs:element>

For the first two arguments a String from the Formats enumeration java class is needed. (some valid values could be: “CSV” or “COMPACT_SDMX” – both taken from the Formats java enumeration).


Page 91

By giving the path of the input file you can create a FileInputStream object which can be converted to a byte array object. For example a client written in Java:

FileInputStream inputStream = new FileInputStream(new File(“D:/ selectedInputFile”)); Byte Array input = new byte [(int) inputStream.available()] inputStream.read(input);

Finally you have to specify a ServiceInputParams object in order to define the auxiliary parameters needed for the conversion. The parameters that could be set are the following:

Parameter Type Description

dataflowAgency String The agency of the dataflow

dataflowId String The id of the dataflow

dataflowVersion String The version of the dataflow

dateCompatibility String

delimiter String The delimiter

escapeCSVOutput String Flag specifying whether the CSV output should be escaped or not

excelMappingParameters Byte Array

The mapping of the excel parameters

excelType String

externalExcelParameter Byte Array

generatedFileComment String

gesmesWritingTechnique String

headerFile Byte Array

The header file

headerWriting Stirng

headerRow String

keyFamilyAgency String

keyFamilyId String

keyFamilyVersion String

mappingFile Byte Array

The mapping file

namespacePrefix String

namespaceUri String

orderedFlatInput String

specifyDataflow String

structureFile Byte Array

structureSetFile Byte the file with the transcoding information


Page 92

Array

useRegistry String

unescapeCSVInput String

levelNumber String

If the DSD is to be retrieved from SDMX Registry then the appropriate configuration should be done in the config.txt file that resides in the ‘package/params/’directory. The Registry URL, the Registry Action and the proxy setting should be set in the config.txt file. (Please consult the SDMX Converter Installation Guide Section 4).

The above parameters are depicted in the WSDL: <xs:complexType name="ServiceInputParams"> <xs:sequence> <xs:element minOccurs="0" name="dataflowAgency" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dataflowId" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dataflowVersion" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dateCompatibility" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="delimiter" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="escapeCSVOutput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="excelMappingParameters" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="excelType" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="headerRow" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="externalExcelParameter" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="generatedFileComment" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="gesmesWritingTechnique" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="headerFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="headerWriting" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyAgency" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyId" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyVersion" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="levelNumber" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="mappingFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="namespacePrefix" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="namespaceUri" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="orderedFlatInput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="specifyDataflow" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="structureFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="structureSetFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="unescapeCSVInput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="useRegistry" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType>


Page 93

The byte array structureFile, headerFile, mappingFile and structureSetFile can be read through a FileInputStream object, given the specific file path, and be converted into a byte array object in the same way as we described for the byte array input parameter.

The web service method

convert(String from, String to, byte array input, ServiceInputParams params)

returns a byte array output which is the converted file. This is depicted in the following snapshot of the WSDL: <xs:element name="convertResponse"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="return" nillable="true" type="xs:base64Binary"/> </xs:sequence> </xs:complexType> </xs:element>

You can write in a file the byte array output in the following way (Java based client): FileOutputStream fileoutStream = new FileOutputStream(new File(“D:/ selectedInputFile”)); fileoutStream.write(outpout);

5.2 A sample C# web service client for “convert”

A C# client invoking the SDMX Web service to perform a conversion from an SDMX Generic message to an SDMX Compact is provided below: ConverterWS.ConverterWebService test = new WindowsFormsApplication1.ConverterWS.ConverterWebService(); ConverterWS.ServiceInputParams @params= new WindowsFormsApplication1.ConverterWS.ServiceInputParams(); //FROM String from = "GENERIC_SDMX"; //TO String to = "CSV"; //Encoding String encoding = "UTF-8"; //Input StreamReader sr = new StreamReader("D:\\Results\\27_IN_Generic_Sts.xml"); Int32 streamlength = (Int32) sr.BaseStream.Length; Byte[] input = new Byte[streamlength]; sr.BaseStream.Read(input,0,streamlength); sr.BaseStream.Close(); //DSD @params.useRegistry = "true"; @params.specifyDataflow = "false"; @params.keyFamilyAgency = "ESTAT"; @params.keyFamilyId = "STS"; @params.keyFamilyVersion = "2.0"; byte[] output = test.convert(from,to,input,@params); //write the output FileStream File = new FileStream("D:\\Results\\outputCSV2.xml", FileMode.Create); File.Write(output,0,output.Length); File.Flush(); File.Close();

5.3 Using the “validate” method of the web service

The “validate” method performs a series of checks on its input: 1. Validation of the DSD which will be further used in the SDMX message

validation: a. XML syntax validation


Page 94

b. DSD against the XSD c. SdmxSource validation checks

2. for the SDMX message:

• XML syntax validation • SDMX data format specific validation • data validation • validation against the constraints

For a more detailed description of the validations performed, please consult Annex K: Checks done by the “validate” method of Converter Web Service

5.3.1 “validate” method parameters

In this version, the web service only accepts 2 types of data messages: • SDMX 2.0 Compact messages • SDMX 2.1 structure specific messages

As for the structure definition, the web service request accepts only a DSD (a Dataflow will not be accepted). Parameter Type Description

inputData Byte Array The SDMX message to be validated. This version of Converter only accepts two types of messages: • SDMX 2.0 Compact messages • SDMX 2.1 structure specific messages

dsdStructure Byte Array the DSD file (A dataflow is not accepted in this version)

maxErrorNumber Integer the max number of errors accepted by the system before stopping the validation

Table 1 - WS validate input parameters

<xs:element name="validate"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="inputData" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="dsdStructure" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="maxErrorNumber" type="xs:int"/> </xs:sequence> </xs:complexType> </xs:element>

5.3.2 “validate” method result

The “validate” method produces an error report as result where each error has an error code and a description. When possible, other details will also be also provided. <soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"> <soapenv:Body> <ns:validateResponse xmlns:ns="http://webservice.converter.sdmx.agilis.com"> <ns:return type="…" xmlns:ax21="…" xmlns:ax26="…" xmlns:ax24="…"> <ax26:error> <code xmlns="…">150</code> <column xmlns="…">5</column> <description xmlns="…">


Page 95

<string> Error While Processing XML: Unexpected XML attribute 'dummyHeader' under node 'ID' at line 4 column 5 </string> </description> <errorClass xmlns="…">SDMX</errorClass> <fatal xmlns="…">false</fatal> <line xmlns="…">4</line> </ax26:error> <ax26:error> <code xmlns="…">150</code> <column xmlns="…">5</column> <description xmlns="…"> <string> The dataset contains a concept dummy that is not defined in the DSD </string> </description> <errorClass xmlns="…">SDMX</errorClass> <fatal xmlns="…">false</fatal> <line xmlns="…">37</line> </ax26:error> <ax26:error> <code xmlns="…">150</code> <column xmlns="…">5</column> <description xmlns="…"> <string> The dataset contains a concept dummyGroup that is not defined in the DSD </string> </description> <errorClass xmlns="…">SDMX</errorClass> <fatal xmlns="…">false</fatal> <line xmlns="…">40</line> </ax26:error> <ax26:error> <attachedTo xmlns="…">GROUP</attachedTo> <code xmlns="…">150</code> <column xmlns="…">0</column> <description xmlns="…"> <string> The dataset group a series with an attribute dummyGroup that is not defined in the DSD </string> </description> <dimensions xmlns="…"> <dimensionValue> <id>REF_AREA</id> <value>LU</value> </dimensionValue> … </dimensions> <errorClass xmlns="…">DATA</errorClass> <fatal xmlns="…">false</fatal> <line xmlns="…">0</line> </ax26:error> … <ax26:errorsFound>5</ax26:errorsFound> <ax26:moreErrors>false</ax26:moreErrors> <ax26:returnCode>5</ax26:returnCode> </ns:return> </ns:validateResponse> </soapenv:Body> </soapenv:Envelope>

Let’s discuss the most important elements of the SOAP response to the validate operation: Xml Response Tag/Attribute

Description

returnCode This is the exit code of the validation: • -2 if there is an error in DSD or associated artefacts (or some of

them are missing) • -1 if there is a syntax error in the input SDMX-ML dataset


Page 96

• 0 if the validation is successful and no validation errors were detected

• >0 if one or more validation errors have been detected (other than the syntax errors)

errorsFound The number of errors found (syntax + structural + data) moreErrors This is a hint to the user that more potential errors can occur after

fixing the current ones because the validation algorithm did not check the whole input (the maxErrorNumber treshold has been reached and the validation algorithm has been stopped) • True – if the validation algorithm has been stopped because the

maxErrorNumber (see the input parameters) has been reached. • False - otherwise

error Error marker tag (see the details below) code The SDMX code of the error fatal True if the error has been fatal and the validation algorithm had to

be stopped because of it errorClass This specified the type of error. Possible values are:

• XML – for syntax errors • SDMX - if the error was a structural error • DATA – if the error is a data error • UNKNOWN – For unexpected or internal errors

line The line in the input file where the error occured. Applies to errors where the “errorClass” value is either SDMX or XML

column The column in the input file where the error occured. Applies to errors where the “errorClass” value is either SDMX or XML

description A detailed description of the error dimension Each “dimension” holds a dimension ID and its value. The whole

set of dimensions under an error represents the Series or Group key where the error occurred. Applies to errors where the “errorClass” value is “DATA”. With the exception when the “attachedTo” value is DATASET since there is no key at that point.

attachedTo The Dataset level where error occurred. Applies to errors where the “errorClass” value is “DATA”. Possible values are: • DATASET – if the error occurred at DataSet level. • SERIES – if the error occurred at Series level. • GROUP - if the error occurred at Group level. • OBS - if the error occurred at Observation level.

Table 2 – Result of the “validate” method explained

5.3.3 A sample request to “validate” using SOAPUI tool

SoapUI is the “de facto” standard in testing web services. The next paragraphs will show in a step-by-step manner how to create a call to the “validate” method of the Converter web service. At first a new project has to be created based on the WSDL of the Converter:


Page 97

Figure 56 - New SoapUI project for Converter Web Service

The picture below shows the two bindings created by the SoapUI based on the provided WSDL, each binding having two methods: convert and validate. In this section we will create a request to the “validate” method.

Figure 57 - Converter WS configured in SoapUI


Page 98

In the sample request created by the SOAP UI, please add two attachments (one for the DSD and the other for the message file). After adding these two parameters, make sure to match them with the ids provided by SOAP UI as in the picture below:

Figure 58 - Adding attachments to the "validate" request

If properly configured, the request should be sent to the server (by pressing the triangular play-like button). The response should look like the picture below:


Page 99

Figure 59 - the SOAP response of the validate method

5.4 Using the “customValidation” method of the web service

In addition to the checks performed by “validate” method, the “customValidation” method checks also the custom validation rules. The custom validation rules specify logical conditions that a dataset must satisfy in order be considered valid against a custom validation profile.

A RuleSet is a unit for packaging custom validation rules. It is defined using an XML schema and is stored on the server side in the form of XML file. Each rule set have a well-known ID that the users can specify as ruleSetID parameter to customValidation method. Converter Web Service will search the rule set file with the provided ID and load all the rules along with its dependencies.

5.4.1 “customValidation” method parameters

In this version, the web service only accepts 2 types of data messages: • SDMX 2.0 Compact messages • SDMX 2.1 structure specific messages

As for the structure definition, the web service request accepts only a DSD (a Dataflow will not be accepted). Parameter Type Description

inputData Byte Array The SDMX message to be validated. This version of Converter only accepts two types of messages: • SDMX 2.0 Compact messages • SDMX 2.1 structure specific messages

dsdStructure Byte Array the DSD file (A dataflow is not accepted in


Page 100

this version)

maxErrorNumber Integer the max number of errors accepted by the system before stopping the validation

ruleSetID String The ID of the rule set used for custom validation (one of the rulesets existing on the server).

This ID will be matched by the webservice against the filenames existing under Converter_Data/customvalidation folder. When a successful match has been found, those validation rules will be applied to the uploaded input file.

The file extension should not be specified in the ruleSetId value.

The available rule sets on the server will be disseminated to the end users by the administrators of the Converter Web Service.

Table 3 - WS “customValidate” input parameters

<xs:element name="customValidate"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="inputData" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="dsdStructure" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="maxErrorNumber" type="xs:int"/> <xs:element minOccurs="0" name="ruleSetID" type="xs:string"/> </xs:sequence> </xs:complexType> </xs:element>

5.4.2 “customValidate” method result

The result of “customValidation” call is the same as the “validate” call described in section 5.3.2. All the custom validation errors have errorClass=DATA, so they looks like any other data validation response.

5.5 Using an implemented SDMX Converter WS Client

In the package of the SDMX Converter Web Service a test client is included through which the web service can be invoked and conversions can be performed.


Page 101

Figure 60 - Test Client in the SDMX Converter Web Service Package

Open the folder “Test Client” and double click the “WindowsFormsApplication1.exe”

Figure 61 - Testing the SDMX Converter WS

The following Form appears:

Figure 62 - Converter WS Client

In the above Form the following can be specified: • The path of the input file in the ‘Input File’ text box • The path for the output file in the ‘Output File’ text box • The input format in the ‘Input Format’ combo box • The output format in the ‘Output Format’ combo box • The path of the header file in the ‘Header File’ text box


Page 102

• The parameters for the dsd specification

1. If the structure file is to be retrieved locally from a file then fill in the ‘DSD File’ text box with the path of the DSD, select the False value in the “Use Registry” combo box, select the False value in the “Use Dataflow” combo box.

2. If the structure file is to be retrieved locally from a file and has multiple DSDs then fill in the ‘DSD File’ text box with the path of the DSD, fill in the ‘DSD Id’ text box with the id of the DSD, fill in the ‘DSD Version’ text box with the version of the DSD, fill in the ‘DSD Agency’ with the agency of the DSD, select the False value in the “Use Registry” combo box, select the False value in the “Use Dataflow” combo box.

3. If the structure file is to be retrieved from the Registry then leave blank the ‘DSD File’ text box, , fill in the ‘DSD Id’ text box with the id of the DSD, fill in the ‘DSD Version’ text box with the version of the DSD, fill in the ‘DSD Agency’ with the agency of the DSD, select the True value in the “Use Registry” combo box, select the False value in the “Use Dataflow” combo box.

4. If the structure file is to be retrieved from the Registry by specifying a dataflow then then leave blank the ‘DSD File’ text box, the ‘DSD Id’ text box, the ‘DSD Version’ text box and the ‘DSD Agency’ text box. Select the True value in the “Use Registry” combo box, select the True value in the “Use Dataflow” combo box.Fill in the ‘Dataflow Id’ text box with the id of the Dataflow, fill in the ‘Dataflow Version’ text box with the version of the Dataflow, fill in the ‘Dataflow Agency’ with the agency of the Dataflow.

• The path of the mapping file (if needed) in the ‘Mapping File’ text box. • The number of levels in the ‘Levels’ text box. If the input or output CSV/FLR file

is a multilevel file then provide the exact number of levels. If it is not (flat file) then fill in the ‘Levels’ text box the number ‘1’.

• The path of the transcoding file (if needed) in the ‘Transcoding File’ text box. • The url of the SDMX Converter Web Service to be invoked.

Press the ‘Convert’ button to perform the conversion. For example for a conversion from a multilevel CSV file to an SDMX Compact file with header file, mapping file, transcoding file and number of levels=3 (the respective files have been provided in Section 4: Calling converter application from a command prompt) an instance of the SDMX Converter WS client is the following:


Page 103

Figure 63 - Instance of a Converter WS Client


Page 104

6 APPENDIX A: Tables of error messages In this section errors that might appear in the conversion procedure are presented. The table below has a unique error code that is associated with a generic message. In order for the user to apply successfully the recovery action some potential detail messages that belong to an error code and thus associate with one generic message are also presented.

6.1 Error messages for all usages of the SDMX Converter

Error code Generic Message Reason Recovery Action 100 Empty Parameters Some required

parameters for the conversion are missing.

Check carefully the parameters.

Potential Detail Message

Please correct the following: Input File parameter is empty

Input file is empty Define the Input File and perform again the conversion

Please correct the following: Cannot have the same Input and output formats

Input and Output formats are the same

Change the selected Input or Output Format according to the conversion you want to submit and perform again the conversion

Please correct the following: DSD File parameter is empty

DSD file is empty Define the DSD File and perform again the conversion

Please correct the following: Header File parameter is empty

Header file is empty

Define the Header File and perform again the conversion

Please correct the following: Input File parameter is empty Output File parameter is empty Cannot have the same input and output formats DSD File parameter is empty Header File parameter is empty

Input file field is empty, output file field is empty, dsd file field is empty, header file field is empty and Input and output formats are the same

Define the Input File and perform again the conversion, define in the Output File and perform again the conversion, define the DSD File field and perform again the conversion and fill in the Header File and perform again the conversion

Cannot convert, please correct the following errors first: Validation is not

The user tried to make a conversion while the input file format is CSV and

Set the SDMX Validation to false check box and perform again the conversion


Page 105

allowed for CSV selected input format

the SDMX Validation is true

Cannot convert, please correct the following errors first: Validation is not allowed for GESMES selected input format

The user tried to make a conversion while the input file format is GESMES and the SDMX Validation is checked

Uncheck the SDMX Validation check box and click Convert

101 Could not read parameters file.

The appropriate files have not been defined.

Read the installation guide for the configuration of required files for the initialisation of the SDMX Converter.

102 Error retrieving Key Family from Registry

The parameters for retrieving the Key Family from the SDMX Registry are not correct

Define the correct parameters, i.e. id, version and agency of the key family and perform again the conversion

103 Error retrieving Key Family

Could not retrieve key Family.

Define the correct parameters for retrieving the key family either from SDMX Registry or from a file and perform again the conversion.

104

Could not retrieve Dataflow from Registry

The user specifies a dataflow to retrieve the DSD, but the dataflow specification is not correct or could not access the Registry service.

Check the Registry URL.

Check out the Dataflow ID, Dataflow Version and Dataflow Agency and perform again the conversion.

105

Conversion cannot be performed! All Dimensions of the DSD must be mapped.

The user has provided a mapping that does not specify for one or more dimensions their position.

Check the mapping that has been provided and perform again the conversion.

106 Could not get the first Key Family from the returned DSD form Registry

The structure xml file does not have key families.

Check the parameters for retrieving the structure file either from the SDMX Registry or from the local file.


Page 106

107 Error while reading the input file

Problem in reading the input file Problem in populating the model and giving the right output.

Check the parameters of the conversion.

Check the input file.

Potential Detail Message

Utility format cannot be used for delete messages. Please see log file for further details.

The user tried to convert a delete input message to an output message of Utility format.

Fill in the ‘Output Format’ with any other SDMX format except the utility

Since DSD has no Time Dimension the conversion to the selected Output Format is not supported

The user tried to convert a message to an SDMX_ML (except for Cross Sectional) message with a DSD that has not Time Dimension.

Fill in the ‘Output Format’ with SDMX Cross Sectional format.

Since DSD has no Time Dimension the conversion from the selected Input Format to the selected Output Format is not supported

The user tried to convert an SDMX_ML (except for Cross Sectional) message with a DSD that has not Time Dimension.

Fill in the ‘Input Format’ with an SDMX Cross Sectional message.

Reader exception. The submitted message is not of CompactData type.

The user gave a wrong input format according with the input file.

Provide the Compact_SDMX format

Reader exception: The submitted message is not of CrossSectionalData type


Provide the CrossSectional_SDMX format

Reader exception. The submitted message is not of GenericData type.


Provide the Generic_SDMX format

Reader exception. The submitted message is not of UtilityData type.


Provide the Utility_SDMX format

Caught OutOfMemoryError. The keySetLimit is

The user gave a very large file for

Increase the java heap size (Please see Installation Guide


Page 107

already set to 1, cannot decrease further. Please increase the amount of memory available to java.

conversion. section 6.1.2)

Please correct the following:Mapping is mandatory for CSV multilevel file

The user tried to perform a conversion with multilevel CSV/FLR file without providing a mapping.

Provide the appropriate mapping in which the level of each component appears.

The dimension: … does not have a value. Please check the input Gesmes file.

The user provided a mapping with unmapped dimensions

Define a mapping for all dimensions.

Could not get observation time. Please check your input Gesmes File

There was not observation time in the ARR segment.

Please check the input file for validity according to the SDMX-EDI standard.

Error parsing Data Wrong DSD provided

Please check the DSD file provided and perform again the conversion.

Error parsing data, SAXParserException at line … column … : XML document structures must start and end within the same entity.

Not well formed xml File

Please check the xml files that have been provided for syntactic validity,

The Receiver Element of the Header is empty

The user tried to make a conversion to a Gesmes message and the receiver element that resides in the header of the Input SDMX_ML message is empty.

Please provide a receiver in the header.

exception while parsing non-associated dimensions: Code values have been reported but not assigned to a specific component. Please

In the input Gesmes file the attributes are defined but the code values for them are missing.

Please check the input Gesmes file.


Page 108

check the validity of the Input Gesmes File.

The time series key has not been parsed correctly. Please check your input Gesmes file.

The ARR segment in the input Gesmes file is not correct.

Please check the input Gesmes file.

The conversion to the output Cross Sectional message cannot be done because:Dimension …does not have a Cross-Sectional attachment level.

The DSD used is non cross sectional.

Please check the DSD. If the DSD is not compliant with producing Cross-Sectional Data, then convert to another format. A DSD that is compliant with producing Cross-Sectional Data contains for all Dimensions and all Attributes a cross-sectional attachment level.

108 Unhanded Error An unexpected error has happened

Please check all the parameters that have been given in the conversion.

6.2 Error messages concerning SDMX Converter GUI

Error code Generic Message Reason Recovery Action 200 Error parsing Key

Family. The DSD file is corrupted

Please check the DSD and click the Convert Button in the SDMX Converter Gui.

201 The Input File could not be parsed.

An input file was not selected from the ‘Browse’ button of the SDMX Converter Gui.

Select the input file again from the ‘Browse’ button and click the Convert Button in the SDMX Converter Gui.

202 Could not write the transcoding File

The path of the transcoding file is not correct

Try to save again the transcoding file.

203 This is not a valid mapping file

The mapping file is not a well formed xml file or it is not consistent with the DSD

Check the mapping file for well-form ness.

Check the mapping file and the selected DSD


Page 109

Click the Convert Button in the SDMX Converter Gui.

204 Could not read mapping file

The path of the mapping file is not correct

Provide the right mapping file path. Click the Convert Button in the SDMX Converter Gui.

205 Could not write the XML mapping File

An error during writing the xml file occurred.

Check the file that has been specified to save the mapping file. Try to save again the mapping file.

206 This is not a valid transcoding file

An invalid transcoding file has been used.

Check the transcoding file and click the Convert Button in the SDMX Converter Gui.

207 Problem loading header file in editing header dialog.

The user tried to load a header.prop file in the edit header dialog.

Check the path of file that has been specified to load the header information.

208 Problem parsing the date.

The user has specified a wrong date in the fields that require a date in the edit header dialog.

Check the date that you have entered.

209 Error while storing the header properties file.

The user tried to save a header.prop file in the edit header dialog.

Check the path of file that has been specified to save the header information. Try to save again the header.

210 Could not write the XML template File

An error during writing the template xml file occurred.

Check the path of file that has been specified to save the template file. Try to save again the template file.

211 Could not load the template file for the conversion

The user tried to load a template file file.

Check the path of file that has been specified to load the template.

212 Could not find the header file.

When saving the template the header couldn’t be parsed.

Check the path of the header file for


Page 110

existence.

213 The template file is not valid.

The template xml file that has been loaded is invalid.

Please check again the template xml file for missing elements or wrong values.

6.3 Error messages concerning SDMX Converter CLI

Error code Generic Message Reason Recovery Action 300 Invalid mapping File The user uses an

invalid mapping file.

Check the mapping file that has been used and perform again the conversion.

301 Invalid transcoding file

The user uses an invalid transcoding file.

Check the transcoding file that has been used and perform again the conversion

6.4 Error messages concerning Excel conversion

Error code Generic Message Reason Recovery Action 400 Error reading Excel

Parameters The parameter file or parameter sheet is not correct

Check the parameter file and parameter sheet, correct it and perform again the conversion.

107 Input Parsing Exception: The Time Value is not formatted according with the FREQ value.

The Time Period value is wrong according with the FREQ value

Modify the input Excel file such as frequency and time have the correct values.

Note for Excel conversion: If you have entered data in columns/rows at the right or end of your spreadsheet’s data area and later deleted that added data, Excel will remember that there was once something in that column or row. This will make Excel file size dramatically increase due to empty rows. Consequently Converter tries to process all these empty rows and sometimes is possible to end with Out Of Memory error. For unclear reason this problem could appear when copy-paste date between different versions of Excel. In order to solve this problem follow these steps: 1. Left click on the first empty row you want to delete after your data. 2. Hold shift+ctrl and then press down arrow.


Page 111

3. All rows (about 65536 in Excel 97 to 2003 or 1048576 in Excel 2007) should be highlighted. 4. Right click and delete all the rows. 5. Immediately SAVE your workbook. The above steps usually decrease the Excel file size such as it could be easily parsed by Converter.


Page 112

7 APPENDIX B – Structure of a multilevel CSV/FLR In this section the structure of a CSV/FLR file that is multilevel will be described.

7.1 Multilevel files

Flat CSV/FLR files imply that they will contain only one level information, i.e. Observation level. Thus, all records are equivalent in terms of structure and semantic. As the sample below all records have the same number of columns (delimiters) with equivalent semantic (per column position), where the first column (with value M is the FREQ - Frequency), the second (values: GR, IT is the REF_AREA (Reference Area), etc. M,GR,W,PROD,NS0020,1,2000,200501,1.1,F,A M,GR,W,PROD,NS0020,1,2000,200502,2.2,F,A ………………………………………………… M,IT,W,PROD,NS0020,1,2000,200501,1.2,F,A M,IT,W,PROD,NS0020,1,2000,200502,2.5,F,A The above sample CSV flat file includes all key values (values for all dimensions, including time), as well as the observed value (i.e. 1.2 and 2.5) and its attached attributes (i.e.F,A). There are cases though, that a CSV file may contain a multi-level representation in order to express metadata that concern a set/group of records. For example: 1,Dataset Attribute1 value,Dataset Attribute2 value 2,GR,W,PROD,NS0020,1,2000,PC,0,2,Elements of the full national etc. 3,M,A,A,P1M 4,200501,1.1,F,A 4,200502,2.2,F,A 4,200503,3.3,F,A ... 3,A,A,A,P1Y 4,2005,123,F,A 4,2006,456,F,A ... 2,GR,W,PROD,NS0021,1,2000,PC,0,2,Elements of the full national etc. 3,M,A,A,P1M 4,200501,1.1,F,A 4,200502,2.2,F,A 4,200503,3.3,F,A ... 3,A,A,A,P1Y 4,2005,123,F,A 4,2006,456,F,A ... The prerequisites in the above example are:


Page 113

• The first column is considered a level identifier, i.e. it defines the level of the record.

• The order of records is significant, since each level is considered a child of the previously higher level encountered within the CSV file.

• The number of columns (delimiters) per level is equal • The column semantic per level is equivalent

In this case the key values may be split among the different levels.

7.2 Sample multilevel CSV file

In order to convert from or to a multilevel CSV/FLR file the number of the levels and the appropriate mapping file should be provided. Also the argument that defines if the input file is ordered or not should be set to true. There do not exist any constraints and the conversion to/from all SDMX formats can be done. Bellow is presented an example with a conversion from a multilevel file to a SDMX message. The mapping in Error! Reference source not found. is applicable for the following csv file, which describes demographic data for the year 2005. 1;RQFI05V1;1;A 2;FI;F;ADJT;0;0;P1Y;PERS 3;2005;35;P 1;RQFI05V1;1;M 2;FI;F;DEATHST;0;0;P1Y;PERS 3;2005;23871;P 1;RQFI05V1;1;Q 2;FI;F;LBIRTHST;0;0;P1Y;PERS 3;2005;28345;P

Source file Output file Command Description/Details/Commen

Example 1

1;RQFI05V2;FI;F;ADJT3;2005;35;P1;RQFI05V2;FI;F;DEAT3;2005;2381;RQFI05V2;FI;F;LBIR3;2005;283

OUT_Cross.xml

converter.bat -header_file "D:\Test Case 8\header.prop" -dsd_file "D:\Test Case 8\ESTAT+DEMOGRAPHY+2.1.xml" -mapping_file "D:\Test Case 8\30.3_mapping.xml" -ordered_input true -level 3 -delimiter ; "D:\Test Case 8\IN_CSV.csv" "D:\Test Case 8\OUT_Cross.xml" CSV CROSS_SDMX

A conversion from a CSV multilevel file to a Cross Sectional file with mapping file“30.3_mapping.xml”, header fi“header.prop” and number of levels=3.

7.3 Sample CSV file with cross sectional measures that appear in one record

When converting from a CSV file using a Cross Sectional DSD, the cross sectional measures that belong to a series might appear in one record rather than in consecutive rows. The following two file structures might appear when converting with a cross sectional DSD.


Page 114

In the first structure each observation value appears along with the cross sectional measure in one row. Each cross sectional measure appears in the same row in each record (4th column). This has been defined in the mapping procedure where the measure dimension is always mapped to the specific column. In the following sample for the Country Greece (GR) and for female (F) the annual (A) observation values are reported consecutively in rows and correspond to Deaths (DEATHST), Emigrants (EMIGT), Immigrants (IMMIT) and Births (LBIRTHST) respectively.

A;GR;F;DEATHST;2005;23871;P A;GR;F;EMIGT;2005;6331;P A;GR;F;IMMIT;2005;10581;P A;GR;F;LBIRTHST;2005;28345;P

In the second file structure the components that correspond to cross sectional measures do not appear in the CSV, but only their values. A;GR;F; 23871; 6331;10581;28345;2005;P In the above record of the CSV file only the observation values that correspond to the cross sectional measure appear. Specifically the value 23871 is the measure of Deaths, the value 6331 of Emigrants, the value of 10581 of Immigrants and the value 28345 of Births. In the mapping procedure the user should select to map cross sectional measures (please see Error! Reference source not found.) and map each measure to a column in the mapping dialog. For this example the measure DEATHST has been mapped to column 4, the measure EMIGT to column 5, the measure IMMIT to column 6 and the measure LBIRTHST to column 7. In that way the user does not have to map the observation value as this will not appear in a specific column in the CSV file but will appear in the column that the respective cross sectional measure has been mapped.


Page 115

8 APPENDIX C – Sample template In this section a sample xml file of a template conversion as well as its xsd file are presented.

• The Template.xml File: <?xml version="1.0" encoding="UTF-8"?> <Conversion_Template xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:noNamespaceSchemaLocation="Template.xsd"> <Input_Output> <InputFileName format="COMPACT_SDMX">/Users/fch/Desktop/QTM1/SDMX_Converter_Documentation_v.2.5.7_31032010/Testing Documentation/Test Plan and Test Results for SDMX Converter_v2.10_31032010/Converter_Test_Cases_V2.10/Test Procedure 34/34.2_IN_Compact_Dem2.1.xml</InputFileName> <OutputFileName format="FLR">/Users/fch/Desktop/Output_FLR.txt</OutputFileName> </Input_Output> <Registry_Parameters> <DSD agency="" id="" version="">/Users/fch/Desktop/QTM1/SDMX_Converter_Documentation_v.2.5.7_31032010/Testing Documentation/Test Plan and Test Results for SDMX Converter_v2.10_31032010/Converter_Test_Cases_V2.10/Test Procedure 34/ESTAT+DEMOGRAPHY+2.1.xml</DSD> </Registry_Parameters> <CSV_Parameters> <Input> <InputOrdered>true</InputOrdered> <HeaderInformation></HeaderInformation> </Input> <Output> <OutputDateFormat>SDMX</OutputDateFormat> <WriteHeader>false</WriteHeader> </Output> <headerRow>true</headerRow> <MappingInfo>Map CrossX measures</MappingInfo> <Mapping> <Concept name="FREQ" level="" value="1-1" fixed="false" /> <Concept name ="COUNTRY" level="" value ="2-3" fixed ="false" /> <Concept name ="SEX" level="" value ="4-4" fixed ="false" /> <Concept name ="PJANT" level="" value ="6-12" fixed ="false" /> <Concept name ="LBIRTHST" level="" value ="14-18" fixed ="false" /> <Concept name ="DEATHST" level="" value ="20-26" fixed ="false" /> <Concept name ="ADJT" level="" value ="27-30" fixed ="false" /> <Concept name ="PJAN1T" level="" value ="32-40" fixed ="false"/> <Concept name ="LBIRTHOUT" level="" value ="41-48" fixed ="false" /> <Concept name ="DEATHUN1" level="" value ="49-52" fixed ="false" /> <Concept name ="MAR" level="" value ="54-60" fixed ="false" /> <Concept name ="DIV" level="" value ="61-68" fixed ="false" /> <Concept name ="IMMIT" level="" value ="70-76" fixed ="false" /> <Concept name ="EMIGT" level="" value ="78-83" fixed ="false" /> <Concept name ="NETMT" level="" value ="84-89" fixed ="false" /> <Concept name ="TFRNSI" level="" value ="90-93" fixed ="false" /> <Concept name ="LEXPNSIT" level="" value ="94-100" fixed ="false"/> <Concept name ="TIME" level="" value ="101-105" fixed ="false" /> <Concept name ="OBS_STATUS" level="" value ="106-106" fixed ="false" /> </Mapping> <Transcoding /> <CSVDelimiter>;</CSVDelimiter> </CSV_Parameters> <Other_Parameters> <GesmesTsTechnique>Time Range</GesmesTsTechnique> <SDMXValidation>false</SDMXValidation> </Other_Parameters> </Conversion_Template>

• The Template.xsd file

<?xml version="1.0" encoding="UTF-8"?>


Page 116

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"> <xs:element name="Conversion_Template"> <xs:complexType> <xs:sequence> <xs:element ref="Input_Output"/> <xs:element ref="Registry_Parameters"/> <xs:element ref="CSV_Parameters"/> <xs:element ref="Other_Parameters"/> <xs:element ref="Namespace" minOccurs="0"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Input_Output"> <xs:complexType> <xs:sequence> <xs:element ref="InputFileName"/> <xs:element ref="OutputFileName"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="InputFileName"> <xs:complexType mixed="true"> <xs:attribute name="format" type="FormatType" use="required"/> </xs:complexType> </xs:element> <xs:element name="OutputFileName"> <xs:complexType mixed="true"> <xs:attribute name="format" type="FormatType" use="required"/> </xs:complexType> </xs:element> <xs:element name="Registry_Parameters"> <xs:complexType> <xs:choice> <xs:element ref="DSD"/> <xs:element ref="Dataflow"/> </xs:choice> </xs:complexType> </xs:element> <xs:element name="DSD"> <xs:complexType mixed="true"> <xs:attribute name="agency" type="IDType" use="required"/> <xs:attribute name="id" type="IDType" use="required"/> <xs:attribute name="version" type="xs:string" use="required"/> </xs:complexType> </xs:element> <xs:element name="Dataflow"> <xs:complexType> <xs:attribute name="agency" type="IDType" use="required"/> <xs:attribute name="id" type="IDType" use="required"/> <xs:attribute name="version" type="xs:string" use="required"/> </xs:complexType> </xs:element> <xs:element name="CSV_Parameters"> <xs:complexType> <xs:sequence> <xs:element ref="Input"/> <xs:element ref="Output"/> <xs:element ref="headerRow"/> <xs:element ref="MappingInfo"/> <xs:element ref="Mapping"/> <xs:element ref="Transcoding"/> <xs:element ref="CSVDelimiter"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Input"> <xs:complexType> <xs:sequence> <xs:choice> <xs:element ref="MultiLevelCSV"/> <xs:element ref="InputOrdered"/> </xs:choice> <xs:element ref="HeaderInformation"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="HeaderInformation" type="xs:string"/> <xs:element name="MultiLevelCSV"> <xs:simpleType>


Page 117

<xs:restriction base="xs:integer"> <xs:minInclusive value="2"/> </xs:restriction> </xs:simpleType> </xs:element> <xs:element name="InputOrdered" type="xs:boolean"/> <xs:element name="Output"> <xs:complexType> <xs:sequence> <xs:element ref="OutputDateFormat"/> <xs:element ref="WriteHeader"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="OutputDateFormat"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="SDMX"/> <xs:enumeration value="GESMES"/> </xs:restriction> </xs:simpleType> </xs:element> <xs:element name="WriteHeader" type="xs:boolean"/> <xs:element name="headerRow" type="xs: boolean "/> <xs:element name="MappingInfo"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="Map measure Dimension"/> <xs:enumeration value="Map CrossX measures"/> </xs:restriction> </xs:simpleType> </xs:element> <xs:element name="Mapping"> <xs:complexType> <xs:sequence> <xs:element ref="Concept" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Concept"> <xs:complexType> <xs:attribute name="name" type="xs:string" use="required"/> <xs:attribute name="value" type="xs:string" use="required"/> <xs:attribute name="level" type="xs:string" use="optional"/> <xs:attribute name="fixed" type="xs:boolean" use="optional"/> <xs:attribute name=" targetconcept" type="xs:string" use="optional"/> </xs:complexType> </xs:element> <xs:element name="Transcoding"> <xs:complexType> <xs:sequence> <xs:element ref="StructureSet" minOccurs="0"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="StructureSet" > <xs:complexType> <xs:sequence> <xs:element ref="CodelistMap" maxOccurs="unbounded"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="CodelistMap"> <xs:complexType> <xs:sequence> <xs:element ref="CodeMap" maxOccurs="unbounded"/> </xs:sequence> <xs:attribute name="id" type="xs:string" use="required"/> </xs:complexType> </xs:element> <xs:element name="CodeMap"> <xs:complexType> <xs:sequence> <xs:element ref="MapCodeRef"/> <xs:element ref="MapTargetCodeRef"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="MapCodeRef" type="xs:string"/>


Page 118

<xs:element name="MapTargetCodeRef" type="xs:string"/> <xs:element name="CSVDelimiter" type="xs:string"/> <xs:element name="Other_Parameters"> <xs:complexType> <xs:sequence> <xs:element ref="GesmesTsTechnique"/> <xs:element ref="SDMXValidation"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="GesmesTsTechnique"> <xs:simpleType> <xs:restriction base="xs:string"> <xs:enumeration value="Time Range"/> <xs:enumeration value="Single Observation"/> </xs:restriction> </xs:simpleType> </xs:element> <xs:element name="SDMXValidation" type="xs:boolean"/> <xs:element name="Namespace"> <xs:complexType> <xs:sequence> <xs:element ref="URN"/> <xs:element ref="Prefix"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="URN" type="xs:anyURI"/> <xs:element name="Prefix" type="xs:NCName"/> <xs:simpleType name="FormatType"> <xs:restriction base="xs:string"> <xs:enumeration value="CSV"/> <xs:enumeration value="FLR"/> <xs:enumeration value="COMPACT_SDMX"/> <xs:enumeration value="GENERIC_SDMX"/> <xs:enumeration value="UTILITY_SDMX"/> <xs:enumeration value="CROSS_SDMX"/> <xs:enumeration value="GESMES_TS"/> <xs:enumeration value="GESMES_2_1"/> <xs:enumeration value="GESMES_DSIS"/> <xs:enumeration value="DSPL"/> </xs:restriction> </xs:simpleType> <xs:simpleType name="IDType"> <xs:annotation> <xs:documentation>IDType provides a type which is used for restricting the characters in codes and IDs throughout all SDMX-ML messages. Valid characters include A-Z, a-z, @, 0-9, _, -, $.</xs:documentation> </xs:annotation> <xs:restriction base="xs:string"> <xs:pattern value="([A-Z]|[a-z]|\*|@|[0-9]|_|$|\-|/|=)*"/> </xs:restriction> </xs:simpleType> </xs:schema>


Page 119

9 APPENDIX D - Examples of Excel to SDMX conversion In this section few examples related to Excel to SDMX conversion are described. The attached 0101QY.xlsx will be used with input format Excel. It has two data sheets 0101 QY V and 0101 QY L and only one parameter sheet. In this case there will be no need for an external parameter file and also no mapping parameter is needed as the existing parameter sheet will be used to process all data sheets. But if an external parameter file is provided then the external file will be used.

0101QY.xlsx

ExcelData.xlsx has two data sheets and two parameter sheets. If no mapping and no external parameter file is provided then the first parameter sheet – Parameter_Data1 is used. If an external parameter file (e.g. the attached ExternalParameterFile.txt ) is given, but no mapping, then the external parameter file is used. If a mapping between data sheets and parameter sheets is provided then the parameter sheets are used according to the mapping. In MappingXlsParameters.txt Data1 will be processed using description from Parameters_Data1 and Data2 with Parameters_Data2.

ExcelData.xlsx

ExternalParameterFile.txt MappinsXlsParameters.txt


Page 120

10 APPENDIX E – How Excel handles decimal numbers

10.1 Excel adheres to the IEEE 754 standard

This means that Excel is not always accurate when performing floating point computations (see this link from Microsoft).

10.2 What Excel displays is NOT what is stored internally

Excel stores numbers differently that you may have them formatted on the worksheet. Under normal circumstances, Excel stores numeric values as "Double Precision Floating Point" numbers, or "Doubles" for short. These are 8-byte variables that can store numbers accurate to approximately 15 decimal places. You may have only two decimal places displayed on the worksheet, but the underlying value has the full 15 decimal places. (Paragraph taken from here)

When a certain value is obtained as the computation of several other cells, due to the floating point arithmetic (see point 1), Excel displays rounded values of what it stores internally. If you export the worksheet to Office 2003 XML format, you will see the real value stored. Example: Make the following computation in Excel: 4.1 – 4 and copy-paste the result into another worksheet The result will be shown in the formula bar as 0.099999999999 but the cell will contain the value 0.1 Any API reading excel will have access to the internal stored value that’s why we have to apply a rounding algorithm when reading values from Excel.


Page 121

11 APPENDIX F – The rounding mechanism implemented in Converter

As we’ve seen in Appendix E, Excel uses the IEEE 754 standard for handling decimal number arithmetic. Due to this standard, for some computations, Excel uses internally approximations of the mathematical results (i.e. for a simple operation like 4.1 - 4 Excel stores internally the result 0.0999999999994). Knowing that this approximation is not at all user-friendly, Excel performs a rounding algorithm to improve the displaying of Numbers (i.e. as expected for the formula 4.1 - 4 the displayed result is 0.1 and this is because of the rounding not due to the correct mathematical computation). Since Excel exports to any API (including the JAVA APIs) the internal stored values (i.e. the approximations) instead of the displayed values (the rounded ones), SDMX Converter is forced to use a rounding mechanism itself in order to display the same numbers as those shown inside Excel. Tests have shown that the HALF_EVEN algorithm provides the best results. HALF_EVEN is the rounding mode to round towards the "nearest neighbour" unless both neighbours are equidistant, in which case, round towards the even neighbour. It behaves as for HALF_UP if the digit to the left of the discarded fraction is odd and it behaves as for HALF_DOWN if it's even. This is the rounding mode that statistically minimizes cumulative error when applied repeatedly over a sequence of calculations. It is sometimes known as "Banker's rounding," and is chiefly used in the USA. This rounding mode is analogous to the rounding policy used for float and double arithmetic in Java.

Some rounding examples:

Excel Converter

Number as provided by Excel

Converted using default rounding precision (= 6)

Converted using rounding precision 0

Converted using rounding precision 15

1.2 1.2 1 1.2

1.49 1.49 1 1.49

1.5 1.5 2 1.5

577658.06999999995 577658.07 577658 577658.06999999995

As shown in the table above, if user sets the precision too high, converter will display the values as they come from Excel (i.e. if the user sets the precision to 15 then we cannot hide the 0.999-like values which we are trying to avoid through this rounding).


Page 122

12 APPENDIX G – CSV Input/Output escaping mechanism

This Appendix addresses the problem of conversion from any input into CSV. The problem arises when the separator is outputted in the values of Converter’s CSV result, confusing the other CSV parsing applications with the extra delimiters. As an example, let’s assume Converter needs to output three values (see below) with comma (,) as delimiter:

aaa bbb cc,c

In this example as input we have three values organized in three columns. The way these values are outputted could be: aaa,bbb,cc,c Any application trying to parse the values outputted by Converter will get 4 values because the last value contains a comma which in this case is also the data-separator:

aaa bbb cc c

For this case Converter will detect that the separator is included in data and the conversion fails as the output file is not a valid one.

In order to solve the situation the user has to check the “Escape CSV Output Fields” checkbox in the GUI (or set to true escapeCSVOutput option of the CLI and WS version). By setting this parameter to true Converter will enclose all the output values in quotes resulting in a valid CSV file which could be further read by other application. The default value for this parameter is set to true, so if is not specified in Converter CLI or WS all the values in the output CSV file will be quoted.

12.1 CSV as Input

When an input CSV file contains escaped values Converter needs to be informed by checking “Unescape CSV Input fields” checkbox in the GUI or set to true unescapeCSVInput option of the CLI and WS version. By default this parameter is set to false.

For example the input file could contain a line like this:

"aaa";"bb";"aa;bb";"aa""bb""cc";"a\d" and the CSV separator is “;”

In this case if “Unescape CSV Input fields” is set to true the resulted output will be:

<Series Concept1="aaa" Concept2="bb" Concept3="aa;bb" Concept4="aa"bb"cc" Concept5="a\d" />

The reference for escaping values in the input CSV file is RFC 4180.

The most important points in the RFC 4180 which are taken into account by Converter when parsing an input CSV file are:


Page 123

1. Each record is located on a separate line, delimited by a line break (CRLF). For example: aaa,bbb,ccc CRLF zzz,yyy,xxx CRLF 2. The last record in the file may or may not have an ending line break. For example: aaa,bbb,ccc CRLF zzz,yyy,xxx […] 5. Each field may or may not be enclosed in double quotes […]. If fields are not enclosed with double quotes, then double quotes may not appear inside the fields. For example:

"aaa","bbb","ccc" CRLF zzz,yyy,xxx 6. Fields containing double quotes and delimiters should be enclosed in double-quotes. For example:

"aaa","b,bb","ccc" CRLF zzz,yyy,xxx

7. If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote. For example:

"aaa","b""bb","ccc"


Page 124

13 APPENDIX H – Converter Web Service WSDL

<?xml version="1.0" encoding="UTF-8"?> <wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:ns1="http://org.apache.axis2/xsd" xmlns:ns="http://webservice.converter.sdmx.agilis.com" xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl" xmlns:http="http://schemas.xmlsoap.org/wsdl/http/" xmlns:ax26="http://errorreport.webservice.converter.sdmx.agilis.com/xsd" xmlns:ax24="http://webservice.converter.sdmx.agilis.com/xsd" xmlns:ax21="http://exceptions.ui.converter.sdmx.agilis.com/xsd" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" targetNamespace="http://webservice.converter.sdmx.agilis.com"> <wsdl:documentation>ConverterWebService</wsdl:documentation> <wsdl:types> <xs:schema

attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://errorreport.webservice.converter.sdmx.agilis.com/xsd">

<xs:complexType name="ErrorReportType"> <xs:sequence> <xs:element minOccurs="0" name="converterVersion" nillable="true" type="xs:string"/> <xs:element maxOccurs="unbounded" minOccurs="0" name="error" nillable="true" type="ax26:ErrorType"/> <xs:element minOccurs="0" name="errorsFound" type="xs:int"/> <xs:element minOccurs="0" name="moreErrors" type="xs:boolean"/> <xs:element minOccurs="0" name="returnCode" nillable="true" type="ax26:ReturnCodeType"/> </xs:sequence> </xs:complexType> <xs:complexType name="ErrorType"> <xs:sequence> <xs:element minOccurs="0" name="attachedTo" nillable="true" type="ax26:DataSetLevelType"/> <xs:element minOccurs="0" name="code" type="xs:int"/> <xs:element minOccurs="0" name="description" nillable="true" type="ax26:TextType"/> <xs:element minOccurs="0" name="dimensions" nillable="true" type="ax26:DimensionsType"/> <xs:element minOccurs="0" name="fatal" type="xs:boolean"/> <xs:element minOccurs="0" name="_class" nillable="true" type="ax26:ErrorClassType"/> </xs:sequence> </xs:complexType> <xs:complexType name="DataSetLevelType"> <xs:sequence> <xs:element minOccurs="0" name="value" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType> <xs:complexType name="TextType"> <xs:sequence> <xs:element minOccurs="0" name="string" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType> <xs:complexType name="DimensionsType"> <xs:sequence> <xs:element maxOccurs="unbounded" minOccurs="0" name="dimensionValue" nillable="true" type="ax26:DimensionValueType"/> </xs:sequence> </xs:complexType> <xs:complexType name="DimensionValueType"> <xs:sequence> <xs:element minOccurs="0" name="id" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="value" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType>


Page 125

<xs:complexType name="ErrorClassType"> <xs:sequence> <xs:element minOccurs="0" name="value" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType> <xs:complexType name="ReturnCodeType"> <xs:sequence> <xs:element minOccurs="0" name="returnCodeType" type="xs:int"/> </xs:sequence> </xs:complexType> </xs:schema> <xs:schema xmlns:ax27="http://errorreport.webservice.converter.sdmx.agilis.com/xsd" xmlns:ax25="http://webservice.converter.sdmx.agilis.com/xsd" xmlns:ax23="http://exceptions.ui.converter.sdmx.agilis.com/xsd" attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://webservice.converter.sdmx.agilis.com"> <xs:import namespace="http://exceptions.ui.converter.sdmx.agilis.com/xsd"/> <xs:import namespace="http://webservice.converter.sdmx.agilis.com/xsd"/> <xs:import namespace="http://errorreport.webservice.converter.sdmx.agilis.com/xsd"/> <xs:complexType name="Exception"> <xs:sequence> <xs:element minOccurs="0" name="Exception" nillable="true" type="xs:anyType"/> </xs:sequence> </xs:complexType> <xs:element name="BaseException"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="BaseException" nillable="true" type="ax21:BaseException"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Exception"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="Exception" nillable="true" type="ns:Exception"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="convert"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="from" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="to" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="input" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="serviceparams" nillable="true" type="ax24:ServiceInputParams"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="convertResponse"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="return" nillable="true" type="xs:base64Binary"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="validate"> <xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="inputData" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="dsdStructure" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="maxErrorNumber" type="xs:int"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="validateResponse">


Page 126

<xs:complexType> <xs:sequence> <xs:element minOccurs="0" name="return" nillable="true" type="ax26:ErrorReportType"/> </xs:sequence> </xs:complexType> </xs:element> </xs:schema> <xs:schema xmlns:ax22="http://webservice.converter.sdmx.agilis.com" attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://exceptions.ui.converter.sdmx.agilis.com/xsd"> <xs:import namespace="http://webservice.converter.sdmx.agilis.com"/> <xs:complexType name="BaseException"> <xs:complexContent> <xs:extension base="ax22:Exception"> <xs:sequence> <xs:element minOccurs="0" name="code" type="xs:int"/> <xs:element minOccurs="0" name="details" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="userMessage" nillable="true" type="xs:string"/> </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType> </xs:schema> <xs:schema attributeFormDefault="qualified" elementFormDefault="qualified" targetNamespace="http://webservice.converter.sdmx.agilis.com/xsd"> <xs:complexType name="ServiceInputParams"> <xs:sequence> <xs:element minOccurs="0" name="dataflowAgency" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dataflowId" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dataflowVersion" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="dateCompatibility" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="delimiter" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="escapeCSVOutput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="excelMappingParameters" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="excelType" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="externalExcelParameter" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="generatedFileComment" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="gesmesWritingTechnique" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="headerFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="headerRow" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="headerWriting" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyAgency" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyId" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="keyFamilyVersion" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="levelNumber" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="mappingFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="namespacePrefix" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="namespaceUri" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="orderedFlatInput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="specifyDataflow" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="structureFile"


Page 127

nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="structureSetFile" nillable="true" type="xs:base64Binary"/> <xs:element minOccurs="0" name="unescapeCSVInput" nillable="true" type="xs:string"/> <xs:element minOccurs="0" name="useRegistry" nillable="true" type="xs:string"/> </xs:sequence> </xs:complexType> </xs:schema> </wsdl:types> <wsdl:message name="convertRequest"> <wsdl:part name="parameters" element="ns:convert"/> </wsdl:message> <wsdl:message name="convertResponse"> <wsdl:part name="parameters" element="ns:convertResponse"/> </wsdl:message> <wsdl:message name="BaseException"> <wsdl:part name="parameters" element="ns:BaseException"/> </wsdl:message> <wsdl:message name="Exception"> <wsdl:part name="parameters" element="ns:Exception"/> </wsdl:message> <wsdl:message name="validateRequest"> <wsdl:part name="parameters" element="ns:validate"/> </wsdl:message> <wsdl:message name="validateResponse"> <wsdl:part name="parameters" element="ns:validateResponse"/> </wsdl:message> <wsdl:portType name="ConverterWebServicePortType"> <wsdl:operation name="convert"> <wsdl:input message="ns:convertRequest" wsaw:Action="urn:convert"/> <wsdl:output message="ns:convertResponse" wsaw:Action="urn:convertResponse"/> <wsdl:fault message="ns:BaseException" name="BaseException" wsaw:Action="urn:convertBaseException"/> <wsdl:fault message="ns:Exception" name="Exception" wsaw:Action="urn:convertException"/> </wsdl:operation> <wsdl:operation name="validate"> <wsdl:input message="ns:validateRequest" wsaw:Action="urn:validate"/> <wsdl:output message="ns:validateResponse" wsaw:Action="urn:validateResponse"/> <wsdl:fault message="ns:BaseException" name="BaseException" wsaw:Action="urn:validateBaseException"/> <wsdl:fault message="ns:Exception" name="Exception" wsaw:Action="urn:validateException"/> </wsdl:operation> </wsdl:portType> <wsdl:binding name="ConverterWebServiceSoap11Binding" type="ns:ConverterWebServicePortType"> <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/> <wsdl:operation name="convert"> <soap:operation soapAction="urn:convert" style="document"/> <wsdl:input> <soap:body use="literal"/> </wsdl:input> <wsdl:output> <soap:body use="literal"/> </wsdl:output> <wsdl:fault name="Exception"> <soap:fault use="literal" name="Exception"/> </wsdl:fault> <wsdl:fault name="BaseException"> <soap:fault use="literal" name="BaseException"/> </wsdl:fault> </wsdl:operation> <wsdl:operation name="validate"> <soap:operation soapAction="urn:validate" style="document"/> <wsdl:input> <soap:body use="literal"/> </wsdl:input> <wsdl:output> <soap:body use="literal"/> </wsdl:output> <wsdl:fault name="Exception"> <soap:fault use="literal" name="Exception"/> </wsdl:fault> <wsdl:fault name="BaseException"> <soap:fault use="literal" name="BaseException"/>


Page 128

</wsdl:fault> </wsdl:operation> </wsdl:binding> <wsdl:binding name="ConverterWebServiceSoap12Binding" type="ns:ConverterWebServicePortType"> <soap12:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/> <wsdl:operation name="convert"> <soap12:operation soapAction="urn:convert" style="document"/> <wsdl:input> <soap12:body use="literal"/> </wsdl:input> <wsdl:output> <soap12:body use="literal"/> </wsdl:output> <wsdl:fault name="Exception"> <soap12:fault use="literal" name="Exception"/> </wsdl:fault> <wsdl:fault name="BaseException"> <soap12:fault use="literal" name="BaseException"/> </wsdl:fault> </wsdl:operation> <wsdl:operation name="validate"> <soap12:operation soapAction="urn:validate" style="document"/> <wsdl:input> <soap12:body use="literal"/> </wsdl:input> <wsdl:output> <soap12:body use="literal"/> </wsdl:output> <wsdl:fault name="Exception"> <soap12:fault use="literal" name="Exception"/> </wsdl:fault> <wsdl:fault name="BaseException"> <soap12:fault use="literal" name="BaseException"/> </wsdl:fault> </wsdl:operation> </wsdl:binding> <wsdl:binding name="ConverterWebServiceHttpBinding" type="ns:ConverterWebServicePortType"> <http:binding verb="POST"/> <wsdl:operation name="convert"> <http:operation location="ConverterWebService/convert"/> <wsdl:input> <mime:content type="text/xml" part="convert"/> </wsdl:input> <wsdl:output> <mime:content type="text/xml" part="convert"/> </wsdl:output> </wsdl:operation> <wsdl:operation name="validate"> <http:operation location="ConverterWebService/validate"/> <wsdl:input> <mime:content type="text/xml" part="validate"/> </wsdl:input> <wsdl:output> <mime:content type="text/xml" part="validate"/> </wsdl:output> </wsdl:operation> </wsdl:binding> <wsdl:service name="ConverterWebService"> <wsdl:port name="ConverterWebServiceHttpSoap11Endpoint" binding="ns:ConverterWebServiceSoap11Binding"> <soap:address location="http://172.22.120.15:8080/ConverterWebService-4.9/services/ConverterWebService.ConverterWebServiceHttpSoap11Endpoint/"/> </wsdl:port> <wsdl:port name="ConverterWebServiceHttpSoap12Endpoint" binding="ns:ConverterWebServiceSoap12Binding"> <soap12:address location="http://172.22.120.15:8080/ConverterWebService-4.9/services/ConverterWebService.ConverterWebServiceHttpSoap12Endpoint/"/> </wsdl:port> <wsdl:port name="ConverterWebServiceHttpEndpoint" binding="ns:ConverterWebServiceHttpBinding"> <http:address


Page 129

location="http://172.22.120.15:8080/ConverterWebService-4.9/services/ConverterWebService.ConverterWebServiceHttpEndpoint/"/> </wsdl:port> </wsdl:service> </wsdl:definitions>


Page 130

14 Annex K: Checks done by the “validate” method of Converter Web Service

14.1 Syntax Validation

The syntax validation is done in a non-recoverable manner by using a standard XML parser.

14.2 Structural Validation

The following recoverable checks are performed against the SDMX message:

Error description Example

Illegal text </message:Header>

BLAH

<message:DataSet >

illegal attribute <message:DataSet dummy="yes"

illegal element <Unexpected a="b"></Unexpected>

series under series <Series ADJUSTMENT="W" FREQ="Q" REF_AREA="IT" STS_INDICATOR="PROD" STS_INSTITUTION="1" STS_ACTIVITY="NS0020" STS_BASE_YEAR="2000" TIME_FORMAT="P1M"> <Series

obs under obs <Obs TIME_PERIOD="2005-08" OBS_VALUE="1.53" OBS_CONF="F" OBS_STATUS="A" > <Obs TIME_PERIOD="2005-12" OBS_VALUE="1.54" OBS_CONF="F" OBS_STATUS="A" >

group under series (2.1 specific)

</Series> <Group xsi:type="ns

Non-flat with flat contents <message:Structure structureID="ESTAT_STS_2_2" namespace="urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:STS(2.2):ObsLevelDim:TIME_PERIOD" dimensionAtObservation="TIME_PERIOD"> … <Obs ADJUSTMENT="W" FREQ="A" REF_AREA="IT" STS_INDICATOR="PROD" STS_INSTITUTION="1" STS_ACTIVITY="NS0040" STS_BASE_YEAR="2000" TIME_FORMAT="P1Y" TIME_PERIOD="2004-02" OBS_VALUE="1.2004" OBS_CONF="F" OBS_STATUS="A" />

Flat file with non-flat contents

<message:Structure structureID="ESTAT_STS_2_2" namespace="urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:STS(2.2):ObsLevelDim:AllDimensions" dimensionAtObservation="AllDimensions"> ... <Series ADJUSTMENT="W" FREQ="A" REF_AREA="IT" STS_INDICATOR="PROD" STS_INSTITUTION="1" STS_ACTIVITY="NS0040" STS_BASE_YEAR="2000" TIME_FORMAT="P1Y">

Invalid XS attachment level. SDMX v2.0 Cross Sectional Specific

<ns1:Section FREQ="A"

Table 4 - Structural Validations


Page 131

14.3 Data Validation

Error description Example

invalid code <Series ADJUSTMENT="W" FREQ="A" REF_AREA="IT" STS_INDICATOR="PROD" STS_INSTITUTION="1" STS_ACTIVITY="NS0040" STS_BASE_YEAR="2000" TIME_FORMAT="P1A">

missing dimension at series <Series ADJUSTMENT="W" REF_AREA="LU" …>

missing mandatory attribute

<Obs TIME_PERIOD="2005-08" OBS_VALUE="1.53" OBS_CONF="F" />

invalid text format <Obs TIME_PERIOD="2004-02" OBS_VALUE="FOO_BAR" OBS_CONF="F" OBS_STATUS="A" />

dim at obs at series/wrong dim at obs

<message:Structure structureID="ESTAT_STS_2_2" namespace="urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:STS(2.2):ObsLevelDim:TIME_PERIOD" dimensionAtObservation="TIME_PERIOD"> … <Series ADJUSTMENT="W" TIME_PERIOD="2004-02" REF_AREA="IT" STS_INDICATOR="PROD" STS_INSTITUTION="1" STS_ACTIVITY="NS0040" STS_BASE_YEAR="2000" TIME_FORMAT="P1Y"> <Obs FREQ="A"

wrong attribute attachment level

<Group xsi:type="ns1:Sibling" REF_AREA="LU" ADJUSTMENT="W" STS_INDICATOR="PROD" STS_ACTIVITY="NS0050" STS_INSTITUTION="1" STS_BASE_YEAR="2000" ... TIME_FORMAT="P1Y"/>

dimension at dataset level <message:DataSet ADJUSTMENT="W"

invalid time period value <Obs TIME_PERIOD="2004-Q5"

missing dimension at group <Group xsi:type="ns1:Sibling" REF_AREA="IT" ADJUSTMENT="W"

duplicate observations <Obs TIME_PERIOD="2004-02" OBS_VALUE="1.2004" OBS_CONF="F" OBS_STATUS="A" /> <Obs TIME_PERIOD="2004-02" OBS_VALUE="14" OBS_CONF="F" OBS_STATUS="A" />

Table 5 - Data validations