Restricted

Views expressed are those of the presenter and not necessarily those of the BIS1

Building a Data Portal with SDMXThe BIS SDMX Sandbox exercise

1

Gabriele Becker, Massimo BruschiBank for International Settlements

METIS, 7 May 2013

Restricted 2

The SDMX vision Users need …

• good quality data, up-to-date numbers, documentation• Single access point for data from different sources• User interfaces: GUI and also a Web service (for

automation) The SDMX vision …

• Data providers (originators) offer their data “in SDMX”• Dissemination = reporting = data sharing … from a single

storage• SDMX registries help users to find data via a GUI and an

SDMX Webservice The BIS SDMX Sandbox exercise

• How “real” is this SDMX vision?• How does it work with difference scenarios?

Restricted 3

The Sandbox setup “in the cloud” 14 participating central banks: 14 Sandboxes with

SDMX registry with user interface and SDMX web service

SDMX data base with data browser and SDMX web service

Space to host SDMX data files User interface for loading files, performing registrations

etc. Detailed tutorials

1 Sandbox for the BIS to experiment and participate “as central bank”

The Data Portal (Unified Data Catalogue or “UDC”) “central” SDMX registry operated by the BIS Data browser Synchronises with the central and CB registries Accesses registered data sources

SDMX Sandbox was developed by Metadata Technology

Restricted

SDMX Sandbox setup 14 Sandboxes for central banks, 1 for BIS as central agency, 1 Data portal (Unified Data Catalogue)

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Data Portal: UDC

SDMX Sandbox Country A

CB SDMX Registy

Data browser

CB SDMXDatabase

BIS SDMXSandbox

Data browser

SDMX Registy

SDMXDatabase

UDC SDMX Registy

Unified Data CatalogueBrowser

4

Restricted

Scenarios1. Data reporting against a “global” SDMX DSD, eg

BOP2. Making internal data available via a “data portal” 3. Bringing external (public) data to your users via a

data portal Common aspects

data is available in SDMX files in a known DSD or …

SDMX web service offers data in a known DSD We just need to know where the data is and

what DSD is follows An SDMX Registry is the place for this

information A simple data browser can offer data selection

based on the known DSDs5

Restricted 6

ECB publishes a lot of data as SDMX files on its website

IMF and BIS offer the Joint Ext. Debt Hub (JEDH) contributions as SDMX files

Users may want to see this data via a common data portal

Organisations may wish to access this data via a SDMX 2.1 web service (that ECB, IMF and BIS currently do NOT offer) demo

Load SDMX DSD into registry demo

• In future: SDMX registries provide this as a service … Define Dataflow, Provision agreement and Categorisation Register the data files as sources for the UDC demo

Scenario 3: “making public data available in a data portal”

Restricted

Demo Movie

7

Restricted 8

Restricted 9

Conclusions I Building blocks are working in the SDMX Sandbox

SDMX registry and SDMX 2.1 webservice SDMX data base (with webservice) Data browser Unified Data Catalogue (data portal)

Connecting different data sources … SDMX files from public websites SDMX files from other (accessible) locations Different SDMX versions (1.0, 2.0, 2.1) SDMX databases (internal and external) SDMX Web service

First implementation of a federated SDMX registry

Restricted 10

Conclusions II

Harmonisation of data access techniques was achieved… User sees all data through one “data portal” (the

UDC) SDMX 2.1 web service for all data, also for those

offered as SDMX 1.0 files Web service supports automated data retrieval

Works best with harmonised data structures! BOP Sandbox example Key deliverables for SDMX: DSD for global use for

BOP and National Accounts … under way BOP Sandbox created by IMF based on this exercise

The SDMX Vision is real!

Restricted 11

Thank you !

Restricted

Background slides

Scenario 1 and 2

12

Restricted 13

Scenario I: “BOP” (Balance of payments) National agencies have to “report” their

data based on a commonly agreed data structure

This data needs to be available at international organisations and for the public

This is a real life scenario for BOP, National Accounts and other data domains. demo

SDMX BOP data structure “for global use” is a prerequisite• We invented a simplified one for the purpose of this

exercise …

Restricted 14

BIS SDMXSandbox

SDMX Sandbox : BOP example

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Another CB Sandbox

Data Portal: UDC

Unified Data CatalogueUDC SDMX Registy

SDMX Sandbox Country A

Data browser

SDMXRegisty

Data browser

SDMXDatabase

SDMX-ML file with Central bank BOP

data

Register file as data source

UDC user can see BOP data from all participating central banks

Uses UDC

Retrieve data from DB or file

Registry sychronisation

Registry synchronisation

SDMX-ML file with Central bank BOP

data

CB data provider

Store on Web location

BOP DSD

BOP data flows, provision

agreements, data source, registrations

CB SDMX Registy

BOP DSD

CB SDMXDatabase

Central bank BOP

data

Import

Register asqueryable

data source

RetrieveDSD info

for browsing,registrations

for datalocation

BOP DSD

CB DBadmin

provider action user action

System interaction

· Data manager registers data file in UDC Registry

· SDMX data file is uploaded to a Web location and then registered as data source against the BOP provision agreement

· It’s immediately visible to a UDC user· Date manager imports data in the

Sandbox DB· It’s immediately visible to a user of the

Sandbox data browser

Restricted 15

Scenario 2: “dissemination database for BIS” BIS wishes to disseminate data to

internal users via a common data portal (example: BIS Banking Statistics)

BIS has/defines SDMX DSD for this data demo

BIS uses the SDMX database in the Sandbox as “dissemination database” and loads the data via SDMX files demo

Internal users use the data browser (similar to UDC) to access the “BIS SDMX database”. demo

For external dissemination the BIS registers its Sandbox database as a data source to the UDC demo

This also enables the UDC search service for this data demo

Restricted

UDC: Data Portal

BIS data provider

SDMX-ML file data for DSD

SDMX Sandbox : dissemination data base example

UDC user

BIS SDMX Sandbox

UDC Registryy

DSD, Data flows, Provision agreements

and data source registrations

Retrieves DSD info for browsing

and registrationsfor data location

SDMXDatabase

with SDMX Webservice

Register asqueryable

data source

Uses UDC

Load

Retrieves data from SDMX database

Registrysynchronisation

provider action user action

SDMX Registry

DSD, Data flows, Provision agreements

and data source registrations

System interaction

BIS data user

Data browser

Use data browser

Retrieves data from SDMX databaseCreate DSD,

data flow etc.

UDC Browser

SDMX-ML file data for DSD

Store on Web location

Register as queryable data source

· Data manager creates or loads a DSD into the registry

· defines data flow and provision agreement

· loads a data file in the SDMX Database. · It is immediately visible to a user of the

Data browser attached to the Sandbox.· Data manager registers SDMX DB as

data source for the UDC.· UDC users can see the data

16