Seminar on Emerging Trends in Data Communication and Dissemination

Statistical Data as a Service and Internet Mashupsby Zoltan Nagy

19 February 2010

What do we have today?

Statistical organizations became competent and capable content providers and users of the Internet. Most National Statistical Offices and International Agencies have their websites with static or dynamic content and with interactive databases or downloadable datasets.

With the current approach we are distributing statistical data as goods but are there other ways to provide access to statistical data?

Providing Data as a Service

Data as Goods(bottled water)

Data as Service(plumbing)

Bulk onetime download Dynamic access

Dated with the time of download

Always latest update

Need for storage Storage is provided

Analysts, researchers, data enthusiasts

Dynamic content providers, mashup creators

What is a Mashup?

• A web application that A web application that combines data from more combines data from more than one source into a than one source into a single integrated toolsingle integrated tool

an example is the use of an example is the use of data from Google Maps data from Google Maps to add location to add location information to statistical information to statistical data, thereby creating a data, thereby creating a new and distinct web new and distinct web service that was not service that was not originally provided by originally provided by either sourceeither source

What are the benefits mashups?

- Creation of new dynamic user experience- Gain valuable insights through information remix- Further promotion of our services and data- Minimized application data management- Reduced development effort- Get results faster by accessing information in

place- Ability to quickly assemble applications for new

situations

How a mashup works?

User

Mashup website

User Request

Website 1 Website 2

API Call Data API Call Data

Data presentation

Data Manipulation

APIs and web services

- API is an abbreviation for Application Program Interface, a set of routines, protocols, and tools for building software applications. A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.

- Web services today are frequently just Application Programming Interfaces (API) or web APIs that can be accessed over a network, such as the Internet, and executed on a remote system hosting the requested services.

How to plan for a mashup?

• Pick a subject A Mashup of What?

Map + Statistical data? Google Fusion tables for transformation + Statistical data?

More sources of data More complicated

• Decide your data sources Who is your data provider?

Google maps, Bing maps, etc.. Online data UNdata, Comtrade, Dallas etc..

Usually language agnostic Varying complexity

• Other concerns How much time do you have? Do you have a server to run it on? Which programming language?

Data dissemination – UNData

UNdata is unique initiative of the Statistics Division to bring statistical information together from various international and national sources and present it in an easily understandable and accessible format.

• Make UN databases freely available• Organize international databases to allow searchability and

open access

• Promote national data dissemination

• Build a global data dissemination infrastructure

Data dissemination – UNdata

Data dissemination – Undata

Data dissemination – UNdata

UNdata API

UNDATA API project – an API version of the great data made available by the United Nations on the UNDATA site. The aim is to make this data accessible and reusable in a variety or ways so it can be easily mashed up and recombined into new applications or analysis.

Mashups with UNdata

Mashups with UNdata

Data dissemination – Comtrade

• Data from over 150 countries processed into a standard format

• Data by partner country/commodity from 1962 covering about 90% of the world trade

• 1.5 billion statistical records, 0.5 Terabyte of data

• Free Web access to any record and paid subscription for use of download services

• 6 billion records downloaded since June 2003

Data dissemination – Comtrade API

Web Server Database Server

Comtrade Web Services

United Nations

Comtrade

Tariff Line

Total Trade

Organization #1

PC XML Files

Organization #3

Database Server

Comtrade Tools

Other Tools

PC

Organization #2

Comtrade Tools

Other Tools

XML Files

Text Files

Internet

Element based XML

SDMX-ML

Http protocol

Http protocol

Trade Data Transfer Architecture