Seminar on Emerging Trends in Data Communication and Dissemination
Statistical Data as a Service and Internet Mashupsby Zoltan Nagy
19 February 2010
What do we have today?
Statistical organizations became competent and capable content providers and users of the Internet. Most National Statistical Offices and International Agencies have their websites with static or dynamic content and with interactive databases or downloadable datasets.
With the current approach we are distributing statistical data as goods but are there other ways to provide access to statistical data?
Providing Data as a Service
Data as Goods(bottled water)
Data as Service(plumbing)
Bulk onetime download Dynamic access
Dated with the time of download
Always latest update
Need for storage Storage is provided
Analysts, researchers, data enthusiasts
Dynamic content providers, mashup creators
What is a Mashup?
• A web application that A web application that combines data from more combines data from more than one source into a than one source into a single integrated toolsingle integrated tool
an example is the use of an example is the use of data from Google Maps data from Google Maps to add location to add location information to statistical information to statistical data, thereby creating a data, thereby creating a new and distinct web new and distinct web service that was not service that was not originally provided by originally provided by either sourceeither source
What are the benefits mashups?
- Creation of new dynamic user experience- Gain valuable insights through information remix- Further promotion of our services and data- Minimized application data management- Reduced development effort- Get results faster by accessing information in
place- Ability to quickly assemble applications for new
situations
How a mashup works?
User
Mashup website
User Request
Website 1 Website 2
API Call Data API Call Data
Data presentation
Data Manipulation
APIs and web services
- API is an abbreviation for Application Program Interface, a set of routines, protocols, and tools for building software applications. A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.
- Web services today are frequently just Application Programming Interfaces (API) or web APIs that can be accessed over a network, such as the Internet, and executed on a remote system hosting the requested services.
How to plan for a mashup?
• Pick a subject A Mashup of What?
Map + Statistical data? Google Fusion tables for transformation + Statistical data?
More sources of data More complicated
• Decide your data sources Who is your data provider?
Google maps, Bing maps, etc.. Online data UNdata, Comtrade, Dallas etc..
Usually language agnostic Varying complexity
• Other concerns How much time do you have? Do you have a server to run it on? Which programming language?
Data dissemination – UNData
UNdata is unique initiative of the Statistics Division to bring statistical information together from various international and national sources and present it in an easily understandable and accessible format.
• Make UN databases freely available• Organize international databases to allow searchability and
open access
• Promote national data dissemination
• Build a global data dissemination infrastructure
Data dissemination – UNdata
Data dissemination – Undata
Data dissemination – UNdata
UNdata API
UNDATA API project – an API version of the great data made available by the United Nations on the UNDATA site. The aim is to make this data accessible and reusable in a variety or ways so it can be easily mashed up and recombined into new applications or analysis.
Mashups with UNdata
Mashups with UNdata
Data dissemination – Comtrade
• Data from over 150 countries processed into a standard format
• Data by partner country/commodity from 1962 covering about 90% of the world trade
• 1.5 billion statistical records, 0.5 Terabyte of data
• Free Web access to any record and paid subscription for use of download services
• 6 billion records downloaded since June 2003
Data dissemination – Comtrade API
Web Server Database Server
Comtrade Web Services
United Nations
Comtrade
Tariff Line
Total Trade
Organization #1
PC XML Files
Organization #3
Database Server
Comtrade Tools
Other Tools
PC
Organization #2
Comtrade Tools
Other Tools
XML Files
Text Files
Internet
Element based XML
SDMX-ML
Http protocol
Http protocol
Trade Data Transfer Architecture