Open Data Trentino presented at the European Commission (JRC)

10/04/231 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - [email protected]

http://dati.trentino.it/

Open Government Data*

*Part of this presentation is taken from the “Open Government Data Tutorial” gave at CLEI2013 Conference by Lorenzino Vaccari and Juan Pane (Universidad Nacional de Asuncion, Paraguay)

Lorenzino Vaccari

Autonomous Province of Trento, Trento, Italy [email protected]


In this presentation…• Introduce Open Government Data

• Intro (Part 1)• Issues (Part 2)

• If you need it, how can you organize it?• Real experience (Part 3)

• Reusing open data• Applications (Part 4)• Semantic layer (Part 5)

10/04/233 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - [email protected] 15/10/2013Juan Pane, Lorenzino Vaccari3http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

http://www.point-fort.com/index.php?2012/01/25/805-why-how-what


What?

“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and

sharealike.” *

*(Source: )

http://www.opendefinition.org


usereuse

“open” = redistributioncommercial reusederivative works

BUT, may require:- attribution- share alike

http://myfbcovers.com/uploads/covers/2012/06/09/16628a1094aa012f7c6e0025902480d2/watermarked_cover.jpg

J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how


The value is in its use


Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q


New visualizations


http://wheredoesmymoneygo.org/


New visualizations

13/11/20139J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how

http://openspending.org


Why The Open data are the knowledge base to:

Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information

Answer to social needs through the publication of innovative services and applications

Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)

Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities



How - PrinciplesTim Berners-Lee (5-Stars of Linked Open Data)vs.Tim Davis (5-Stars of Open Data Engagement)vs.OGD: Ten principles for opening up government information…

http://sunlightfoundation.com/policy/documents/ten-open-data-principles/

http://5stardata.info/

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/


5 Stars Linked Open DataTim Berners-Lee

http://5stardata.info


Create Communityhttp://msnbcmedia.msn.com/j/MSNBC/Components/Photo/_new/pb-121007-spain-tarragona-pyramid-nj-02.photoblog900.jpg

5-Stars of Open Data Engagement

Tim Davis


Open Government Data: Ten principles for opening up government information1. Completeness

2. Primacy (primary source)

3. Timeliness

4. Ease of Physical and Electronic

Access

5. Machine readability

6. Non-discrimination

7. Use of Commonly Owned

Standards

8. Licensing

9. Permanence

10. Usage Costs


State of the ArtWhat is happening around us?• Globally• Europe• Italy


Open Data Charter - G8The principles are: Open Data by Default Quality and Quantity Useable by All Releasing Data for Improved Governance Releasing Data for Innovation

http://opensource.com/government/13/7/open-data-charter-g8

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex



http://census.okfn.org/

Open Data Census (OKF)



http://census.okfn.org/country/



http://census.okfn.org/

Open Data Barometer (ODI)


OGD in Europe

screenshots

http://epsiplatform.eu/content/european-psi-scoreboard


OGD in EuropeInsert table

http://epsiplatform.eu/content/european-psi-scoreboard http://epsiplatform.eu/content/psi-scoreboard-indicator-list


http://open-data.europa.eu/


OGD in Italy

http://www.dati.gov.it/content/infografica


OGD: Part 2 - Issues

10/04/2325 Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - [email protected] 08/10/2013Juan Pane, Lorenzino Vaccari25http://evian-thesource.com/kids-having-fun/http://evian-thesource.com/kids-having-fun/


Open Data. Oh ohh

08/10/2013Juan Pane, Lorenzino Vaccari26

LegalLegalOrganizationalOrganizational TechnicalTechnicalAdoptionAdoptionBarriersBarriers

ContextualContextual

http://www.wallpapermania.eu/wallpaper/trick-or-treat-cute-pumpkins-lanterns-halloween-wallpaper


http://de.straba.us/wp-content/uploads/2012/08/barrieres_for_implementation_of_ogd.png


Organizational Barriers

• Not ready• Lack of resources

• IT• Human

• Don’t want to be ready

http://montcomediation.org/images/MCMC_MyWayYourWay.jpg


Legal barriersOpen the Data

All the data that was produced using public money has to be made publicly available (with exceptions)

vs PrivacyYou cannot open data that could allow

correlation of private personal data

Or the complete lack of legislation!


Adoption barriersData is not contextualizedPeople are not informedOpening data is a complex task, opening cleaned

data is even more complex.Unclear licenses


Technical BarriersAccess to data:

OrganizationalTechnical, Downtimes, logins, Payment fees

Fragmentation, incomplete data, scattered

FormatCataloging, indexing, searchLack of explicit semantics, metadataData is not reliableConflicting standards, models,

ontologies


BarriersZuiderwijk et al 2010

Listed 118 socio-technical impediments for opening data in the literature.FindabilityUsabilityUnderstandablityQualityLinkingComparability and compatibilityMetadata….

http://www.ejeg.com/issue/download.html?idArticle=255


Context Barriers

Privileged access to dataOther companies what to avoid legislation of

privacy.Transparency is bad for fraudulent business

http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg


http://netdna.webdesignerdepot.com/uploads/photo_manipulation/manipulation-9.jpg


Part 3 - Real Experience


Our story started with GeoData…

http://www.territorio.provincia.tn.it


5 Stars Linked Geo Data Catalog

DBpedia TrentinoGeoData Freebase


The “Open Data in Trentino” project

• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model

• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity


Workplan – Best practices Not only a Project, but also a “Change management process”

Best Practices:- Guidelines (metadata, formats, licences)- Point of contact (domain, operator)- ONE dataset each provider- Community Building- Distributed catalog- Clear Licences- Enterprises- Courses- Contest


Nome (Acronimo) Descrizione

Tipo di Dato Estensione del file

Comma Separated Value (CSV) Formato testuale per l'interscambio testuale di tabelle, le cui righe corrispondono a linee e i cui valori delle singole colonne sono separati da una virgola (o punto e virgola)

Dato tabellare .csv

Geographic Markup Language (GML) Formato XML utile allo scambio di dati territoriali di tipo vettoriale

Dato geografico vettoriale

.gml

Keyhole Markup Language (KML) Formato basato su XML creato per gestire dati territoriali in tre dimensioni nei programmi Google Earth, Google Maps


.kml

Open Document Format (ODF) Formato per l'archiviazione e lo scambio di documenti di testo, fogli di calcolo, diagrammi e presentazioni

Dato tabellare .odc

Resource Description Framework (RDF) Basato su XML, e' lo strumento base proposto da World Wide Web Consortium (W3C) per la codifica, lo scambio e il riutilizzo di metadati strutturati e consente l'interoperabilità tra applicazioni che si scambiano informazioni sul Web

Dato strutturato .rdf

ESRI Shapefile (SHP) Lo Shapefile ESRI è un popolare formato vettoriale per sistemi informativi geografici. Il dato geografico viene distribuito normalmente attraverso tre o quattro files (se indicato il sistema di riferimento delle coordinate). Il formato è stato rilasciato da ESRI come formato (quasi) aperto


.shp, .shx, .dbf,

.prj

Extensible Markup Language (XML) E' un formato di markup, ovvero basato su un meccanismo che consente di definire e controllare il significato degli elementi contenuti in un documento o in un testo attraverso delle etichette (markup)

Dato strutturato .xml

Guidelines


…MeteoMeteo GeoDatiGeoDati StatisticaStatistica Comune

TrentoComuneTrento TrasportiTrasporti Etc…Etc……

Tecnological platform


Data Sources


Data Sources Plan

Novembre

Dicembre Gennaio

Catasto # 18

SGC CSW #9

10 20 30 302010 10 20 30

Attività Culturalii #59

Servizio Istruzione #57Attività Form #58

Dati Energia #30

Dati Progettone #63

Dati Motorizzazione #72

Elettorali #35

Gestione Strade #16

Bilancio PAT #37, 38

PersonalePAT #41

Turismo STU #53Idrometr

ici#26

Trentino Cultura #32

Ufficio Rifiuti #34

Servzio Europa #56

Aff FinanziariConsuienze #36

Min. Linguistiche #48

Pub. Esercizi #49

Imp Funivie #50

Immigrazione #52

Sovr. Beni Arch #60Dati Scuola #61

Agenzie Forestali #64Incendi #65

Cinformi Stranieri #66

Imp Depurazione #68Opere Civili #69

Dati Traffico Stra #70

Gestioni Patrimonialii #71

Dati SAT #28

Dati Cons. Prov #3

Trasporti 2.0 #6

OsservatorioLavori Pubb #17

Comune Trento update

Dati Cons. Prov #3

Dati SAT #28

Dati Cons. Prov #3

Dati SAT #28

Dati Cons. Prov #3


Catalog

The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.

(2004)

http://okfn.org


http://dati.trentino.it*

Analysis: http://dati.trentino.it/stats Admin: http://dati.trentino.it/admin Harvesting: http://dati.trentino.it/harvest

* Available for all the data providers of Trentino


Services


Legal Issues

Permissions: share, create, adapt

Actual interoperability!

Constraints: nothing!

http://www.hoax-slayer.com/images/privacy.jpghttp://www.destateparks.com/images/general_info/privacy_policy.jpg

Permissions: share, create, adapt

Actual interoperability!

Constraints: nothing!


Organizational Issues - Macro


Organizational Issues - Micro


Community buildingMunicipalities“Consorzio dei

Comuni”

Municipalities“Consorzio dei

Comuni”

“Comunità di Valle”

of Trentino

“Comunità di Valle”

of Trentino

Private Companies

Private Companies CitizensCitizens

Educational Institutes

Educational Institutes

Research InstitutesResearch Institutes


International Community


Also Trentino is going to launch a challenge to build software applications and creative products (multimedia, audiovisual products, posters, illustrations) based on the datasets published on the http://dati.trentino.it open data catalog.

#ODTChallenge will be the official hashtag for our first open data challenge in Trentino!



8 months until now68.555 visits 7.988 unique visits2.516 downloads

37,36% returning visitors

62,64% new visitors

NOW- ALL the departmnets demand to be involved- Plus other local actors

AgricultureCultureGeographical DataWelfareWeather ForecastSocial policiesStatisticsTransports…MUNICIPALITY OF TRENTO, and

INFORMATICA TRENTINA

580 datasetsprovided by 10 departments of PAT…

20 reporting errors15 asking for new data10 new suggestions6 OD Applications

100% ENTHUSIASTIC REACTIONS


Want to Know more? A couple of links


http://www.theodi.org/


http://schoolofdata.org/


http://schoolofdata.org/online-resources/


OGD: Part 4 - Applications


Apps4Italy


Best Application: http://parlamento17.openpolis.it/


Open Bilancio

Best Idea: http://opendata.comune.fi.it/open_bilancio/


http://limaio.innovacion.pe/ http://www.limaio.com/demo

Open Source, Open Data, Open Hardware


http://www.mysociety.org/2007/more-travel-maps/morehousing


Johann MITTHEISZ (CIO der Stadt Wien)

http://www.slideshare.net/BrigitteLutz/keynote-mittheisz-cio-stadt-wien/16

Total hours to develop 38 applications:around 2.600

City of Wien saved around 208.000 Euro


Beyond Data (The OpenStreetMap Case)


OpenStreetMap

~

OpenStreetMap project creates and provides geographical data, such as road maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use or availability of map information across much of the world and the advent of inexpensive portable satellite navigation devices.

OpenStreetMap is a free map of theworld, created by someone like you


http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=11.12042&lat=46.07224&zoom=18


Watercolor maps

http://content.stamen.com/files/cartography/index_watercolor.html#18.00/46.07204/11.12097


From maps to blankets…

http://softcities.net


Sharing Data Globally(the eHabitat example)


The Group of Earth Observation

Source: http://www.slideshare.net/angeled/geoss © GEO secretariat84 GEO members and 61 Participating organizations


GEOSS Data Sharing Principles • Full and Open

Exchange of Data, recognizing Relevant International Instruments and National Policies

• Data and Products at Minimum Time delay and Minimum Cost

• Free of Charge or minimal Cost for Research and Education

http://www.geoportal.org/web/guest/geo_home


GEOSS for biodiversity

http://www.eurogeoss-broker.eu/


The eHabitat Model

http://ehabitat-wps.jrc.ec.europa.eu/ehabitat/


OGD: Part 5 – Semantic Layer


Available

Structured

Open formats

Redefenceable

Linked

Linked Open Data

The best data is an open data

Vs.

All data must be perfect


Lack of explicit semanticsThe real meaning of the data was kept in the developers mind when creating the data

78http://goo.gl/npEHKr


Lack of explicit semanticsCan lead to things like:


Semantic heterogeneityDifference in the meaning of local data


Available

Structured

Open formats

Redefenceable

Linked

Data Catalog

Data Catalog

Entity centric

Importing tool

Entity centric

Importing tool


Entity centric: Added valueAggregated dataAccurate data, manually curatedUnique identifiers, distributed perspectives

Re-think identifiersSemantified values

E1

name Juan Pane

nationality italian

lives in Trento

affiliation Univ. Trento

E2

name Ignacio P. F.

born in Paraguay

date of birth 1980

affiliation PF-UNA


EntitiesReal world: is something that has a distinct,

separate existence, although it need not be a material (physical) existence. Has a set of properties, which evolve over time. Example:

Mental: personal (local) model created and maintained by a person that references and describes a real world entity.

Digital: capture the semantics of real world entities, provided by people.


Entity based Semantic Layer:• Address the integration problems due to

semantic heterogeneity:• Different formats• Different identifiers• Implicit semantics• Homonyms, synonyms, aliases• Partial knowledge• Knowledge evolution

http://www.webfoundation.org/2011/11/5-star-open-data-initiatives/


ImportingTool

ImportingTool

The semantic Layer: why?

ImportingTool

ImportingTool

ImportingTool

ImportingTool

REST/HTTPREST/HTTP

i i+1v0

Applications use entities instead of raw data


Importing steps

Selection

Schema Matching

Data Validation

Semantic Enrichment

Reconciliation

Exporting

Publishing

Visualization

1.

2.

3.

4.

5.

6.

7.

8.

Take raw data from dati.trentino.it

Cleanse data

Map to an EntityType

Link data to entities/concepts

Update/insert entities

Export to Entitypedia

Publish to dati.trentino.it

Get insights about entities


1. Source SelectionImport one data file at a time


2. Schema MatchingSelect a target type of entity -> correspondences between the input columns and the output attributes

nome provincia descrizione funivie lat long

Andalo (1047) Provincia di Trento

Sorge su un'ampia sella prativa al centro...

3 654463 712857

Canazei (1450) Trento Prov. Situato all'estremità settentrionale della...

2 511504 147444


3. Data ValidationApplies format and structure validation and possible automatic transformations needed to have the input data in the expected format.


4. Semantic Enrichment (1/2)Entity disambiguation: Transform text references into links to existing entities.


4. Semantic Enrichment (2/2)Natural Language Processing: Extract concepts and entity references from free-text.


5. ReconciliationRun Identity Management Algorithms to identify each row as a new or existing entity.

Result•No Match•Match•Multiple Matches

Action:•Use ID•New ID•Ignore Row


6. ExportingAt this point:We know what to export.All values for target attributes conform to the expected format.All text has been semantified (NLP).All textual references to entities are converted to linksEach row has an identifier

i i+1v0


7. PublishingPut back the semantified entities into CKAN so that

the entities can be Open Data and can be found in the same catalog as the original data.

Developers and find the data files of the cleaned, aggregated entities

But can also interact with the entities via the Entitypedia APIs

8. VisualizationSearch and Navigation


Our Goal

TN

UK

BEES


http://www.youtube.com/watch?v=Bq_ZWl1ZXA0

BEYOND


Thanks to all the Open Data in Trentino Team and in particular to:Juan Pane, Maurizio Napolitano, Marco Combetto, Moaz Reyad and Luca Paolazzi

Technology

Open Data Trentino presented at the European Commission (JRC)