8 May 2014Update from the Transportation Data Management Working Group Leighton Christiansen, Iowa DOT, Kendra K Levine, ITS Berkeley Library, and Mary Moulton, National Transportation Library

Trans/HackOpening and Preserving Transportation Data Hackathon

Leighton Christiansen, Iowa DOTKenda K. Levine, UC BerkeleyMary Moulton, National Transportation LibraryAmanda J. Wilson, National Transportation Library

#TransHack

International Data Curation Conference 2014San Francisco, February 27

Welcome

We want to thank our sponsor:

#TransHack

Agenda

Section 1: Introduction● Opening and Preserving Transportation Data (45

min)● Break (30 min)

Section 2: Hackathon● Small group/Large group procedural hacking (90

min)● Conclusions (20-30 min)

#TransHack

Who We Are

Leighton Christiansen, Iowa DOT

Kenda K. Levine, UC Berkeley

Mary Moulton, National Transportation Library

Amanda J. Wilson, National Transportation Library

#TransHack

Who Are We?

all images used under claim of educational fair use

http://libraryconnectivity.org/datamgt/index.php/Main_Page#TransHack

http://libraryconnectivity.org/datamgt/index.php/Main_Page

#TransHack

What is Transportation?

A coordinated system made up of multimodal services serving a common purpose, the movement of people and goods.

(Source: AASHTO Glossary) http://trt.trb.org

all images from http://www.trb.org/ used under claim of educational fair use

#TransHack

What is Transportation Data?

Data Types - ● Traffic (volume, flow, speed)● Vehicle (emissions, fuel consumption, sales)● Logistics (routes, commodities)● People (census, choice, modes)● Land-Use

#TransHack

What is Transportation Data?

Data Types ● Sensor data (large databases, exported to

CSV)● Video and image data● Survey data (CSV)● Hand counts● Models and visualizations● GIS

#TransHack

What is Transportation Data?

Research Projects● The SHRP2 Naturalistic Driving Study.

Thousands of hours of video and sensor data.

● PeMS from Caltrans. Traffic data for CA state highways from sensors.

#TransHack

What is Transportation Data?

Research Projects● Bike Count Data Clearinghouse from UCLA

Luskin Center. ● Capital Bikeshare Data includes realtime

system data and surveys.● GTFS transit data for routes and

schedules

#TransHack

Big Bytes

Kilobyte

Megabyt

e

Gigabyt

e

Terabyte

Petabyt

e

Exabyte

Zettabyt

e

#TransHack

1,000

1,000,000

1,000,000,000

1,000,000,000,000

1,000,000,000,000,000

1,000,000,000,000,000,000

1,000,000,000,000,000,000,000

White House OSTP Memo•February 22, 2013 – Office of Science & Technology Policy issues

memorandum entitled “Increasing Access to the Results of Federally Funded Scientific Research.”

•Memorandum addresses new requirements for both intramural and extramural publications and digital data sets resulting from federally-funded scientific research.

•Expand on Data.gov, Open Government requirements already in place•RITA assigned responsibility for preparing response.

#TransHack

USDOT Response

#TransHack

Publications

•Most publications arising from DOT-funded research are already made available to the public (including intramural reports); principal DOT shortfall is availability of peer-reviewed, “scholarly” publications (i.e., journal articles)

•Per MAP-21, NTL is now the federal repository for all Department research information and a clearinghouse for all Government transportation research information

•Plan requires submission of accepted, final manuscript to NTL

§Internal reports will be available through, and preferably stored by, NTL

§To protect publishers, all manuscripts will be embargoed for a period not less than 12 months (prefer 18 months)

§Will require minimal upgrade of NTL software – open source software is preferred meaning minimal additional costs

§All publications, data sets and authors will have unique permanent identifiers for correlation of articles with authors and relevant underlying data

USDOT Response

#TransHack

Data

•DOT makes some, but not all data available

•excludes that which has confidentiality, privacy, proprietary, IP, security, and other exemptions and protections

•Intramural

▪DOT’s Data Release Policy is the governing document for research data management

▪Management will remain with each OA

▪Will require minor updates to OA capabilities, including permanent identifiers

•Extramural

▪Will require submission and DOT approval of data management plan from all awardees

▪Awardees will determine repository for depositing data; MUST be accessible by NTL

▪Data will be included in DOT Enterprise Data Inventory

USDOT Response

#TransHack

Research Project Records

•Goal: to enable linkage of publications and underlying data via the NTL, to enable public access through a simple unique identifier

•DOT will require the submission of project records by DOT funding recipients (intramural and extramural) that fully describe project activities

•Project records will be required to be submitted to the Transportation Research Board’s Research in Progress (RiP) database

•Summary descriptions of any documented project outputs and outcomes resulting from research implementation will be required to be submitted to the USDOT Research Hub

•Project records will be required to be updated over the duration of the project

•DOT will develop step-by-step guidance for the submission process

Data Available from USDOT

NTL Data Catalog http://ntlsearch.bts.gov/repository/ntlc/btsdd/index.shtm● Statistical data sets● Geospatial data● Sensor data● Administrative (e.g. bridge inventories)● Naturalistic study data● Simulations and models

#TransHack

US DOT Data Challenge

Data Innovation www.transportation.gov/datachallenge● Safety: address concerns● Transportation access: how transportation connects people to jobs, school,

housing, community resources● Traffic management and congestion: understand and reduce traffic and

congestion

#TransHack

Roadblocks

Administrative Issues● Many funding sources = many headaches● Many funding sources = many terms of

deliverables● Lack of coordination for tracking compliance

#TransHack

Roadblocks

Legal Issues● Privacy and NDA issues

o Human subject testingo Industry secrets

● Rights and ownership● Ability to license, re-use, and buy/sell

data?

#TransHack

Roadblocks

Open Issues● Interoperability● Data formats● Metadata schema● Data sets are scattered by funder (if at all)● When is “final” dataset made available?

#TransHack

Roadblocks

Funding Issues● Unfunded Mandates● Long-term vs.Short-term funding● Recharge model and pricing● Cost of infrastructure and overhead

#TransHack

Next Steps

● Continuing education plan● Continued development of TransDM wiki● Presentation for TRB State Representatives

Meeting (with AASHTO RAC), July 2014● Webinar for CUTC, Fall 2014

#TransHack