IPY Data management

A brief summaryØystein Godøy

IPY Organisation

ICSU/WMOJoint Committee

Data Policy & Management

Subcommittee

ObservationsSubcommittee

Education, Outreach & Communication Subcommittee

ProgrammeOffice

Space TaskGroup

Twelfth WMO Congress, Resolution 40 (1995)

Thirteenth WMO Congress, Resolution 25 (1999)

ICSU General Assembly Resolution (1996)

ICSU Assessment on Scientific Data and Information (2004)

Article III-1c from the Antarctic Treaty (1959)

the Intergovernmental Oceanographic Commission Data Exchange Policy (2003)

International consensus on free and open access

The primary purpose should be to serve the objectives of IPYInternational exchange (>60 countries)Interdisciplinary science (>50,000

participants representing physical, life, and social scientists)

Building a legacyIncrease the value of observations

Overarching purpose

Data used by IPY

Data generated by IPY

Special Cases:• Human subjects• Intellectual property of LTK• Where data release may cause harm

“…the IPY Joint Committee requires that IPY data, including operational data delivered in real time, are made available fully, freely and on the shortest feasible timescale.”

http://www.ipy.org/Subcommittees/final_ipy_data_policy.pdf

IPY Data Policy—Defining IPY Data

Implications

Publish and share your dataIncreased data valuePromote interdisciplinary useHave someone measured something you

need?

Preserve and archive your dataGenerate a legacyThe basis for the next IPY

What has been done?

The Data committee has defined a data policy

Workshop on data management http://classic.ipy.org/international/joint-committee/data-management.h

tm

IPY metadata profile http://ipydis.org/metadata.html

Data coordinators in US, Canada, UK and Norway

Data released for IPY use by ECMWF, ESA and others

Some data management projects have been funded

But not enough...IPY Layer in Google Earth available at

http://ipy.org

IPY data policy - metadata

“All IPY data must be accompanied by a full set of metadata that completely document and describe the data.”

“Regardless of any data access restrictions or delays in delivery of data itself, all IPY projects must promptly provide basic descriptive metadata in an internationally recognized, standard format to an appropriate catalog or registry.”ISO19115/ISO19139

“All data registries and repositories collecting data and metadata from IPY projects are required to collect and share sufficient information to adhere to the IPY Metadata Profile”

Basic who, what, where, when in either FGDC, DIF, THREDDS (ISO coming, but could use some help), plus some information on metadata provenance.

Controlled vocabulary from GCMD for some fields.

The “bare minimum of information necessary to allow simple discovery across disciplines and to ensure we can track the heritage of the metadata in a broadly distributed data management environment.”

Details are available at http://ipydis.org/

IPY Metadata Profile (and crosswalk)

• Entry ID (controlled)• Data set title• Data set progress• Data set summary• Data set citation

information including Online Resource

• Parameters• Locations• ISO topic categories• Temporal coverage• Spatial coverage

• Data center contact information

• Access restrictions• Use constraints• Data Set Language• Metadata contact

information• Metadata authority• Metadata version• Last revision• IPY flag• IPY Project ID

IPY Metadata profile

Formats:• Negotiate common formats and conventions

• ASCII is useful but not really a precise format• avoid proprietary formats • some suggestions: netCDF is popular for some, OGC (WMS/WCS/WFS)

compatibility is nice• Archives and users may have different needs

Access:• Integrate with many systems to allow increased user discovery (register

with the IPYDIS)• Use open source software when possible, use open standards everywhere.

Preservation• Open Archive Information System Reference Model• Attribute and provide info for attribution readily through all gateways

The Data

What is out there?

http://www.ipy-ice-portal.org

ELOKA The Exchange for Local

Observations and Knowledge of the Arctic

works to provide data management and user support to facilitate the collection,

preservation, exchange, and use of local observations and knowledge of the

Arctic.

http://nsidc.org/eloka PI: Shari Gearheard

18 May 2006

http://arcticportal.org

• Assist on compliance with standards, identification of archives, development of the union catalogue, and other data management requirements for IPY.

• Visibly track the data flow for IPY.

• In collaboration with the IPO, develop a data registry that will continue throughout the IPY.

• Survey the planned projects and the data they intend to collect and identify existing archives, portals, experts, and significant gaps in the IPY data infrastructure.

Mark Parsons—Overall, US

Øystein Godøy—Operational Data, Norway

Canadian Coordinator—Overall, Canada

National coordinators in Netherlands, China, UK

Data coordinators

International Operational Data Coordinator (1)

Fist version based upon DAMOCLES systemPostGRES metadatabaseOpeNDAP ⇒ THREDDS (OpeNDAP/WCS

support) or early 2008Adding WMS support for selected

fields/datasetsExpected late 2007

Second version based upon SIMDAT and first versionExpected 2008

International Operational Data Coordinator (2)

Final version should be WIS DCPCExpected 2009

Have focused on ECMWF data so farDirect access at ECMWF disks if using WIS

DCPC software

Interface for metadata submissionAs well as notifications on interesting

datasets

Import/export of metadataMethod and format open issue

International Operational Data Coordinator (3)

How can this service help promoting remote sensing data during IPY?Access to metadata?

Provide metadata information on available products (??)

Users get forwarded to data catalogues at the space centres(??)

OGC access to selected datasets(??)

National recommendations (1)

NorwayNational IPY-projects should relate to the

relevant standards forData collectionProcessingArchivalDescriptionTransferMetadata (ISO19115, ISO19139)OpenGIS

National recommendations (2)

NorwayData and metadata shall be freely

available with minimal time delay

Metadata shall be transmitted to relevant national and international data centres

Data shall be transmitted to relevant national and international data centres for long term archival

Delivery of data and metadata is the responsibility of the project coordinator

National recommendations (3)

NorwayA minimum of standardised file formats

should be used

Existing national and international infrastructure shall be utilised for long term archival

Data used in publications shall be correctly cited

The use of licences will be evaluated if this can improve data access

National requirementsNorway

IPY projects with NFR-contractsshall submit a data management plan within

3 months of contract signingDescribing who is responsible for overall data

managementWho is responsible for datasetsWhere datasets will be stored and finally archived

shall submit yearly updates of the data management planWhat have been collectedWhere is it stored

National IPY-data serviceNorway

DokIPYFunded by NFRShall connect existing metadata- and

databases in NorwayExtract and exchange metadataUtilise existing infrastructure and national

mandates atInstitute of Marine ResearchNorwegian Polar InstituteNorwegian Meteorological Institute

Robust/simple exchange of metadata

METNO version

Relies on the same software as DAMOCLES and International Operational Data Coordination

New feature – metadata exchangeExamining international

requirements/specifications, not much found yet

Funding situation require simple and robust solutions

Special issue

Remote Sensing of EnvironmentSpecial issue on Remote Sensing of the

CryosphereRemote sensing of Environment, Vol. 111,

Issues 2-3, 30 November 2007.Available online

How can we interface?