U.S. Department of the InteriorU.S. Geological Survey

Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science CenterEllyn MontgomeryUSGSNERACOOS/NECOSP Data Management Workshop, Sept. 26, 2012

NECODP 2012

This presentation applies only to Oceanographic measurements acquired by Woods Hole researchers and colleagues to address scientific questions

The data is mostly used for validation of sediment transport and coastal process models

Management of the data is part of the larger Data Life-cycle that includes everything from planning to archival and publication

USGS Coastal and Marine Geology Program

NECODP 2012

Our holdings describe conditions at specific locations for limited durations. Except for a 25+ year program in Massachusetts Bay, we don’t do long term monitoring programs.

Once the data files collected during an experiment are processed and QC’d that data is published (Federal requirement)

Updates are made only if a problem is found in the data to correct that problem

About our data

NECODP 2012

The data are stored in netCDF format files using EPIC conventions

NcML is used to convert to CF conventions Files are delivered on-line via

file download OPeNDAP using THREDDS

FGDC metadata for each experiment is being created and harvested to ocean.data.gov

Each file has ISO 19115 metadata (ncISO)

About our data II

NECODP 2012

Our raw data is included in the WHSC field activity tracking system

We store 3 levels of data on local disks that are backed up nightly, with periodic backups made and stored off-site. These levels are : Raw data as off-loaded from the instruments Processed data (and stages leading to it) Published data

As EPIC As CF

How we manage our data - 1

NECODP 2012

Our data may be published directly to the website after approval or with an Open File Report that documents the context of the measurements.

We periodically check the website for broken links or non-responsiveness

We add new interfaces as available (for example making KML files to use with GoogleEarth)

Publication and access

NECODP 2012

Web access

NECODP 2012

Web access

NECODP 2012

THREDDS

NECODP 2012

NetCDF allows metadata to be stored as attributes with the data in the file

We use standards and conventions to enable interoperability

We have GCMD DIF, FGDC and ISO19115 metadata in as many catalogs and registries as possible

We enhance the metadata as new standards emerge (added Unidata Dataset Discovery terms in 2011)

Metadata

NECODP 2012

Community Standards development has evolved to enable LOT of great functionality

We are happy with how THREDDS allows remote clients to access data for analysis in real-time, without having to down-load files

Clients like Matlab, IDV, ncBrowse and ncView allow access and visualization

Registering our THREDDS catalogs in larger repositories of ocean measurements enhances discoverability

Data Management Successes

NECODP 2012

Getting links to our holdings into the catalogs of the regional observatories Since our data is historical, it hasn’t really fit the

observatory mission, but including links to historical measurements in the area would be useful for enriching potential analyses

Identifying which of the emerging technologies will persist into the future

Finding ways to let potential users know the data is available

Data Management Challenges

NECODP 2012

Use standards to enhance discoverability and enable use of community developed tools

Enable web-based access so the users can access what they need without assistance

Tailor the process to what your clients need to accomplish their missions

Adapt your data and services to take advantage of new techniques as they evolve

Take home ideas

NECODP 2012

The human-friendly interface to our experimental data, (EPIC conventions) is at: http://stellwagen.er.usgs.gov

The computer-friendly (CF conventions) version of our data holdings, served by THREDDS at: http://geoport.whoi.edu/thredds/allts_reg_catalog.html

Contact me at: emontgomery@usgs.gov (508)457-2356

Web access