Developing an IOOS Data Management Plan that

Satisfies NOAA’s Requirements and IOOS

Certification Requirements

John Ten Hoeve

9/11/13

Agenda• Discuss NOAA requirements for DMPs (10 min)

• Discuss DMPs in the context of IOOS Certification (10 min)

• Introduce the IOOS DMP Template (5 min)

• Have groups sketch out responses to DMP items (15 min)

• Reconvene and discuss issues/concerns (20 min)

• Development of Coordinated QC Software Effort (15 min)

• Wrap up (5 min)

Data Management Plan Overview• A Data Management Plan (DMP) is a document that

describes how data will be collected, documented, made accessible, and preserved for future use.

– IOOS Certification requires supplementary information about quality control (QC) and the RICE’s DMAC team

• DMPs are required by NOAA and IOOS– NOAA now requires that all data production projects have

a data management plan on file with NOAA EDMC.– IOOS Certification guidelines require RICEs develop Data

Management Plans.– RICE DMPs will satisfy requirements of Data Sharing by

NOAA Grantees policy

NOAA Requirements

• NOAA Administrative Order 212-15 states that environmental data is to be managed based upon a lifecycle that includes a Data Management Plan.

• The NOAA Data Management Planning Procedural Directive states that managers of all environmental data projects shall write a DMP, and that the plan is to be followed and maintained throughout the full lifecycle of the data.

• Authority for the Directive is provided by the NOAA Environmental Data Management Committee (EDMC)

IOOS Certification

• For the purposes of determining liability “arising from the dissemination and use of observation data” a certified RICE is considered part of NOAA

IOOS Certification Requirements• The RICE shall develop a DMAC Plan that:

1. Identifies individual responsible for data management and describes qualifications

2. Identifies procedures for evaluating the capability of this individual

3. Describes the lifecycle of the data created/captured by the RICE

4. Describes the quality control procedures that are applied

5. Describes how the RICE will make its data accessible using IOOS protocols

6. Describes how the RICE will implement data management protocols promulgated by the IOOC and the IOOS Program Office.

7. Documents the RICE’s data archiving process

• Review and approval of RICE DMPs will be conducted by the IOOS Program Office

– Consultation w/ EDMC or QARTOD Board as needed

IOOS Certification Standards• Quality Control:

– For each dataset disseminated by the RICE, the RICE shall ensure the data is quality controlled

– For variables with documented QC procedures (e.g. QARTOD), these algorithms/standards must be used.

– The RICE may publicly document/reference quality control procedures upstream if the procedures meet community best practices (e.g. non-Federal data from NDBC). LINK

• Long-term Archiving:

– The RICE will provide a Submission Information Form, or a description of a future SIF the RICE intends to create.

– The SIF should strive to cover all datasets that are at risk of being not archived elsewhere.

IOOS Certification Standards Cont.• Lifecycle Documentation:

– The RICE must document the data lifecycle of each dataset (via text or a schematic) which includes inputs, processes, transformations, modifications, storage/databases, and outputs.

– DMP may be hierarchical, with elements common to multiple datasets factored out to a higher level

• Data Sharing:

– The DMP will satisfy NOAA’s Data Sharing Policy for Grants and Cooperative Agreements.

• Include DMP in grant proposal (will serve as "data sharing plan")

• Provide access to data as described in the DMP

Introduce IOOS DMP Template• Adjusted generic NOAA template to be IOOS-specific

• 39 items total

• 35 items required by NOAA

• 5 common NOAA / certification items

• 4 items added by IOOS per certification

• Main sections1. General Description of Data to be Managed

2. Points of Contact 6. Initial Data Storage and Protection

3. Data Stewardship 7. Long-Term Archiving / Preservation

4. Data Documentation

5. Data Sharing

Useful Links• IOOS DMP Template (https://code.google.com/p/ioostech/downloads)

• NOAA GEO-IDE Wiki DMP Page (https://geo-ide.noaa.gov/wiki/index.php?title=Category:Data_Management_Plans)

– Contains NOAA DMP template (not IOOS-adjusted), NOAA Data Management Planning Procedural Directive, and some sample plans

• NOAA Data Sharing Policy for Grants and Cooperative Agreements (https://www.nosc.noaa.gov/EDMC/PD.DSP.php)

• NOAA Data Documentation Procedural Directive (https://www.nosc.noaa.gov/EDMC/DAARWG/docs/EDMC_PD-Data_Documentation_v1.pdf)

• NOAA Procedure for Scientific Records Appraisal / Archive Approval (https://www.ngdc.noaa.gov/wiki/images/0/0b/NOAA_Procedure_document_final.pdf)

• NODC Submission Information Form (http://www.nodc.noaa.gov/media/pdf/submit/NODC_SubmissionInfoForm_v1.3.pdf)

Review DMP Template (15 min)• Get into small groups.

• Sketch out responses for each of the items. – Please try to develop actual responses for your regions– Focus on Sections 3, 4, 7 – Please note any potential issues/concerns!

• Use the template to guide you. – The information is made up for a fake RA– These theoretical responses would likely satisfy both NOAA and

certification requirements.– These responses are ONLY EXAMPLES.

• After 15 minutes, we’ll convene and discuss.

Discuss Issues/Concerns (20 min)• Let’s begin with Sections 3, 4, 7.

• We’ll discuss any potential issues/concerns and we’ll put them on the board with possible solutions.

• Hopefully this exercise has helped you think about how to write your DMP!

Quality Control Implementation

ISSUE: Should the IOOS DMAC community initiate a cooperative effort to develop quality control software to implement QARTOD/other standards, or should individual RAs be responsible for developing the necessary software?

QARTOD Manuals• Existing QARTOD Manuals

(http://www.ioos.noaa.gov/qartod/) :

– Dissolved Oxygen

– In-Situ Surface Waves

– In-Situ Current Observations

• Planned QARTOD Manuals in FY14

– Temperature/Salinity

– Water Level

– Wind Speed / Direction

Options

1. Initiate a coordinated QC software development effort, directed by a small team from the RAs and the IOOS Program Office

2. Each RA is responsible for developing the QC software needed to implement the QC standards/protocols for their data

3. An informal development effort is initiated on GitHub, where developers can build and share software at their discretion