Caro-COOPSData and Information Management
in Support of Integrated Ocean Observing System Efforts
Part I: Where we are
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
What is Caro-COOPS?
A wholly-integrated system for coastal observations and their application to user needs
• An extensive array of instrumented moorings• A comprehensive data and information
management system• An advanced suite of integrated models
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
IOOS DMAC High-priority Recommendations
Data suppliers wishing to participate in IOOS immediately should adopt initial DMAC recommendations including (IOOS DMAC, 2003):
- Creating FGDC-compliant metadata- Enabling data discovery- Making data available through the
OpeNDAP data access protocol- Making data browsable on-line through the
LAS- Ensuring long-term archival of contributed
data
A comprehensive data and information management system consisting of three primary components:
1. Reception, management and transfer of primary and ancillary observation data
2. Delivery of QC’d data, metadata and data products to a Web-based data and information portal
3. Provision of user-friendly tools and information products for model output
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Specifically, our system includes:
1. Offshore and shore-based near real-time data transmission via the low-orbit Iridium satellite cellular communications system, the Orbcomm low-orbit satellite communication system, and NOAA’s GOES
2. Data assimilation from observing systems and ancillary data providers (e.g. NDBC) to facilitate data parameterization for modeling and product generation
3. Archival within a relational database system for near real-time display and permanent storage in netCDF format
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Specifically, our system includes (cont.):
4. Access to data via DODS/OPenDAP5. Access to data and derived products available to
the public and research community via a web-based portal
6. Enhanced FGDC metadata development, search and query capabilities
7. Management and visualization (e.g. GIS, LAS) tools allowing users to generate products over the web
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Current activities focus on:
- Maintain and enhance techniques for data assimilation processing and product development- Support models with improved input source data- Maintain and enhance QA/QC procedures- Enhance data documentation tools and techniques- Develop products and web access for user communities Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Maintain and enhance techniques for data assimilation processing and product development- IOOS DMAC-compliant but not complacent!
- Using relational databases to address diverse dataset types and formats
- Working closely with SEA-COOS, CORMP, GoMOOS, SURA, NWLON, NDBC, NCDDC to be cognizant of and accommodate diversity of data types
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Maintain and enhance techniques … (cont.)
- Assimilating real-time observations via data telemetry, and using a variety of techniques (e.g. screen scraping, ftp) for ancillary data receipt
- Using web functionality and Internet-based GIS tools for data ingest, data browsing, product development, and data and product distribution
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Support models with improved input source data
- We recognize the importance of quality input source data for modeling and applications
- We understand the limits and implications of our source data and products
- Plans to discuss collaboration with interested groups (e.g. USCoE, SCDNR, county governments) for mutually important data development activities
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management SubsystemMaintain and enhance QA/QC procedures
Relevant QA/QC issues:
- On-instrument QA/AC
- QA/QC documentation detailing instrument calibration and maintenance procedures
- Organizational QA/QC (I.e. the QA/QC link with ancillary data providers)
- Regional QA/QC (I.e. data are checked against known or expected environmental range for geographic region)
- Feedback QA/QC (I.e. data are checked against immediate past and future readings)
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management SubsystemMaintain and enhance QA/QC procedures (cont.)
- Obviously, the quality of our models and products are dependent upon the quality of our data
- Goal is to minimize turnaround time for release of QA/QC’d data
- Initial QA/QC efforts involve:
1.) Raw data checked for transmission errors
2.) Instrument range checks
3. Time continuity checksScientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management SubsystemEnhance data documentation tools and techniques
- Importance of metadata development within Caro-COOPS cannot be overstated- Metadata standards include:
1. FGDC2. EML3. SensorML
- Key operational areas supported by metadata standards include observational data, instrumentation, methods and QA/QCScientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Enhance data documentation tools … (cont.)
- Caro-COOPS is developing a metadata editing tool to be incorporated with a data upload wizard
- Initially this tool will be a web-enabled metadata editor based on the model developed by the SURA-sponsored Cast-Net initiative.
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Develop products and web access for user communities
- Multiple user communities (e.g. researchers, managers, fisheries) requiring different products
-Website provides access not only to data but also information concerning the observational array, instrumentation, products, news and updates
- IOOS-supported data access tools (DODS, LAS) as well as GIS functionality have been implemented
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
A HypotheticalCat. 3 hurricane (MCP=960 mb)moving 50 km offshore along the coast (from SW to NE)
An advanced suite of integrated models
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Products with specific audiences in mind (e.g. emergency planners)
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
(Future) potential products with specific audiences in mind (e.g. recreational fishermen; courtesy NOAA website)
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Visitors and users of the system are on the increase!
Caro-COOPSData and Information Management
in Support of Integrated Ocean Observing System Efforts
Part II: Where we are going
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Caro-COOPS data and information management efforts provide for …
‘developing in situ and remote sensing techniques for near real-time provision of data on key biological and chemical variables’ (Malone 2003).
‘designing and implementing an enhanced, distributed data and information management system that links all observational and data management systems (across agencies and programs) to all data users’ (Ocean.US 2002).
‘providing an effective and sustained program for enhanced public awareness and education’ (Ocean.US 2003)
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Data and Information Management Subsystem
‘enhancing existing federal networks for in situ measurements … and to improve spatial and temporal resolution … to include physical, chemical and biological variables’ (Ocean.US 2002)
‘adding to the national program of … surveys to establish … multidisciplinary observations (assess the effects of land-based sources of pollution and to improve stock assessments of exploitable fish stocks)’ (Ocean.US 2002).
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Caro-COOPS data and information management efforts provide for …
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Current and proposed Caro-COOPS data and information management efforts are …
- Improving data assimilation, delivery and documentation
- Taking information and information products to the user communities
- Putting the “I” in IOOS!
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Improving data assimilation, delivery and documentation
- Program enhancements include:
establish true 24/7/365 connectivity to moorings with two-hour routine transmissions and on- demand data dumps
improvements in mechanisms for data delivery from Federal backbone data providers
enhance GIS capabilities focusing on both Internet-based GIS tools and stand-
alone GIS applications develop more rigorous QA/QC processes
and evaluate appropriate mix of automated versus human-intervention approaches
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Taking information and information products to the user communities
- Efforts will focus on developing products based on needs identified by specific user communities. Examples include:
working with state and local level emergency preparedness organizations to enhance GIS-based inundation tools
establish data kiosks at public venues including the visitors’ center at Hunting Island, and on the Sunset Pier
Data and Information Management Subsystem
Courtesy of
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Putting the “I” in IOOS!
- A major thrust will be the development of Metador, which is:
an enhanced data discovery and documentation tool
focuses on the use of an XML schema (EML) and SensorML while providing for FGDC-compliant metadata
will benefit all IOOS partners, including ocean, terrestrial and estuarine-based systems
Data and Information Management Subsystem
Scientific Advisory Board Meeting
Columbia, SC 16-18 December 2003
Putting the “I” in IOOS!
Data and Information Management Subsystem
- A second major thrust will be in providing leadership via interoperability projects in designing and demonstrating how local, regional and national systems can successfully integrate
Example of an integrated data product developed within the Caro-COOPS, SEA-COOS and SURA/Inter-operability framework