Upload
miles-summers
View
213
Download
0
Embed Size (px)
Citation preview
1 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Addressing NOAA’s Data Management Challenges
Presentation to the World Data Center for Glaciology 30th Anniversary Workshop
Dr. Christopher G. FoxDirector
NOAA National Geophysical Data Center
2 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
NOAA Encompasses a Challenging Diversity
NOAA currently manages >90 environmental observing systems, some with hundreds of stations: including land-, sea-, air, and space-based observing platforms
These systems gather >300 diverse environmental parameters (e.g. marine biological health, economic fisheries data, physical and chemical state of the atmosphere and ocean, paleoclimate proxy data, geodetic survey points, etc.)
NOAA also requires other national, international and commercial data in its operations (some in real-time)
NOAA data management infrastructure includes numerous significant stovepipe systems
Future observing systems will produce vastly increased data volumes that will need to be archived and efficiently accessed by an expanding number of users
NOAA is migrating from this current stovepipe environment to an information enterprise
3 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
NOAA’ Top Ten Challenges1
10) Alphabet Soup
9) Stove Pipes
8) Integration
7) Architecture
6) Data Sharing
5) User Needs
4) Maximizing Benefits
3) Communication
2) Data Management
1) Execution
1. Vice Admiral Lautenbacher in address to American Meteorological Society Annual Meeting, January 30, 2006
4 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Data Management - a top priority
Improving data management is among the highest priority challenges facing NOAA
5 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Response - NOAA’s GEO-IDEGlobal Earth Observation Integrated Data Environment
Scope – NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systems
Vision – NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi-independent systems
Approach – evolution of existing systems into a service-oriented architecture built upon agreed standards, principles and guidelines
Result – a single system of systems (user perspective) to access the data sets needed to address significant societal questions
6 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Project Management
Undersecretary for Oceans and Atmosphere
DMITData Management Integration Team
NOSCNOAA Observing System
Council
DMCNOAA Data Management
Committee
All
NO
AA
Go
als
All
NO
AA
Lin
e O
ffic
es
7 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Scope
Concerned with environmental and geospatial data and information obtained or generated from worldwide sources to support NOAA's mission
Does not consider administrative support systems such as finance, personnel, acquisition or facilities management
Includes all aspects of data management, including data acquisition, ingest, data processing, archive and access (CLASS)
8 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Vision
“System of systems” – a framework to effectively and efficiently integrate NOAA’s many systems
Minimize impact on legacy systems Utilize standards Work towards a service-oriented architecture
9 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Approach
Each NOAA LO/program/project continues to manage its data independently
Standards– Adopt, adapt and only as a last resort, create– Open, inclusive process for adoption– Inclusive not exclusive use of standards
Service Oriented ArchitectureReference: Federal CIO Council, Jan ’06 "Services and Components Based Architectures: A Strategic Guide for Implementing Distributed and Reusable Components and Services in the Federal Government"
10 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Standards
Standard names and terminology Metadata standards
– e.g. FGDC and ISO 19115 w/ remote sensing extensions
Standard formats for delivery of data/products– WMO, NetCDF, HDF, GeoTIF, JPEG, etc.
Web Services Standards– World Wide Web Consortium– OGC (Maps, Features, Coverage, GML)– Community Standards: OPeNDAP, Unidata’s
Common Data Model (CDM),COARDS, CF– REST /SOAP / UDDI / WSDL where appropriate
11 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
NOAA GEO-IDE Standards Process
Submitted Standard – Standards can be submitted by anyone in NOAA for evaluation to see if they address needs and could be applicable to NOAA
Proposed NOAA Standard – Can be provisionally used within NOAA for technical evaluation
Recommended NOAA Standard – NOAA data systems should consider supporting the standard wherever applicable for evaluation in real-world NOAA systems
NOAA Standard – Approved and mandated where appropriate
Phased approach of submission, evaluation and adoption
12 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Fast-track submissionAn initial set of well known and/or widely used standards are being considered as an initial set of “submitted” standards
• Discovery Metadata: FGDC, ISO 19115 (w/ extensions), OBIS
• Keyword Lexicon: Start with Global Change Master Directory
• Metadata Exchange: XML compliant with FGDC CSDGM and OBIS
• Catalog Search: Compatible with Geospatial One Stop specifications (currently Z39.50 or OAI-PMH)
• File Transfer: FTP and HTTP
• Data Base Access: ODBC and JDBC
• API and Web Services: OpenDAP and OGC service specifications
• Data and product formats: The usual suspects…
13 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Service-Oriented Architecture
Under an SOA, capabilities are built one at a time to create “Web Services”
The fabric of the SOA is built upon standards for:– discovery (e.g. FGDC, ISO, DIF,CSW)– transport (e.g. HTTP, FTP, OPeNDAP)– use (e.g. netCDF, HTML, etc.)
Can be SOAP, REST, or some other standard
Define core common services and build Service layer agreements across NOAA
14 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
SOA (continued)
Operational Public Access Services: for public access to data, products and information services
Operational Services: where security, timeliness, and reliability are paramount
Scientific Services: where efficient and flexible discovery and access to data sets are required
Commercial value-added services
Four general classes of web services are anticipated for NOAA:
15 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Key Development StrategiesCoordinate activities through Communities of
Interest organized by “Data Types”– Grids, time-series, moving-sensor multi-dimensional, profiles,
trajectories, geospatial framework, point data and metadata
Evolutionary development through pilot projectsMinimize impact on legacy systemsDevelop Services and Component based
Architecture (SCBA) using a top-down, bottom-up iterative software development process (Federal CIO Council)
16 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Communities of interest based upon data typesAn initial list
Grids (e.g. model output, gridded data products) Moving-sensor multidimensional fields (e.g. satellite
swaths, side-scan sonar, weather radar) Time series (e.g. fish landings, sun spot activity, climate data,
paleo-records) Profiles (e.g. atmospheric soundings, ocean casts, profiling floats) Trajectories (e.g. underway ship measurements, aircraft track
data, ocean surface drifters) Geospatial Framework Data (e.g. shorelines, fault lines,
marine boundaries, map annotations) Point data (e.g. tsunami or seismic occurrences, geodetic
control) Metadata - information needed for the use and interpretation of
data
17 WDC for Glaciology 30th Anniversary Workshop
25 October 2006
Future Direction - Priorities FY07 Work with scientists/data system managers to assess
requirements and systems– Develop enterprise architecture and GEO-IDE Implementation
Plan– Implement standards process– Active out-reach activities
FY08/09 Incrementally execute work packages– Adopt/Adapt/Develop data standards and interoperability
mechanisms, e.g., translators and directory services– Direct, test and evaluate changes being made to data
management systems
FY10/11 Re-evaluate architecture related to new data systems (across NOAA & with national & international partners)