Manager’s Survival Guide to

OPeNDAP2

This is all about sharing data over the internet

Dr. Matthew HowardDept. of OceanographyTexas A&M University

14-January-2003

Talk Outline

• Introduction and Brief History

• Terms and Definitions

• What Resources Will You Need?

• Practical Aspects

• Available Resources

Introduction and Brief History

• 1992 The Vision

• 1994-2000 DODS

• 2000-2003 NVODS

• OOS and OPeNDAP2

The Vision

• 1992* WWW and NCSA Mosaic• 1993 Fall NASA/NOAA Workshop• Cornillon & Flierl CoIs + 40 others• Distributed Ocean Data System (DODS)

The system would allow one to access subsets of remote data with their favorite software analysis package across the network. DODS would subset the data and convert it from the originator’s data format to yours automatically.

• Designed by James Gallagher and Glen Flierl 1yr* Same year consensus reached for GOOS

DODS

• 1994 NOAA/NASA 2 Programmers X 3yrs• 1997-98 NOAA $1.5M - to continue core

system development.• 1998-2000 NASA $2.7M to continue core

system development, & entrain users from other disciplines. Included 13 partners NASAs, NOAA, AGU, UCAR, NCAR and 6 Universities.

National Ocean Partnership Program (NOPP)

• 1997 ONR/NOPP Received Proposals for National Virtual Ocean Data System (NVODS)

• One team proposed1) Make 3 distributed data systems (MEL, NOAAServer & DODS) interoperable2) Create network-enabled clients for visualization3) Facilitate data fusion4) Create regional hubs of data and expertise

• None of the proposals were funded

Meanwhile….

NOPP 1999 RFP VODHub

• NOPP VODHub Awarded in 2000 $4.5M for 3yr• “Development of an Integrated Regional,

National, and International Data System for Oceanography”

• 23 1st tier partners - 27 letters of support• Coordinate regional efforts (identify & entrain)• Get DODS data sets online - Populate• Continue development of core system

Back to our story … btw You are here

NOPP-2000 $4.5M/3yrs - Program Partners

Program Management and SupportPeter Cornillon University of Rhode Island  PI Richard Chinman UCAR Project Manager Margaret Davidson NOAA/CSC Workshops 

Regional CoordinatorsMark Abbott Oregon State UniversityAnders Andren U. WisconsinLinda Mercer Dept. of Marine Resources  State of MaineWorth Nowlin Texas A&M

Data ProvidersBrad Butman USGS  Coastal and Marine ProgramsDon Collins Jet Propulsion Laboratory  NASAJames Cummings Naval Research Laboratory NavyPeter Hamilton SAICHerve Roquet Meteo-FranceNeville Smith Bureau of Meteorology Research CentreKen Tenore University of MarylandKen Turgeon Minerals Management Service

System DevelopementGlenn Flierl Massachusetts Institute of TechnologyDave Fulker UNIDATA  UCAR User support James Gallagher University of Rhode IslandChris Lynnes NASA-Goddard DAACTed Habermann NOAA-NGDC GIS Steve Hankin NOAA/PMEL WebCrawler &Web Interface Richard Lawrence ESRI, Inc  GISLola Olsen NASA-Goddard Directory/Search

Series of Meetings and VODHub > NVODS

• Annual Reports• First-year Report and Status as of February 2002

• Regional Meetings• Synthesis Report

• Gulf Coast - 31 Oct to 2 Nov 2000• Southeast - 17 to 19 Nov 2000• Northeast - 8 to 10 Jan 2001

• West Coast - 17 to 18 January 2001

• Technical Interchange Meeting• ESRI/GIS - 7 to 8 Nov 2000

• National Meeting• Washington, D.C. - 25 to 27 Apr 2001

• Second Executive Committee Meeting• Ocean.US, Arlington, Va. - 26 to 27 Feb 2002

• NVODS/DODS Technical Working Conference• Boulder, CO - 9 to 11 Jan. 2002

NVODS and OOS

• After 2001 National meeting VODHub is called NVODS and program has broader scope (hooked up with ISOOS).

• GOOS and NVODS and DODS converge• OOS have emphasis on transporting data from

sensors to users in real and near-real time, and delayed modes.

• And some choices must be made about how.

Motivation for Using OPeNDAP2

• In draft paper: A Data and Communications Infrastructure for the U.S. Integrated Sustained Ocean Observing System (Hankin et al 2002) wrote:

• “The authors of this report recommend that OPeNDAP2 be considered as the preferred middleware solution to achieve the goals of the DAC subsystem in a rapid and cost effective manner”.

Terms and Definitions

• DODS and DODS-DAP

• NVODS and (OPeNDAP and OPeNDAP2)

• DODS Servers and Clients

• APIs and CGIs

• Metadata

The Data Provider:Installs free DODS-Server software on their web serverand places properly formatted data in a directoryaccessible by the web server.

The Data User:Obtains a DODS-enabled client program (eg. Matlab ora web browser) and enters a URL for the requesteddata set.

The requested data appears in user’s Matlab session orbrowser window.

Context for Terms and Definitions

Terms and Definitions

• DODS - Distributed Ocean Data System• “a data system intended to allow researchers transparent access to

oceanographic data - stored in any of several different file formats across the Internet”. (DODS User Guide, Tom Sgouros March 2000).

• DODS-DAP Data Access Protocol (The heart)• “An extension to the web. Html provides a basic structure to move

text and pictures over the network. DODS-(DAP) provides a basic structure to move data over the network. Both use http as the underlying transport protocol. (P. Cornillon- Presentation 2002).

• Aka middleware• Intermediate interchange format

NVODS

• National Virtual Ocean Data System• NVODS is the realization of the DODS vision. • NVODS is DODS-DAP plus everything needed to

make it a useful and widely used system such as catalogs, numerous high-quality desirable data sets, and popular software clients for most platforms.

OPeNDAP and OPeNDAP2

• One failure mode for DODS-DAP is the loss of the lead programmer and programming team that produced the DODS-DAP.

• Since DAP is useful to other disciplines one way to protect it against loss is to give it away, ie open it up to community-wide development.

• OPeNDAP (Open Source Project for a Network Data Access Protocol) is public non-profit corporation established to continue DAP development. This allows other programmers to become involved which can lead to more rapid development and wider platform support.

• DODS-DAP will become OPeNDAP2

DODS Servers and Clients

• A DODS-Server is a software program that reads, converts, subsets, and transmits data.

• Different servers for different data formats• The DODS-server software resides on a

web server and executes on web server’s host computer

• More than one server can be installed at a time

DODS Servers and Clients - cont

• DODS-Clients are software packages used to examine or manipulate data delivered by DODS-Servers

• Any web browser can be used to examine DODS data sets

• High-level commercial packages (eg Matlab) have been made DODS-aware.

• Free programs exist too.

Terms and Definitions - cont

• CGI (common gateway interface) are software programs that enhance the capabilities of a web server. -DODS-Servers are CGI programs

• API (Application Program Interface)-Software tools (subroutine libraries) that facilitate using one program with another.-Software clients are made DODS-aware by recompiling with DODS APIs.

Terms and Definitions - continued

• Metadata (Data about the data)

Syntatic metadata (byte stream is 100 real numbers in 10 by 10 array)Semantic metadata (data are temperatures in degrees C, collected 10-May-2002 by Joe Blow, on 100m grid with corners at X & Y).

Transmitted by DODS in separate structures(useful to decide if you want the data or not)

Terms and Definitions Summary

• DODS Vision in wide use = NVODS• DODS-DAP -> OPeNDAP2 (10X12X4)• DODS-Servers = CGI Web Server Programs.• DODS-Clients = Software Packages + DODS

APIs.• Syntactic metadata (necessary to use data at all) • Semantic metadata (necessary for interpretation)

What Resources Will You Need?

• DODS-Server software

• Web Server

• Data in a servable format

• Human resources

Supported Hardware Platforms for

DODS Data Servers

Sun Sparc Solaris 2.6

Intel PC Linux RedHat 6.2

Intel PC Linux RedHat 7.0

Intel PC Linux RedHat 7.1

Dec Alpha OSF 4.0f

SGI Irix 6.5 (n32/mip3)

free

Free$1k -$15K

Web Servers

• Most sites run free Apache Web Server

• Netscape’s FastTrack Web Server works fine too.

• Limitation is DODS-Server programs.

• OPeNDAP should yield Windows based server solutions.

• At present UNIX is it.

Data in Servable Format

• Most gridded self-describing formats are supported such as netCDF, HDF, and maybe GRIB.

• Matlab binary on some platforms• DSP on some platforms• ASCII can be handled by DODS-FreeForm server

and the DODS-JGOFS server or converted to netCDF using several methods.

• No support for ESRI GIS formats yet.

Human Resources

• Web Server Administratordownload DODS-Servers and copy to

CGI area of web server

• Data formatter/re-formatter– Programming or scripting ability

• C++ or Java programmer for building DODS-enabled clients

Practical Aspects

• Existing Data Format: – If netCDF or HDF you are in great shape– If not you’ll probably need to do some work

• Duplication of effort and files– Separate data streams– Files in two formats (for legacy and for DODS)– Double storage and management issues– Ideally serve your files as they are.

• Metadata– Is your metadata complete? In separate files?– FGDC requirements

ScriptnetCDFcnv cdl ncgen

Convert Existing Files to netCDF

Create netCDF files as you go.

Your Program netCDF API+ netCDF

Practical Aspects

• Problems area– Security is basically web site security– Data discovery (advertise, register, web

crawler)– Data Selection (GUI assisted 4D selection)– Hardware platform client-server matrix– GIS/SIS

Outside Resources

• DODS website

• UCAR contract help

• Regional Centers

• URI

• Catalog websites