View
28
Download
2
Category
Tags:
Preview:
DESCRIPTION
OPeNDAP (The Open-source Project for a Network Data Access Protocol) APAC Tutorial October 12, 2007 pfox@opendap.org. Some Definitions. DAP = Data Access Protocol Model used to describe the data; Request syntax and semantics; and Response syntax and semantics. OPeNDAP The software; - PowerPoint PPT Presentation
Citation preview
APAC, Perth, WA 20071012
1
OPeNDAP(The Open-source Project for a Network Data Access Protocol)
APAC Tutorial October 12, 2007
pfox@opendap.org
APAC, Perth, WA 20071012
2
Some DefinitionsDAP = Data Access Protocol
Model used to describe the data; Request syntax and semantics; and Response syntax and semantics.
OPeNDAP The software; Numerous reference implementations; Core/libraries and services.
OPeNDAP Inc. OPeNDAP is a 501 c(3) non-profit corporation; Formed to maintain, evolve and promote the
discipline neutral DAP that was the DODS core infrastructure.
APAC, Perth, WA 20071012
3
Some DefinitionsSyntax
The computer representation of a data object - the data types and structures at the computer level; e.g.,
T is a floating point array of 20 by 40 elements.
Semantics The information about the contents of an object; e.g., T is sea surface temperature in degrees Celsius for a
certain region of the Earth.
APAC, Perth, WA 20071012
4
Distributed Oceanographic Data System
(DODS) Conceived in 1993 at a workshop held at URI.
Objectives were:– to facilitate access to PI held data as well as data held
in national archives and– to allow the data user to analyze data using the
application package with which he or she is the most familiar.
Basic system designed and implemented in 1993-1995 by Gallagher and Flierl with NASA and NOAA funding.
From 1994 to present it has been extended with NASA, NOPP, NSF and NOAA funding.
APAC, Perth, WA 20071012
5
Considerations with regard to the development of OPeNDAP
Many data providers
Many data formats
Many different semantic representations of the data
Many different security requirements
Many different client types
APAC, Perth, WA 20071012
6
Broad Vision
A world in which a single data access protocol is used for the exchange of data between network based applications regardless of discipline.
A layer above TCP/IP providing for syntactic and semantic consistency not available in existing protocols such as FTP.
APAC, Perth, WA 20071012
7
Practical Practical Considerations
The broad vision:
Is syntactically achievable, but
Is not semantically achievable, at least not in the near term.
APAC, Perth, WA 20071012
8
OPeNDAP Mission Statement
To maintain, evolve and promote a data access protocol (DAP) for the syntactically consistent exchange of data over the network.
The DAP should provide syntactic interoperability across disciplines and allow for semantic interoperability within disciplines*****.
APAC, Perth, WA 20071012
9
OPeNDAP Vision StatementTo achieve the mission:
OPeNDAP will be Non-profit. Easier to obtain federal funds. The DAP is more likely to be adopted.
OPeNDAP software will be open source. More likely to be adopted. Need community contributions to software.
OPeNDAP will mix implementation with research. Implementation - to encourage use. Research - to keep the protocol current.
APAC, Perth, WA 20071012
10
OPeNDAP Vision Statement (cont)
OPeNDAP will rely primarily on federal funding. Unlikely to obtain private funding for middleware. Development to be use case driven. Aligned with Vision/Mission.
Strategic direction will be sought from an Advisory Board consisting of data system experts and with input from you the community of developers.
OPeNDAP will seek partners.
OPeNDAP will utilize community working groups to develop ‘standards’ related to the DAP, OPeNDAP
APAC, Perth, WA 20071012
11
OPeNDAP tactics• The fundamental objective of OPeNDAP and
OPeNDAP Inc. is to facilitate internet access to scientific data
• This is done by:– Providing a protocol (DAP) to access data over the internet,– Hiding the format (and organization) in which the data are
stored from the user, and– Providing subsetting (and other) capabilities for the data at
the server
• OPeNDAP is based on a multi-tier architecture• OPeNDAP software is open source• Working groups formed on specific topics
APAC, Perth, WA 20071012
12
OPeNDAP relies on projects
• To guide user-based requirements for application needs as well as OPeNDAP core development (use cases)
• To provide maintenance and evolution of the core software and documentation
• Currently: 7 active projects, covers next ~2-3 years, 2 pending proposals
• ……
APAC, Perth, WA 20071012
13
SuccessWhat constitutes success of the OPeNDAP mission?
Adoption of the DAP across a broad range of disciplines with extensive use in several of these.
In order to achieve this the DAP must do the following:
It must be sufficiently flexible, all encompassing, etc. that it can be used across a broad range of data types.
Its implementation must be robust, secure, easy to use, provide for a broad range of services, etc.
The funding stream must be robust.
Active and engaged developer and user community
APAC, Perth, WA 20071012
14
Risks
It is still the case that
Some other data access protocols are seen as more attractive regardless of whether or not they are, or
Other data access protocols are developed because their community is not aware of OPeNDAP or of what its capabilities are.
We will compare and contrast some of these today
APAC, Perth, WA 20071012
15
To SucceedOPeNDAP must make sure that:
It’s (server and client) capabilities are well known across a broad range of disciplines.
The data model is inclusive.
The implementation is robust and meets user’s needs.
The DAP coexists with other protocols.
It has a robust funding base.
It has extensive documentation.
APAC, Perth, WA 20071012
16
Robust Funding?What is an appropriate level of activity (funding) for OPeNDAP? What should OPeNDAP be doing?
Core only? + Clients and servers? + Demonstration projects?
Should OPeNDAP be a small staff (core only) or…?
What priority should be assigned to the elements currently being developed? Which of these go beyond the core?
DAP4
AIS - metadata consistency
THREDDS - Catalogs
Toolkits - Matlab, IDL,…
Grid capabilities
OTHERS?
APAC, Perth, WA 20071012
17
Releases/Support
Periodic releases User services - OPeNDAP (support@opendap.org)
User support - first line of defenseManages the opendap-dev discussions
APAC, Perth, WA 20071012
18
Binaries Generated
There are approximately 80 binaries built on a nightly basis. They are built for the following platforms/operating systems:
Linux FC4 FC5
MacOS-X (universal binaries when possible)
Windows XP, win32
Java 1.5 (Tomcat 5.5)
IRIX (in four variants), Solaris, AIX, OSF
APAC, Perth, WA 20071012
19
Communication Website (http://www.opendap.org and test.opendap.org) SVN - Code repository
(http://scm.opendap.org:8090/trac/browser/) Trac - Task/milestone repository
(http://scm.opendap.org:8090/trac/) Telecons
Management - Weekly Developers - Weekly on Monday 11am MT
Twiki -> MediaWiki Management Developers Coming soon http://docs.opendap.org
opendap-dev e-mail list - main mechanism for messages
APAC, Perth, WA 20071012
20
OPeNDAP Community Working Groups Modeled after best practices of IETF, W3, OGC, IEEE,
ISO, and others Working Groups:
Authentication Security Server-side Functions Virtualization (Aggregation) Server-side processing Geospatial Interoperability Hyrax and *DS (TDS, GDS, FDS, etc.) Semantics DAPPER netCDF C++ client Response types Metrics Asynchronous transactions DAP4 Relational Database access via DAP
Wiki (http://docs.opendap.org/index.php/Working_Groups)
APAC, Perth, WA 20071012
21
OPeNDAP Community Working Groups Terms of Reference (abridged)
http://docs.opendap.org/index.php/Terms_of_Reference Each OPeNDAP Working Group is established to apply
members' expertise in their focus area to produce specific deliverables or outcomes
Types: Software and Documentation Minimum 3 people 6 month time-frame
APAC, Perth, WA 20071012
22
Servers Processing Servers Aggregating Servers - OPeNDAP chains Clients Ancillary Information Services Browser Interfaces Data System Integrators (ODC)
The OPeNDAP data access protocol is used by a variety of system elements.
OPeNDAP System Elements
APAC, Perth, WA 20071012
23
Servers
Servers receive requests and provide responses via the DAP.
Servers convert the data from the form in which they are stored to the OPeNDAP data model.
Servers provide for subsetting of the data and more.
APAC, Perth, WA 20071012
24
Data Data Data Data Data Data Data
HDF5
HDF4 JDBC
FreeFormFITS
CDF CEDAR
Data
netCDF
netCDF HDF4 HDF5
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
General
ESML
OPeNDAP Servers
APAC, Perth, WA 20071012
25
Data
GRIBBUFR
OPeNDAP
GDS
Data
CODAR
CODAR
Data
FDS
netCDFOPeNDAP
Data
General
pyDAP
Data
DAPPER
netCDFOPeNDAP
Data
netCDFOPeNDAP
TDS
Data
General
pyDAP
Data
netCDFOPeNDAP
TDS
OPeNDAP Servers (specialized)
Data
ESG
netCDFOPeNDAP
APAC, Perth, WA 20071012
26
Servers
Servers may also provide other services
Directory traversal.
Browser-based form to build URL.
ASCII version of data.
Metadata associated with the data.
Server side functions.
APAC, Perth, WA 20071012
27
Data
General
pyDAP
OPeNDAP Aggregation Servers
Data
GRIBBUFR
OPeNDAP
GDS
Data
CODAR
CODAR
Data
FDS
netCDFOPeNDAP
Data
DAPPER
netCDFOPeNDAP
Data
TDS
netCDFOPeNDAP
Data
General
JGOFS
Data
ESG
netCDFOPeNDAP
APAC, Perth, WA 20071012
28
The Aggregation Server: An Example
AggregationAggregationServerServer
File
DSP Data Set
FileFileFile
netCDF Data Set
File File
Matlab
Local
OPeNDAP
HTML, GIFMatlabClient
DSP
APAC, Perth, WA 20071012
29
Servers Processing Servers Aggregating Servers - OPeNDAP chains Clients Ancillary Information Services Browser Interfaces Data System Integrators (ODC)
The OPeNDAP data access protocol is used by a variety of system elements.
OPeNDAP System Elements
APAC, Perth, WA 20071012
30
Clients
Clients make requests and receive responses via the DAP.
Clients convert data from the OPeNDAP data model to the form required in the client application.
APAC, Perth, WA 20071012
31
netCDF C
Ferret GrADS
netCDF Java
IDV VisAD ncBrowse
Matlab
MatlabClient
Access ExcelIDL
IDLClient
ArcGIS
pyDAP
OPeNDAP Clients
ArcGIS
pyDAP
NCL
NCLClient
APAC, Perth, WA 20071012
32
Servers Processing Servers Aggregating Servers - OPeNDAP chains Clients Ancillary Information Services Browser Interfaces Data System Integrators (ODC)
The OPeNDAP data access protocol is used by a variety of system elements.
OPeNDAP System Elements
APAC, Perth, WA 20071012
33
Ancillary Information Service
• Current capability: Attributes only• Client-side only• Local and remote resources• Local resource databases
The AIS enables users to augment the metadata for a data source in a controlled way without requiring write access to the original data. By using the DAP, users are also isolated from data format issues.
APAC, Perth, WA 20071012
34
AIS enhancements
• Remote resource databases• AIS server• AIS for variables
These enhancements will greatly expand the usefulness of the AIS: Remote resource databases and an AIS server will enable third-party ‘AIS sites’ which may be sponsored by project offices or institutions, et cetera.
AIS for variables will enable adding metadata which are stored as ‘data.’
APAC, Perth, WA 20071012
35
Proposed AIS Server
Client linkedw/DAP
Software
DataSource
AISServer
AISResource
1
2
0
3
0. Client requests metadata from the AIS server (which appears no different from any other DAP server).
1. The AIS server gets metadata from data source2. The AIS server gets matching the AIS resource using the AIS database and
merges it into the metadata.3. The AIS server returns resulting the metadata object.
APAC, Perth, WA 20071012
36
Servers Processing Servers Aggregating Servers - OPeNDAP chains Clients Ancillary Information Services Browser Interfaces Data System Integrators (ODC)
The OPeNDAP data access protocol is used by a variety of system elements.
OPeNDAP System Elements
APAC, Perth, WA 20071012
37
OCAPI
A pure OPeNDAP C API (OCAPI) for the client-side
Applications:DAP-aware ‘commands’ for commercial
analysis programs (e.g., IDL)Scripting tools (e.g., Perl)
APAC, Perth, WA 20071012
38
ODC
IDLIDL
GrADSGrADS
FerretFerret
IDVIDV
VisADVisAD
ncBrowsencBrowse
MatlabMatlab
ExcelExcel
AccessAccess
OP
eND
AP
OP
eND
AP
GFDLnetCDF
GSFCBinary
URIHDF
GCMD
ODCODC
NVODS
- a Data System Integrator
APAC, Perth, WA 20071012
39
The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view.
The Data Access Protocol (DAP)
The DAP is a discipline neutral data access protocol; it is being used in astronomy, medicine, earth science,…
APAC, Perth, WA 20071012
40
Data Access Protocol (DAP2) - Current
DAP2 currently a NASA/ESE ‘Standard’
Current server (OPeNDAP 3.x; aka SERVER3)
XML responses (implemented)
DAP3
APAC, Perth, WA 20071012
41
DAP4 - Late 2007 (?) DAP4 improvements over DAP3:
Additional datatypesSwathBlob - GIF, MPEG,…
Additional functionality Check sumModulo
The additional datatypes will enable the DAP to be used in a wider variety of circumstances and are a direct response to users’ requests.
APAC, Perth, WA 20071012
42
OPeNDAP’s Hyrax (‘Server4’)
• Uses a modular architecture to support different application-level protocols– Data access using DAP– Catalogs using THREDDS– Browsing using HTML and ASCII
• Modules for data access– Different file types– Potential for database and scripting
• Modules for commands– Commands provide varying operations for different
protocols
APAC, Perth, WA 20071012
43
Hyrax Architecture
OLFS BES
OPeNDAP Lightweight Front end Server (OLFS) Receives requests and asks the BES to fill them Uses Java Servlets Does not directly ‘touch’ data
Data
Back End Server (BES) Reads data files, Databases, et c., returns info May return DAP objects or other data
APAC, Perth, WA 20071012
44
GridFTPDAP2
GridFTPDAP2
HTTPDAP2HTTPDAP2
ASCII outputASCII output
HTML outputHTML output
Info outputInfo output
OPeNDAP Lightweight Front end ServerOPeNDAP Lightweight Front end Server
THREDDSTHREDDS
Request Formulation**Request Formulation**
Req
uest
fro
m c
lient
Res
pons
e to
clie
ntB
ESSOAP-DAP
DAP2
APAC, Perth, WA 20071012
45
BES
Network Protocol andProcess start/stopactivities
Data Store Interfaces
BES Framework
PPT*Initialization/Termination
DAP2Access
NetCDF3 HDF4 FreeForm…
DataCatalogs
Commands**BES Commands/ XML Documents
*PPT is built in (other protocols)**Some commands are built in
APAC, Perth, WA 20071012
46
Today’s Overview• DAP Servers and Services• DAP Clients and Services
Recommended