GrADS-DODS ServerAn open-source tool for distributed data access and analysis
Joe Wielgosz, Brian Doty, Jennifer AdamsCOLA/IGES - Calverton, MD
http://www.iges.org/grads/gds
OPeNDAP Working Conference - March 19, 2003
<?xml version="1.0" encoding="ISO-8859-1"?>
<gds name="my data server" home="/"> <catalog temp_size_limit="1000"> <data> <dataset name = "test” file = "testdata/little_endian.ctl“> </data> </catalog>...
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients
What the GDS can do
Perform server-side analysis and comparisons against other distributed datasets
servers clients
and more...
web browserLAS
GDS
OPeNDAP server
OPeNDAP server
GDScomparisondata
GRIB
HDF4
netCDF
BUFR
Unidata IDV
ncBrowse
GrADS
Ferret
Matlab
IDL
OPeNDAP
binary
GrADS
Java servlet
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
What's new since 1.1
New data type support
Station data - GrADS format and BUFR
Remote OPeNDAP data
Subsampling ("striding") for gridded data
Core code refactoredAnagram - generic data server frameworkSwappable, reusable modulesDesigned for efficiency - streaming I/O
XML-based configuration,with more flexibility in:Dataset loadingLoggingSecurityResource management
Improved web interfaceCustom links to help, home, dataset infoURL-based administration interface
Scales better to 1000's of datasetsOrganizes data catalog into directoriesFaster startup and smarter caching
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
Stable and fast COLA public GDS currently handling >600k hits/month
Administrator-friendly
Complete online documentation: http://www.iges.org/grads/gds/doc
COLA GDS Usage
0
100
200300
400
500
600
700800
900
1000
Sep-01Oct-01Nov-01Dec-01Jan-02Feb-02Mar-02Apr-02May-02Jun-02Jul-02Aug-02Sep-02Oct-02Nov-02Dec-02Jan-03
Thousands
Hits per month
0
100
200
300
400
500
600
700
Unique IPs each month
Total hits unique IPs
<gds> <catalog temp_size_limit="1000"> <data> <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /> <datadir name="/mnt/data1" suffix=".ctl"/> </data> </catalog> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" /> <grads> <invoker grads_bin="/users/joew/bin/grads"/> </grads> <servlet> <filter-abuse enabled="true" hits="1000" timeout="24" /> <filter-overload enabled="true" limit="20" /> <filter-analysis enabled="true" /> </servlet> <mapper> <service-admin enabled="true" auth="open-sesame" /> </mapper> <privilege_mgr default="public"> <ip_range mask="127.0.0.1" privilege="private" /> <privilege name="public"> <deny path="private_data" /> </privilege> <privilege name="private" /> </privilege_mgr></gds>
Install in minutes (really!)No root privileges neededCross-platform Java and ANSI C
Easy to configureEdit one (simple) XML file, and make updates on-the-fly
SecureRestrict dataset access & resource usage by IP address
And more...Automatic scans for new datasets Detailed loggingGraceful handling of heavy loadsEasily integrated with Apache...
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
In use, well-received
Positive response from:
• COLA scientists
• GrADS user community - research, corporate, hobbyists
• NOAA/CIRES CDC (earliest adopters outside COLA)
Some public GDS servers: (google on "grads dods server")
COLA Public Data Server: cola8.iges.org:9191 (soon to be wxdata.org)
COLA Monsoon Data Server: monsoondata.org
NOAA/CIRES CDC: www.cdc.noaa.gov/dods
FNMOC / GODAE: usgodae.org
NCEP: nomad2.ncep.noaa.gov
NASA / GSWP: voda.gsfc.nasa.gov:9090
NASA / NSIPP: beta.gsfc.nasa.gov:9090
plus activity at centers in France, Britain (BADC), and Japan...
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
What to expect
Continuing refinements to existing code
- uncovering bugs (some suspected to still exist)
- improving documentation where needed
- addressing usability issues
Station data support
- Finishing up BUFR I/O interface in GrADS
- Adding daily station reports to COLA public server
- Releasing station-data-enabled GrADS client
- Working on interoperability with other archives and clients
Under consideration:
- built-in metric collection
- catalog server
- login-based authentication
- near-line data support
- listening for ideas
GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds
COLA thoughts on OPeNDAP"DODS" vs "OPeNDAP"
With apologies, waiting for 4.0 - don't want to confuse user community
New protocol
I see DAP 4.0 as a chance to reduce complexity and increase robustness
We all have limited development resources - keep size of protocol proportional
Interoperability means common ground, not sum of all features
Anagram / Server-side Java
Java is an ideal server platform - stable, cross-platform, easy to code
Consider Anagram framework + Java-DODS as servlet counterpart to DAP C++ CGI framework? Slot in custom functionality to a standard core - good for gateways, access control / resource limits, web services, etc.
Analysis protocol
Increasing usage at COLA - perceived by scientists as extremely valuable
Time to think about interoperability? Complex issues involved
Modeler's needs
Grid / ESG II
Authentication important for collaborative projects
Can OPeNDAP provide a way for small shops to get Grid-enabled?