OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 1
Ocean ESIPImproved Ocean Radar Altimeter
and Scatterometer Data, and Atmosphere-Ocean Simulations,
for Coastal and Global Change Studies
Victor Zlotnicki, JPL, [email protected]
Robert Raskin, JPL, [email protected]
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 2
Project Members
• Victor Zlotnicki, JPL• Rob Raskin, JPL• W. Tim Liu, Ernesto Rodriguez, JPL• Michael Stonebraker, UCB• Richard Troy, Science Tools, Inc.• Roberto Mechoso, Y. Chi, UCLA• C.K. Shum, Ohio State• John Ries, U. Texas
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 3
Problems to be Solved
• Long-term, global sea level time series require modeling & removing systematic differences between different sat. altimeters
• Derived sea surface wind products with the most physical meaning (wind divergence, wind stress and its curl) are not readily available
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 4
Problems to be Solved (cont.)
• A conceptual model of Earth science data processing, and an associated toolset, would help automate high level data generation, ensure data heritage information, and lower the cost of data processing.
• Mechanisms to extract corresponding hyperslices from large and diverse datasets, and deliver them on demand to end-user applications, are not readily available
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 5
Scientific Objectives
• Create and distribute improved global sea level and ocean wind products
• Facilitate the joint use of multiple sea surface variables
• Facilitate comparison between satellite data and ocean-atmosphere models
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 6
Technology Objectives
• Provide user-defined data and model subsets on demand via the internet
• Refine and implement the BigSur model of database-centric Earth science data processing, distribution and lineage tracking, its schema and toolkit.
• Further develop an application based on BigSur and Web technologies, to handle the processing steps from creation to delivery to end-users
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 7
Time Line
11/98 Late start due to lawyer negotiations
1999 2000 2001X
Data TP- SSH SST QuikScat TP Retrack GCMProducts: Output NSCAT TP-Curr ERS-SSH
Data SubsetAccess: GUI Digital Earth DODS Data Miner
BigSur TPGDR Ingest Qscat gridsAutomation: TPResiduals TPGrids
11/98 11/01
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 8
Primary Data Products• Altimeter:TOPEX/Poseidon, ERS-1,2
– Sea surface height, wave height, geostrophic current
– Retracked data
• Scatterometer: NSCAT and QuikScat– Sea surface wind, pseudo-stress
– Divergence and curl fields
• AVHRR:– Sea surface temperature (Reynolds )
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 9
Innovations• Spatial/temporal subsets selected from GUI, with plots
generated on the fly• Output types:
– Latitude-longitude maps– Animation of time-sequenced maps– Time series plots– Lon-time, Lat-time profiles– Line-time profiles along arbitrary lines
• Key technology ingredients:– Research Systems Inc’ IDL– Java
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 10
Example: NSCAT Wind
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 11
Example: Along-Track TOPEX Sea Surface Height
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 12
Example: Time Series Plot (Sea Surface Height Residual)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 13
Example: Time Series Plot (Sea Surface Temperature Anomaly)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 14
Example: Animation Option
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 15
Example: Longitude-Time ProfileEquator
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 16
On-line Subsetting Provided Through 3 Interfaces
• GUI (Java Applet)
• Digital Earth - Web Mapping Testbed (WMT)
• DODS (in progress)
• FTP of entire datasets
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 17
Digital Earth Interface
• Based upon the interface standard of the OpenGIS Web Mapping Testbed (WMT)
• Provides data access (maps) to other clients or any browser
• Maps have transparency option to facilitate overlay– WMT viewers can overlay Ocean ESIP data
with other WMT-compliant data
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 18
Vegetation Index (GLOBE server) & Sea Surface Temperature
(Ocean ESIP Server)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 19
Sample Call to Ocean ESIP Server w/ Digital Earth Interface
• Sea surface height on Oct 25, 1997 over the region: 180W, 20S, 80W, 40N:
http://oceanesip.jpl.nasa.gov/cgi-bin/de.pl
?BBOX=-180.,20.,80.,40.&LAYERS=ssh
&FORMAT=gif&REQUEST=map
&WIDTH=640&HEIGHT=480&SRS=4326
&WMTVER=1.0&DATE=1997102
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 20
GIS Compatibility• GeoTIFF output available from our Digital Earth
Interface• GeoTIFF encodes spatial information along with
TIFF– Bounding box (in latitude-longitude)– Map projection (equal angle)
• GeoTIFF can be imported directly into Arc/Info, ArcView, and most other commercial GIS products
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 21
DODS Interface
• Part of cluster with the DODS ESIP
• Sea surface height, geostrophic current, sea surface winds available through DODS as of June 00
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 22
Gridded
Datasets (TOPEX/Poseidon)
Geolocated
Residual
Parameters
Processing Steps
Lineage
BigSur Schema & Tools:Data and Data Processing Procedures
all Reside Within DBMS
Processing Plans Process Queue
Atmospheric Corrections Ocean Surface Corrections Interpolation Removal of Tides Removal of Time-Mean High-level
Low-level
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 23
Potential Advantages•Reliability
–Maintaining data and processing steps within an Informix DBMS insures quality control
•Efficiency–Subsequent processing steps are automatically scheduled and carried out
•Repeatability–Reprocessing is easily carried out by changing parameters and reinitiating processes
•Verifiability–Lineage provides an audit trail showing the processing that led to the creation of the dataset
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 24
Ocean ESIP/BigSur Processes
A “process” is a conversion of one dataset into another (higher-level) dataset
Completion of a process automatically triggers new processes. A “process queue” manages computer workload and schedules
new jobs.
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 25
Example: Conversion of TOPEX GDR Dataset into “Gridded” Dataset
• Along-track grid pointsTriggered by:
Newcycle of
GDR data available
Triggers:
Createresidual dataset
• Gridded dataset
• GDR dataset
grid_data.f merge_GDR.f
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 26
VACS: Visualization, Analysis, and Control System
• Developed at UCLA
• Web-based interface to:
– maintain meta-data and visualize 2D and 3D variables from datasets that are output by model runs
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 27
•Web-based User interface to maintain metadata and visualize 2D and 3D variables from model runs.
•Automatic 2-dimensional plot generation using IDL scripts (longitude-latitude and vertical cross-sections)
•Graphical interface to select geographic region of interest•Menu options to customize plots (maps, min and max values, filtering, contour labeling, etc., )
•Output plot delivered in GIF format
VACS - Visualization Subsystem
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 28
Web-based User interface to maintain meta-data and analyze variables and fields from datasets that are output by model runs.
•Automatic generation of a 2-dimensional plot of the difference of two output datasets.
VACS - Analysis Subsystem
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 29
Web-based User interface to submit local and remote ESM runs.
•Dynamic configuration of command scripts
•Can monitor results from a run on line
•Book-keeping of users and available machines
•Automatic startup of a run on a remote host with rsh or ssh
capabilities
VACS - Control Subsystem
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 30
Help Facility http://oceanesip.jpl.nasa.gov/gui
• GUI contains context-sensitive help menus
• User questions and comments will be entered into a DBMS, accessible to all– Creates a dynamic FAQ
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 31
Inventory-Level Metadata
• Data products are all created on the fly
• Granules exist, but they are sliced and diced and recombined for delivery
• Is it worth inventorying granules for outside view? (V0 issues)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 32
Metrics• 4 month test period (Feb-May 2000)
• 798 data requests via GUI
• 104 distinct users– Interactive nature of web access encourages multiple data
requests
• 5.9 Gb of data subsets extracted (2 bytes/data point)– Approximately 70Kb per request– Delivered amount varied depending on output type (GIF,
ASCII, averaged fields, etc.)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 33
Metrics (cont.)• No user registration or cookies
• Voluntary customer survey to be implemented this summer
• Only 4 questions submitted to help e-mail
• Home page listed on major search engines– Yahoo, AltaVista, Excite, WebCrawler, Lycos,
InfoSeek, Northern Light
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 34
Domain Source of Data Requests
050
100150200250300
edu
fore
ign
net,
com go
v
unkn
own
Total DataRequests
Distinct Hosts
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 35
Current Status
• Data subsets of TOPEX/Poseidon, NSCAT, and SST available via 3 different interfaces
• BigSur currently automates steps:T/P GDR ingest, ATGrid, ATResids, VanGrids (~40% of processes)
• VACS replaced ESMDIS.• Retracking of TOPEX/Poseidon data in
progress (Requires 3 months, wall clock time)
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 36
To Be Done• Use BigSur (60% of processes)
• Add QuikScat, ERS, TOPEX Retracked• Advertise our data products through PODAAC
and other sources• Keep our data products up-to-date• Implement Customer registration• Use customer surveys• VACS- Extend the Analysis module to include
correlations and other statistics
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 37
Lessons Learned (so far)
• Data centers /products must be advertised to be widely used
• Tight labor market for computer expertise is a big problem
• Collaboration with scientists and computer scientists off-Lab is challenging
• Some technologies are not worth implementing
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 38
TheEnd
OCEAN ESIP TECH REVIEW 6/8/2000JPL R.RASKIN / V. ZLOTNICKI 39