Cooperative Agreement # NNS04AA78A 1 Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses SENSOR TO USER Progress

Cooperative Agreement # NNS04AA78A

1

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated AnalysesSENSOR TO USER

SENSOR TO USER

Progress Report - 2nd Year (Q2-Q4)

Prepared by

Dr. Erick Malaret, ACT Corp. (703-742-094)Email: [email protected]


2


Description and ObjectivesApplied Coherent Technology Corporation (ACT), the Naval Research Laboratory (NRLSSC) Oceanographic Division, and our NOAA partners, are under a cooperative agreement with NASA to maximize the value of NASA/EOS Earth science observations to decision makers related to coastal, marine, and ocean environments.

ACT and NRLSSC are integrating real-time ocean systematic measurements from NASA and NOAA satellites and available coastal observations along with coastal ocean model output into an automated real-time database of ocean weather in the Gulf of Mexico.

This project focuses on monitoring coastal ocean properties and it is aimed at providing new real-time support to NOAA efforts initially in the following focus areas: Harmful Algal Blooms (HABs), Coastal Flooding /freshwater discharge, and Hypoxia.

ACT in collaboration with NOAA will expand the project to other U.S. coastal areas and add additional NASA,NOAA,DOD and foreign data sources.

ApproachThe on-going effort is divided into an application component and a

and product research/development component.

• The Application Component [by ACT] – provides an architecture that addresses issues related to data ingestion, processing, distribution, access, and decision support. The key Decision Support Server and Fusion Server will be installed at NRL/SSC, followed by other servers at different NOAA sites.

• The Product Research/Development component [by NRLSSC] – provides state of the art oceanographic expertise and research addressing issues related to generation of new geophysical data products as part of a production pipe-line.

Schedule and Key DeliverablesYear 1: Application Component – a) Req. Gathering, b) Install high end HW/SW to

ingest/process/serve data, c) enhance current Fusion server & Decision Support Tool Technology.

Product R&D Component - ….

Year 2: Application Component – a) Demonstrate widely available internet access to all data levels, b) develop initial Hazardous Algal Bloom product based on satellite/model data in GIS and other compatible formats, c) add new data streams (e.g. NPP VIIRS)


Year 3-5: Application Component – a) Expand to other application areas, geographic areas, and data sources, b) identify and customize architecture to provide additional decision support tools, assist in transition of application to operation.


Initial Implementation

2nd year Started May 2005

Note: Big Scope of Cooperative Agreement Scope and participants remains unchanged


3


De

The next slides will concentrate on Application Component progress in the

last 7 months


4


WIPE(server)

REACT(CSC User)

Fusion Server at NAVY/NRL-SSC

REACT(NOS User)

REACT(Field Analyst)

WIPE(servers)

REACT

NOOA Fusion Servers at ACT

Web Browsers(HTML and Applet Access)

SOAPWSDL

HTTP/

CGI/XML

SOAP

WSDL

HTTP/CGI

Local +Satellite Imagery+MODELS + …

…

HAB Test-bed for NOAA: Present Network Configuration

NOAA/CW•MODIS dataNOAA/NDBC•Buoy data

FNMOC•Many models

OTHER+in-situ obs.+satellites Obs.

NOAA/CWNOAA/NDBCNOAA/NCEP

optional access directly by REACT

NRL-SSC•MODIS OBS•MODELS•MODIS VAPs

FTP &

HTML xfers

Final HAB Bulletin For Decision Maker (user Type 3)HAB bulleting creation Wizard

NOAA/NOS/CSC Analyst Team (user Type 2)

Data providers(user Type 1)


5


Apply WIPE/REACTTo HAB’s needs byNOAA


6


DATA INGESTION PROGRESS


7



8


• Ingestion Data Aspects and Engines (continue)– NRL Model Data

• Model data ingestion engines were enhanced in REACT to rapidly open any of the NRL volumetric NetCDF files. The files can be open via the REACT interface.

• Visualization tools were added to REACT to simplify the inspection of volumetric model data.

– Layer animation– File animation– Data inspection with row/col/depth proves– Interactive Intensity Mapping

– NRL Level 2-4 HDF Products• Integrated the ability to read any of the sub-datasets included in the NRL HDF files. • The REACT interface provides a hierarchical file layout over the file system and the

HDF file• User can load desired set of sub-dataset layers simultaneously, and remap into desired

cartographic projection• Visualization capabilities similar to the model data were added. Done: multiplicative and additive scaling constants are applied.

– NOAA Coast Watch• After Meeting with Coast Watch it was clear that ALL NOAA/CoastWatch products are

being saved in CW HDF format. • Added capabilities to read/load all the sample data provided by CW to ACT

– NOS Needed Data• Incorporated the Buoys data and filtering algorithms• In-situ observations can not be accessed automatically at this time

DATA INGESTION PROGRESS (continue)


9


Data Ingestion and Status of WIPE Server(s) NRL/SSC• The WIPE server at NRLSSC has been working since October 05. • NRL/SSC has given security approval to the machine, hence it can be access over

the internet as a web server. – The web page is: http://cesium.nrlssc.navy.mil

• Met with NRL/SSC at ACT, and identified automatic process to populate the WIPE server at NRL/SSC with both MODIS L2-L4 products and with MODEL data.

– The advantage of the approach taken is that it does not replicate in any way Hark Disk storage at NRL/SSC. It uses the same hard disk area that their DODS server is using.

• Training on WIPE was done in November– ACT is presently incorporating the suggestion received from NRL

Basemap MODIS COMPOSITE NCOM TEMP


10



11



12


Data Ingestion and Status of WIPE Server(s) (continue)NOAA CW Servers at ACT:• Working. Tested with Sample CW-HDF format data. All modes

working, except the Swath data sets.• Bow-tie projection problems in the MODIS L1B data.

– To save computational time we were not using all the ground control points in the MODIS data. This resulted in visible bow-tie projection problems that were not being handled correctly.

– Investigated used of third party SW decided against it after discussing with NRL/SSC that all that is needed is to use ALL the GCP information.

– Made significant modifications to the processing of Ground Control Point to generate a properly corrected geo-referenced image.

– The new approach is more computationally intensive, hence it takes longer to generate a full area mosaic at full resolution. 5-10 minutes.


13


Status of HAB WIPE Server at ACT• Presently working on getting all the data automatically pushed into the WIPE server

at ACT.

• Some of the cell count data is at FTP sites. Presently embedding at FTP library into

WIPE to simplify process of getting data from remote servers via FTP.

Data Ingestion and Status of WIPE Server(s) (continue)


14


SOFTWARE DEVELOPMENT PROGRESS


15


•MSHELL - Image Processing Interpreter Language developed by ACT. The language has a high level syntax similar to Matlab and IDL.

•ProVIEW - Professional Windows Environment for Windows (ProVIEW). ProVIEW has extensive capabilities in image processing, analysis, and map generation.

•WIPE – WWW Information Processing Environment. Data Assimilation, Processing, and distribution system with batch and interactive capabilities.

MSHELL Provides the Computational Engine Used by ProVIEW & WIPE ProVIEW Provides Many of the Graphical Capabilities Used Under WIPEWIPE’s building blocks have been used in multiple mission critical operations

Rapid Environmental Assessment Composition Tools (REACT) has been implemented and continues under rapid development.

It captures elements of MSHELL+ProVIEW+WIPE all under an easy to deploy stand alone application, and portable GUI environment.

During 2nd-3rd quarter continue expanding the tool space in REACT in response to needs of the HAB needs

CORE SW Development Progress


16


• As part of our NASA Cooperative Agreement, ACT has been participating in the Technology Infusion working group. The Technology Infusion has identified many capability blocks required in systems to support the EOS needs.

• REACT inherits key components of the ACT’s SW remote sensing SW tools– MSHELL - interpreter language with extensive capabilities for multidimensional data

manipulation– ProVIEW - graphical user interface around MSHELL– WIPE - provides fusion center architecture, with over 200 streams

• ACT has been using REACT and WIPE as a platform/architecture to demonstrate technology infusion from the OpenSource community, i.e.

– Qt – provides a portable library in C++ that allows GUI and system level support (A prototype version of REACT has been implemented for LINUX and Windows.)

– GDAL – the Geographic Data Abstraction Library is used by REACT to perform raster IO over many geo-graphic data formats new formats can be easily added following an OpenSource approach empowering developers to rapidly contribute in the adaptation of new formats. GDAL has been extended tosupport : CW-HDF, NRL-HDF, PCI-AUX (extended SRS support)

– gSOAP – used to SOAP enable REACT the developer needs to primarily concentrate on the core subroutine functionality. CONTINUED EXPANDING THE SOAP INTERFACE

– PROJ4 – a cartographic library with support to most of the projections used by USGS– SQLITE – a built-in SQL engine– OpenGIS/OGC – describes protocols/methods used for the request of products (This is work in

progress)

CORE SW Development Progress (continue)

GDAL + PROJ4 + GSOAP+…

MS

HE

LL

(Interp

reter+ scrip

ts!)

REACT


17


• REACT is been configured to provide Web Services for geo-spatial data– Web Services Requirements are:

• Available over the Internet• Not tied to any operating system or programming language• Self-describing via a common XML grammar• Discoverable via a simple find mechanism

• REACT captures a WIPE like type of interface for accessing inventory queries and generating mosaics, a set of five core functions are being added to the REACT/MSHELL scripting language resulting in a LARGE space of remote computing capabilities/options! (implemented using SOAP)

– $result = rmshell_execute($server, $scriptCommandList )– Data = rmshell_getVar( $server, $variableName)– $result = rmshell_putVar( $server, $variableName)– $result = rmshell_getFile( $server, $inFile, $outFile )– $result = rmshell_putFile( $server, $inFile, $outFile)ADDITIONAL FUNCTIONS WERE ADDED TO HANDLE ASYNCHRONOUS REQUESTS

• The above functionality can be exported to other applications via WSDL (an XML grammar for specifying public interface to the web service)

GAVE A PRESENTATION TO THE ESIP FEDERATION MEETING IN SAN DIEGO, ON WEB SERVICES USING WIPE&REACT.



18


REACT REACT

REACT WIPE

Script/algorithm or data

result or data


result or data

WIPE and REACT share the same processing engineREACT is configured to handle local user and a small number of remote applications. REACT can support remote applications as a WIPE-LITE system

Client side (Field System)

Server side (Field or Fusion Center System)

Web EnabledApplication

WIPEOr

REACT


result or data



19


• REACT can be seen as a WIPE-LITE system with more flexible visualization capabilities, and network centric capabilities.

• A prototype windows version of REACT can be downloaded from ACT, i.e.

http://www.actgate.com/reason



20


DECISION SUPPORT TOOLS PROGRESS


21


Decision Support Tools means different things to different users, e.g. – Data Provider need coverage and cartographic maps/mosaics

with simple to use query engines– Analyst at Fusion Center

• Visualization Tools, e.g animations of layers, volumetric vis.• Implementation of a Process in the form of a Wizard• Analysis: Multivariate Detection/Estimation/Prediction/…

– High Level Decision Maker, e.g. Coastal Manager GO/NoGo results with access to supporting data

• ACT has been working during the last year in the above end-user categories with requirements driven by the needs of:– NRL, – NOAA/CW – NOAA/NOS– NOAA/CSC– NOAA’s end-users

DECISION SUPPORT TOOLS …. The Challenge


22


• NOAA/NOS, NOAA/CSC, NASA/SSC, NOAA/NCDDC, and ACT met in February-05 to discuss the HAB Bulleting creation process and future architecture developments.

• As the result of the meeting a number of concrete suggestions were provided by NOS and CSC. (see following slide).

DECISION SUPPORT TOOLS PROGRESS (continue)


23


Development Operations Delivery

•[done]Use WIPE for new data stream ingestion•[done]Easy Incorporation of other geographies

•[done]Mosaic Images•[done]Alternate look from same data => flicker•[done]View data values from multi-sensors at same time •Skill assessment match in-situ data and imagery

•[done] Output data•[done] Oriented geo-tiff

Requirements• [done] Save view settings• [done] Archive data used in view• [done] Access from multiple sites• [done] Use of consistent color tables (use same tables as CW)

Outcome: NOAA/NOS/CSC requested ACT to demonstrate how WIPE/REACT could be used reproduce current HAB Bulleting creation process in REACT/WIPEPossible Advantages:

•Proposed approach could provide clear view of process and problems•Need web centric creation process (can be a browser or application)•Explore REACT/WIPE as next generation environment

DECISION SUPPORT TOOLS PROGRESS (continue)


24



25



26



27



28



29


HTML output + PDF output


30


SPEED Data Cube Generated on the Fly from U&Vcomponents


31


EXAMPLE OF READING AND DISPLAYING: generic formats, NRL’s MODEL DATA, MODIS L1B, CW Products, …


32


EXAMPLE OF READING AND DISPLAYING NRL’s MODEL DATA

DATAProbesOn Active Image


33


EXAMPLE OF READING AND DISPLAYING NRL’s MODEL DATA

Metadata information

Easy Access to Any Layer, Including Volume Animation


34


EXAMPLE OF NRL L2-L4

File

Sys

tem

Nav

igat

ion

and

HD

F D

ata

Str

uctu

re N

avig

atio

n


35



36


Meetings/Training/Demos– Supported Different ESIP Federations Meetings– Gave Training on WIPE at NRL (2 days)– Explore Training on REACT (over the WEB)

• For Example, Gave Presentation to CSC on REACT using VNC


37


ACT’s REASON Web page Is up an runninghttp://www.actgate.com/reason

Provides:•Report Summaries•Actual Data Access•DST Demos Products•Tutorials•…

…LIVE DEMO …


38


• NRL Related– Work over WIPE enhancements/customization requested by NRL-SSC

• Custom product generation• Virtual products on the fly…• Legend• …

– Further Training to NRL on Use of WIPE/REACT• NOAA Related

– Fine-tune HAB Wizard based on inputs from NOAA/NOS/CSC/…– Provide Training as needed to NOS and CSC

• ESIP Federation Related– Demonstrate Web Services Capabilities of WIPE/REACT (SOAP/WSDL)

• Improve interfaces with OGC and OpenDap • Demonstrate with ESIP Federation Web Services Chaining (using WIPE as a prototype)• Use REACT as a test case for Technology Infusion Process being Developed by the SEEDS Working

Group on Technology Infusion

• Investigate interface WIPE within NOAA/CLASS Architecture

• Start Investigation Next Year Target Application Areas Based on Team Recommendations

Next Quarter Main Effort …


39


Summary

• All ACT WIPE/REACT resources are accessible via ACT’s reason web page, i.e.– http://www.actgate.com/reason – On-line demos are being constructed to simplify end-user training

• NRLSSC WIPE Server is actively receiving NRL data streams– NOAA can access all NRL data directly from WIPE/Applet interface or from WIPE/REACT

interface! (over 1,200 different fields populated daily)

• REACT has matured to a state that NOAA/NOS/CSC can:– Test its use for HAB analysis tools– Test its use for HAB bulleting creation– ACT WILL WORK WITH THEM IN SIMPLIFYING USAGE OF TOOL

• WIPE Axon Architecture for Decision Support Tools is working– Need testing and use …


40


Possible Items for Discussion

• Are we heading (REACTing) in the right direction?

• How can we help make the NRL products easier to evaluate by NOAA?

• How can we make the emerging architecture more useful to NOAA?

• How can we make the large volume of unique data more useful to a larger audience?

• Can the data in the WIPE servers be accessed by non-team members?– The goal is to raise awareness of the many products available and the web-

services infrastructure

• Can we interface WIPE better with NOAA’s CLASS architecture?

…NRL and NOAA suggestions go here …

Documents

Cooperative Agreement # NNS04AA78A 1 Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses SENSOR TO USER Progress