NOAA Earth Information Service (NEIS) Overview GSD Technical Seminar February 28 th , 2012

NOAA Earth Information Service

(NEIS)

OverviewGSD Technical Seminar

February 28th, 2012

ESRL Global Systems Division

NEIS Agenda

IntroductionEvaluation and FindingsConceptActivities and Next StepsDemonstration


NEIS Background

GSD Director Discretionary Funds (DDF) Proposal

NEIS Requirement• Fast access to any data, any time, right now, on any platform.

NEIS Group Objectives• Build upon NEIS work in the past• Assess GSD capabilities pertinent to the concept of NEIS.• Evaluate existing, developmental technology, and solutions relevant

to concept.• Evaluate envisioned needs of primary users.• Provide recommendation of how to best use GSD resources to

contribute to a National capability.


NEIS Team

Research Team

Jebb Stewart - LeadChris MacDermaid Woody RobertsDarien Davis Jeff SmithMike Romberg Paul Schultz

Development Team

Julien Lynge Jeff SmithChris MacDermaid Jebb StewartRandy Pierce


NEIS History

January 18, 2007, Roberts, et al. A OneNOAA Concept Prototype for Data Viewing, 23rd IIPS.

March 10, 2009, Hansen, et al. Earth Information Services, GSD Technical Review on GFE and Other Activities.


NEIS Evaluation

Interviewed/Discussed concepts with variety of groups across NOAA Line Offices. More than 15 individuals.

Questions Asked:How are people using your data?How are you using NOAA data yourself?What tools/systems/services/standards are you using?What related recommendations/lessons learned exist? What obstacles or roadblocks have you encountered with using/disseminating NOAA data?

Listened for items to help facilitate projects or help NOAA data become more interoperable.


Interviews

Person Role/Responsibility

Michele Jacobi, NOS ORR/ERMA. Just finished OAR detail with GoogleOceans, part of DWH ERMA response team

Karen Sender NMFS inPort

Katie Fisher NOS Center for Operational Oceanographic Products and ServicesHarmful Algae Bloom POC

Dan Pisut NESDIS NVL

Chris Ortiz Space Weather Center

Charley Alexander IOOS

Tiffany Vance PMELHPCC

Glen Rutledge NOMADS

Ted Haberman NGDC

Tony Lavoi CMSP, CSC, NOAA GIS

Tim Haverland NMFS GIS group

Cecelia DeLuca, OAR OAR NESII

Steve Hankin PMEL

John McDonough OER

Mike Little NASACathy Smith OAR/PSD

Jeff de La Beaujardiere NESDIS/OSD/TPIO also DMIT

NEIS System Evaluation


xkcd.com

Part of the evaluation looked at existing systems and standards including:THREDDS LAS ERDDAPGoogle Earth NASA World Wind GeoPortalMERMAid REVERB iRODSGrads Data Server More….

Lot of Great Technology, which can be leveraged, no ‘one’ solution.

NEIS Committee andGroup Involvement


GSD is aware of many ongoing activities

We have active membership in a variety of committees and groups including:OGC Standards CommitteesEDMCDMITUAFNextGen/NNEW programFIM/WRF ModelingGIS Committee

GSD Central Facility has vast experience in working with data and standards

NEIS Findings

• Existing metadata systems contain stale or obsolete information and are not useful Search for common term like wind returns hundreds of results. Expertise required to understand what metadata you are looking for.

• Many “Stovepipes” but no ability to combine/integrate data from different disciplines (Ecological, Meteorological, Oceanographic, Fisheries, etc...)

• Data sets are large and getting bigger. Future data sets will be difficult if not impossible to move around network to consumer applications.

• Cross data source data processing service does not exist. Some data processing services exist but usually only for data available from that particular service.

• Feedback (broken links, data usage, etc…) to data providers and consumers is non-existent.

Many more…ESRL Global Systems Division

11

Biological

ChemicalPhysical

• Biological, Chemical, and Physical data is all interrelated• NEIS allows data integration in 4D time and space, providing a scientific tool for

everything NOAA does professionally and the world NOAA affects.

Carbon TrackerFIM

Data Integration


12ESRL Global Systems Division

NOAA Vision - 2020

Ability to discover, integrate, understand all NOAA data (past, present, and future) from homepage.

TerraViz


NOAA Vision - 2020

• Most of NOAA's environmental data will be accessible via web services and will be described by good metadata

• A central and intelligent search engine, that has harvested the metadata of NOAA's datasets, will exist to find data

• NEIS server side analytics/processing will enable subsetting, aggregating, diff'ing, and other operations to be performed on data

• Data will be efficiently streamed to a variety of platforms and clients, including mobile devices

• TerraViz will be a general purpose client running on multiple platforms capable of searching for NEIS data and visualizing it in novel ways

• Framework is extensible, other groups can add knowledge and services.

NEIS – Concept Components

Build new Graphical User Interface system integrating and leveraging new and emerging technologies to meet NEIS goal ‘any data, any

location, any platform, now’

• Perform processing within cloud environment and with high speed connectivity to data sources, taking advantage of large processing power within clouds.

• Send graphics and server side processed/rendered/streamed data to GUI, improving bandwidth utilization.

• Take advantage of fast networking to make remote requests and processing appear like local application.

• Similar to how the new Amazon Silk Browser works.

Looking to the Future


Improve existing Metadata services making information useful.

• Improve searching by understanding what data means and providing improved filtering capability similar to how airline or hotel type search engines work.

• Incentivize/encourage people to use proper Metadata.

• Create a Metadata Dashboard to:oGather information from users on relevance, ratings, usage patterns, search

key words used (ie crowd sourcing). This information can be used to determine what users are searching for,

how they are finding data, what data they are not finding, etc...

oProvide feedback on broken links, service uptime

oProvide feedback on adherence to standards (Services, Metadata).

oProvide feedback to users of similar or related data sets.

Looking to the Future – Part 2




Impacts

• NOAA data ready for action. Services model facilitates agile

response to events. Services can be combined or reused quickly• Any data available through NEIS system can be operated on or

combined with other data. Integrated standardized formats and access.

• New and Existing systems have access to wide variety of NOAA data. Any new data added, easy incorporated with minimal to no changes required.


NEIS Use Case – Deepwater Horizon

• There was no way to pull disparate data sets into one display in order to track the impacts of the oil spill

oOcean currents

oEcosystems

oWeather

oLocation of booms, ships, and other response equipment

• Many GIS developer labor hours spent working on converting and formatting data to make it available.

Are We Prepared?

What would happen if another hurricane or other disaster occurred?

With a framework such as this:

• Improved searching capability would allow those in need to quickly and easily find pertinent data to the situation.

• All data available through framework would be readily accessible to be integrated across 4D (time and space).

• New data that is properly documented could be added to services and become immediately available to the framework of services. Spin up time for new data reduced dramatically.

• Because framework exists as a collection of services, any changes or modification can be completed quickly without taking entire system down.


Demonstration


Prototype Demonstration Background


What is TerraViz?• A system developed from the ground up by GSD based on Unity3D.• There are no licensing restrictions in distributing the TerraViz executable.• Unity3D is a well supported commercial game and data visualization engine

with 750,000 active users • Unity/C# is designed for 3D development, making it easy to create multiple

views, create 3D meshes representing environmental objects, etc.• Develop once, then will run under Windows, Mac, web browsers, iPhones,

iPads, Android devices, and even game systems (Wii, Xbox 360, etc.)• We can modify the user interface and add any feature we need.• TerraViz Handles millions of polygons and performs quickly

TerraViz is one possible visualization tool built to develop, test, and showcase NEIS framework.

25

Google Earth, works great for certain applications and static products, however:

• Handles ~10,000 polygons before slowing to a crawl.• Closed system (Google Earth source code not available) so we can’t add needed

features.• Google Earth plug-in works on Windows and Mac 32-bit browsers only (won’t run

on iPhones, iPads, or Android devices.)• Developing GE plug-in user interfaces in browser-dependent Javascript is

tedious.• Very Limited integrated data discovery or browsing, most uses require a

separate application.• Limited animation/data interaction support.• Limited support for non-static data (ie Time Series)

Why not Google Earth

Documents

NOAA Earth Information Service (NEIS) Overview GSD Technical Seminar February 28 th , 2012