A PROPOSED EARTH SCIENCE COLLABORATORY

IGARSS 2011, Vancouver, Canada 1

A PROPOSED EARTH SCIENCE

COLLABORATORY K-S Kuo1,2, Chris Lynnes1, Rahul Ramachandran3

1NASA Goddard Space Flight Center, USA2Caelum Research Corporation, USA

3University of Alabama-Huntsville, USA

7/27/11

Why ESC?

7/27/112IGARSS 2011, Vancouver, Canada

Data Intensive ScienceMany forms and sources of data

In situ measurementsRemote sensing observationsModel simulations

Large volumes of dataEffectiveness as a scientist

Increasing proportion of effort in data managementThreatening:

ReproducibilityCorrectnessProductivity


What is an ESC?Vision of a rich model development/simulation and data analysis environment that:

Provides access to various Earth Science modelsFacilitates model and analysis software developmentProvides access across a wide spectrum of Earth Science dataProvides a diverse set of science analysis services and toolsSupports the application of services and tools to dataSupports collaboration, i.e. sharing of data, tools and resultsSupports discovery and publication of all science artifacts

7/27/11

Basically, a new and natural place for Earth scientists to conduct their work and

collaborate with others!


The Situation TodayIslands of data and services with selective

connectivity

Data Center A

Data Center C

Data Center B


High-Level View

7/27/11

Cyberinfrastructure

Tool Library

Data Library

Laboratory Notebook

Workflow

Mediator

Data Centers


Tool Library• Discovery• Social

oSharingoTaggingoDiscussion

• Configuration ManagementoTestingoVersioning

Packager• autoconf• RPM• Web

wrapper

PROVISIONED

• GrADS• IDL• MatLab• ncl• nco• cdat

COMMUNITY• Quality filter• Coincidence• Feature

detection• Event service• Visualization

CONTRIBUTED

• [Tool 1]• [Tool 2]• [Tool 3]• [Tool 4]• [Tool 5]• …

PERSONAL• [Tool 1]• [Tool 2]• [Tool 3]• [Tool 4]• [Tool 5]• …


Data Library• Cache• Discovery• Social



Packager• data probe• format

check• metadata

wizard

PROVISIONED

• EOSDIS

COMMUNITY• Field

campaigns• MEaSUREs• ACCESS• Validation

CONTRIBUTED

• [Dataset 1]• [Dataset 2]• [Dataset 3]• [Dataset 4]• [Dataset 5]• …

PERSONAL• [Dataset 1]• [Dataset 2]• [Dataset 3]• …


Workflow Library• Discovery• Social



Packager• Workflow editor

PROVISIONED

• Processing Algorithms

COMMUNITY• GeoBrain• SciFlo• Data Mining• Giovanni

CONTRIBUTED

• [Workflow 1]• [Workflow 2]• [Workflow 3]• [Workflow 4]• [Workflow 5]• …

PERSONAL• [Workflow 1]• [Workflow 2]• [Workflow 3]• …


Laboratory Notebook• Discovery• Social


• Configuration ManagementoVersioning

Packager• Project Manager

• Experiment manager

• Notebook editor

PROVISIONED

• Tutorials• User guides• Example

uses• Educational

packages

COMMUNITY• Project results• Publications• Example

cases• Educational

packages

PROJECT• [Project 1]• [Project 2]• [Project 3]• [Project 4]• [Project 5]• …

PERSONAL• Notes• Journals• …


Mediator• Mediates tool interaction with data• OPeNDAP – a common data model

(accessible by most tools)• Custom modules reformat data for

the rest of the tools• Ontology matches tools with data,

and vice versa.


CyberinfrastructureServices used by all other

componentsSecurity

authenticationauthorizationcode audit/padded cell integrity checking

Socialtaggingsharingdiscussionsgroups

Cloudelastic provisioned storage and computing

Discoverydata, tools, workflows, experimentssearch by keyword, variable, time, author

Information Management

provenanceidentifiersarchive

Semantic Webdata ontologytools ontology 7/27/11


Key Advantages of ESC

Tool availability will be a force multiplierMore tools will be usable with more datasetsMore tools will be more available to more users

Knowledge sharing evolves from text on paper to a rich mixture of data, tools, workflows and articlesA “wikihow” for Earth Science data analysis

Incorporating live data, services and workflowsESC maintains a record of the analysis process

Share, repeat, build upon analysis techniquesTransparency of the process is built in

7/27/11


Prior ArtTalkoot, myExperiment.org – workflow sharing, virtual notebooksEarth System Grid – provisioned tools, format standards/checkersNASA Earth Exchange (NEX)Land Information System – OPeNDAP as access infrastructureEarth Science Modeling Framework – programmatic approach to integrationGiovanni, LAS – community services/toolsCanadian Space Science Data Portal (EOS, Feb. 22, 2011)Nebula – cloud provisioning

7/27/11

A Use CaseGPM Precipitation Retrieval Algorithm

Development


GPM Core Satellite: Dual-Frequency Precipitation Radar (JAXA) and GPM Microwave Imager (NASA)GPM Constellation: International partner satellites with mostly microwave radiometersRetrieval algorithms – 3 types

Radar-onlyRadiometer-onlyRadar-radiometer-combined

Participants in algorithm development are distributed in Japan, NASA centers (GSFC, MSFC, JPL), NCAR, and universities (FSU, Uwisc, etc.)

A Use CaseGPM Algorithm Development – Current

Situation


Interdependence among 3 types of algorithmsCommunication/Coordination – Narrow bandwidth

Periodic workshop meetings and teleconferences

Data access – DuplicativeEach location/group has a copy or subset of required data

Sharing of data/tools – Individual, not concertedthrough ftp/email

Knowledge sharing – Delayed

A Use CaseGPM Algorithm Development – with ESC


Tools

Data

ESC ClientmyS

ci Cat.

A

Tools

Data

ESC ClientmyS

ci Cat.

Z

Cloud

VM ImageTool

s Data

A

VM ImageTool

s Data

B

Community Catalog

ESC

A Use CaseGPM Algorithm Development – Multi-level

MembershipDC

B

A

K

J I H

G

F

E

GPMRadar-Only

Radiometer-Only

Combined

Algorithm

M

L

A Use CaseGPM Algorithm Development – in ESC


Enhanced communication/coordination – wide bandwidthEfficient data access – less duplicationImproved sharing – more pervasiveEffective knowledge sharing – immediate

Thank you!


Why now?Because we can do it (finally)!

Advances in standards acceptance andimplementation (OPeNDAP, autoconf)A consistent, loosely coupled architecture encapsulates complexity and maximizes flexibilitySocial networking has reached the mainstreamKey lessons can be learned from prior efforts

The need is growingInterest in working with multiple datasets is growingCalls for transparency and reproducibility are growing


What’s New?Macro View (forest-level)

Systematic approach to making data available to services and vice versaIntegration of all major analysis componentsConsistent view of all architectural componentsCyberinfrastructure services for all architectural components

Micro View (tree-level): Nothing!


How to move forward?

Option 1RFC to community on feasibility, challenges, approachFollowed by RFPs for component and integration

Option 2Narrow end-to-end prototypeFollowed by refactoring and broadening


Documents

A PROPOSED EARTH SCIENCE COLLABORATORY