The Earth System Grid:A Visualisation Solution

Gary Strand

Introduction

Acknowledgments

•PI’s

–Ian Foster (ANL)

–Don Middleton (NCAR)

–Dean Williams (LLNL)

•ESG Development Team

–Veronika Nefedova (ANL)

–Ann Chervenak (ISI/USC)

–Carl Kesselman (ISI/USC)

–David Bernholdt (ORNL)

–Kasidit Chanchio (ORNL)

–Line Pouchard (ORNL)

–Alex Sim (LBNL)

–Arie Shoshani (LBNL)

–Bob Drach (LLNL)

–Dave Brown (NCAR)

–Gary Strand (NCAR)

–Jose Garcia (NCAR)

–Luca Cinquini (NCAR)

–Peter Fox (NCAR)

Current PracticesoScientist (or others) wants a visualisation

oVisualisation person gets appropriate data after verifying with data manager as to the name, location, total size, etc.

oData moved to “local” machine that has visualisation tools

oVisualization created on “local” machine

oHopefully, someone remembers to archive the visualisation

Simple Vis Example

Problems in the processoWhat if the data cannot be found (e.g. we have 1.2 million files, 73 TB of data), or the data manager is unavailable?

oWhat if there isn’t enough disk space or sufficient other resources?

oWhat if a better visualisation tool is located elsewhere?

oWhat if the visualisation should be shared?

oWhat if the visualisation is lost?

o ESG is part of the answers to these questions

What is ESG?

ORNL: Climate storage &computational resources

ORNL: Climate storage &computational resources

LANL: Next generationcoupled models & computing

LANL: Next generationcoupled models & computing

ANL: Computational grids,& grid-based applications

ANL: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

NCAR: Climate changepredication and scenarios

NCAR: Climate changepredication and scenarios

LBNL: Climate storage facility

LBNL: Climate storage facility

LLNL: Model diagnostics& inter-comparison

LLNL: Model diagnostics& inter-comparison

ESG Architecture

Tomcat servlet engine

Tomcat servlet engine

MCSMetadata Cataloguing Services

MCSMetadata Cataloguing Services

RLSReplica Location Services

RLSReplica Location Services

SOAP

RMI

MyProxyserver

MyProxyserver

MCS client

RLS client

MyProxy client

GRAMgatekeeper

GRAMgatekeeper

CASCommunity Authorization Services

CASCommunity Authorization Services

CAS client

diskMSS

Mass Storage System

HPSSHigh PerformanceStorage System

disk

HPSSHigh PerformanceStorage System

disk

disk

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

gridFTP

gridFTP

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver

openDAPgserver

openDAPgserver

gridFTPStripedserver

gridFTPStripedserver

LBNL

LLNL

ISI

NCAR

ORNL

ANL

SolutionsoWhat happens when data cannot be found, or the data manager is unavailable?

•Metadata catalogue service (MCS)

•Replica location service (RLS)

MCS and RLS and Metadata Services

METADATAEXTRACTION

METADATAEXTRACTION

METADATADISPLAY

METADATADISPLAY

METADATABROWSING

METADATABROWSING

METADATAQUERY

METADATAQUERY

ESG CLIENTS API & USER INTERFACES

Data &MetadataCatalog

Dublin CoreDatabase

COARDSDatabase

mirrorDublin CoreXML Files

COMMENTSXML Files

METADATA HOLDINGS

METADATAANNOTATION

METADATAANNOTATION

METADATAVALIDATION

METADATAVALIDATION

METADATA ACCESS(update, insert, delete, query)

METADATA ACCESS(update, insert, delete, query)

SERVICE TRANSLATIONLIBRARY

SERVICE TRANSLATIONLIBRARY

CORE METADATA SERVICES

METADATAAGGREGATION

METADATAAGGREGATION

METADATADISCOVERY

METADATADISCOVERY

METADATA & DATA REGISTRATION

METADATA & DATA REGISTRATION

PUBLISHINGPUBLISHING

HIGH LEVEL METADATA SERVICES

SEARCH & DISCOVERYSEARCH & DISCOVERYADMINISTRATIONADMINISTRATION BROWSING & DISPLAYBROWSING & DISPLAY

ANALYSIS & VISUALIZATIONANALYSIS & VISUALIZATION

Solutions (contd.)oWhat if there isn’t enough disk space or sufficient other resources?

•Hierarchical Resource Manager (HRM)

HRM

Solutions (contd.)oWhat What if a better visualization tool is located elsewhere?

•Distributed visualization

CDAT Example of an ESG Script Access

The next-generation language, Python, is used to access the Earth System Grid (ESG) at LLNL

Import cdms, vcs

db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”)f = db.open( “ncep_reanalysis_mo”)ds = f(‘ts’)x=vcs.init( )x.plot(ds)

CDAT: Example of an ESG GUI Client Access

Solutions (contd.)oWhat if the visualization should be shared?

• Access Grid plus Visualisation Tool

Collaborative Environments

Science Portals + AccessGrid:University of Michigan (Knoop, Hardin)Vegetation & Ecosystem Mapping Program

(VEMAP)NCAR/SCD VETS/KEGArgonne National Labs

ConclusionsVisualisation can require as many

services and resources as the initial computation

Many sites do not offer sufficient resources for the visualisations earth sciences require

ESG provides, and will provide, the tools that enable visualisation on a grander scale

Conclusions (contd.)ESG tools enable better data

access, better data knowledge, and the processes of collaboration for the needs of investigating, visualising, and learning