L inked E nvironments for A tmospheric D iscovery Linked Environments for Atmospheric Discovery (LEAD) Kelvin K. Droegemeier School of Meteorology and

Linked Environments for Atmospheric Discovery

Linked Environments for Linked Environments for Atmospheric Discovery Atmospheric Discovery

(LEAD)(LEAD)

Linked Environments for Linked Environments for Atmospheric Discovery Atmospheric Discovery

(LEAD)(LEAD)Kelvin K. Droegemeier

School of Meteorology and Center for Analysis and Prediction of Storms

University of Oklahoma

Jay AlamedaNational Center for Supercomputing Applications

University of Illinois at Urbana-Champaign

David McLaughlin

greeting. introduce team present.explain why chandra isn't there.


Geosciences CI ChallengesGeosciences CI Challenges• Enormously complex human-natural system

– Vast temporal (sec to B yrs) and spatial (microns to 1000s of km) scales

– Highly nonlinear behavior

• Massive data sets– physical and digital– static/legacy and dynamic/streaming– geospatially referenced– multidisciplinary and heterogeneous– open access


Geosciences CI ChallengesGeosciences CI Challenges• Massive computation

– weather, space weather, climate, hydrologic modeling

– seismic inversion– coupled physical system models

• Inherently field-based, visual disciplines with the need to manage information for long periods of time

• Bringing advanced CI capabilities to education at all levels

• Connecting the last mile to operational practitioners


• Each year, mesoscale weather – floods, tornadoes, hail, strong winds, lightning, and winter storms – causes hundreds of deaths, routinely disrupts transportation and commerce, and results in annual economic losses > $13B.

Where Where ALLALL These Elements These Elements Converge: Mesoscale WeatherConverge: Mesoscale Weather



What Would What Would YouYou Do??? Do???


What What Weather TechnologyWeather Technology Does… Does…Forecast ModelsNEXRAD Radar

Decision Support Systems


What What Weather TechnologyWeather Technology Does… Does…Forecast ModelsNEXRAD Radar

Decision Support Systems

Absolutely Nothing!Absolutely Nothing!


The LEAD GoalThe LEAD GoalProvide the IT necessary to allowProvide the IT necessary to allow

PeoplePeople (scientists, students, (scientists, students, operational practitioners) operational practitioners)

andand

TechnologiesTechnologies (models, sensors, data (models, sensors, data mining)mining)

TO INTERACT WITH WEATHERTO INTERACT WITH WEATHER


The RoadblockThe Roadblock• The study of mesoscale weather is stifled by rigid

IT frameworks that cannot accommodate the – real time, on-demand, and dynamically-adaptive needs of

mesoscale weather research; – its disparate, high volume data sets and streams; and – its tremendous computational demands, which are

among the greatest in all areas of science and engineering

• Some illustrative examples…


Analysis/Assimilation

Quality ControlRetrieval of Unobserved

QuantitiesCreation of Gridded Fields

Prediction/Detection

PCs to Teraflop Systems

Product Generation, Display,

Dissemination

End Users

NWSPrivate Companies

Students

Traditional MethodologyTraditional Methodology

STATIC OBSERVATIONS

Radar DataMobile Mesonets

Surface ObservationsUpper-Air BalloonsCommercial Aircraft

Geostationary and Polar Orbiting Satellite

Wind ProfilersGPS Satellites








Dissemination

End Users


Students


STATIC OBSERVATIONS





The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!

The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!


The Consequence: Model Grids The Consequence: Model Grids Fixed in Time – No AdaptivityFixed in Time – No Adaptivity








Dissemination

End Users


Students

STATIC OBSERVATIONS





The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static

Models Responding to Observations


10 km

3 km

1 km

20 km

Model Dynamic AdaptivityModel Dynamic Adaptivityt = tt = too


t = tt = to o + 6 Hours+ 6 Hours

10 km

3 km

3 km

3 km3 km

10 km

20 km


Today’s Standard Computer ForecastToday’s Standard Computer Forecast

12-hour NationalForecast (coarse grid)

Radar

Radar(Tornadoes

in Arkansas)


Today’s Standard Computer ForecastToday’s Standard Computer Forecast

12-hour NationalForecast (coarse grid)

Radar

Radar(Tornadoes

in Arkansas)


Radar(Tornadoes

in Arkansas)

6-hour Mesoscale Forecast

(medium grid)

Radar

Experimental Experimental MesoscaleMesoscale Window Window

Radar


Radar(Tornadoes

in Arkansas)

6-hour Mesoscale Forecast

(medium grid)

Radar

Experimental Experimental MesoscaleMesoscale Window Window

Radar


Radar 6-hour LocalForecast (fine grid)

Experimental Experimental Storm-ScaleStorm-Scale Window Window

Xue et al. (2003)


Dynamic Adaptivity in ActionDynamic Adaptivity in Action


11 h Forecast 11 h Forecast 20 June 200120 June 2001

(6 km)(6 km)

Courtesy Weather Decision Technologies, Inc.



(6 km)(6 km)




(6 km)(6 km)




(6 km)(6 km)



MesoscaleWeather

LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather


MesoscaleWeather

NWS National Static Observations & Grids



MesoscaleWeather



Local Observations


MesoscaleWeather


UsersADaM ADAS Tools


Local Observations


MesoscaleWeather



Local Physical Resources

Remote Physical (Grid) Resources

Virtual/Digital Resources and Services

MyLEADPortal


Local Observations


MesoscaleWeather






MyLEADPortal


Interaction Level I

Local Observations








Dissemination

End Users


Students


STATIC OBSERVATIONS





Observing Systems OperateLargely Independent of theWeather – Little Adaptivity

Observing Systems OperateLargely Independent of theWeather – Little Adaptivity


NEXRAD Doppler Weather NEXRAD Doppler Weather Radar NetworkRadar Network


The Limitations of NEXRADThe Limitations of NEXRAD


The Limitations of NEXRADThe Limitations of NEXRAD#1. Operates largely independent#1. Operates largely independent

of the prevailing weather conditionsof the prevailing weather conditions



#2. Earth’s curvature prevents 72% of the atmosphere below 1 km from being observed

#1. Operates largely independent#1. Operates largely independentof the prevailing weather conditionsof the prevailing weather conditions



#2. Earth’s curvature prevents 72% of the atmosphere below 1 km from being observed

#1. Operates largely independent#1. Operates largely independentof the prevailing weather conditionsof the prevailing weather conditions

#3. Operates entirely independent from#3. Operates entirely independent fromthe models and algorithms that use its the models and algorithms that use its

datadata

Linked Environments for Atmospheric DiscoverySource: NWS Office of Science and Technology

The Consequence: 3 of Every 4 The Consequence: 3 of Every 4 Tornado Warnings is a False AlarmTornado Warnings is a False Alarm








Dissemination

End Users


Students

The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static

DYNAMIC OBSERVATIONS

Models and Algorithms Driving Sensors


New NSF Engineering Research New NSF Engineering Research Center for Adaptive Sensing of the Center for Adaptive Sensing of the

Atmosphere (CASA)Atmosphere (CASA)• UMass/Amherst is lead institution• Concept: inexpensive, dual-polarization phased array Doppler radars on

cell towers – existing IT and power infrastructures!• Adaptive dynamic sensing of multiple targets (“DCAS”)












MesoscaleWeather

Local Observations

MyLEADPortal






Experimental DynamicObservations



MesoscaleWeather

Local Observations

MyLEADPortal






Experimental DynamicObservations



MesoscaleWeather

Local Observations

MyLEADPortal





Interaction Level II


The LEAD Goal RestatedThe LEAD Goal Restated• To create an integrated, scalable framework

that allows analysis tools, forecast models, and data repositories to be used as dynamically adaptive, on-demand systems that can– change configuration rapidly and automatically in

response to weather;– continually be steered by new data (i.e., the

weather);– respond to decision-driven inputs from users;– initiate other processes automatically; and – steer remote observing technologies to optimize

data collection for the problem at hand;– operate independent of data formats and the

physical location of data or computing resources


CS Challenges/BarriersCS Challenges/Barriers• Workflow

– Dynamic/agile/reentrant

• Data– Synchronization, fault-tolerance, metadata, cataloging,

interchange, ontologies

• Monitoring and performance estimation– Detection of vulnerabilities, recovery, autonomy

• Mining– Grid functionality, scheduling, fault tolerance


Meteorology Challenges/BarriersMeteorology Challenges/Barriers

• “Packaging” of complex systems (WRF, ADAS)• Fault tolerance• Continuous model updating for effective use of truly

streaming observations• Storm-scale ensemble methodologies• Hazardous weather detections based upon gridded

analyses versus use of “raw” sensor data alone• Dynamically adaptive forecasting (models and

observations) – how good compared to current static methodologies?


LEAD ArchitectureLEAD Architecture

DistributedResources

ResourceAccess Services

UserInterface

Desktop ApplicationsLEAD Portal

Portlets

CrosscuttingServices

Con

figu

rati

on a

nd

E

xecu

tion

Ser

vice

s Application ResourceBroker (Scheduler)

Application & Configuration Services

Client Interface

Dat

a S

ervi

ces

Wor

kfl

ow S

ervi

ces

Cat

alog

Ser

vice

s


LEAD ArchitectureLEAD Architecture

DistributedResources

ComputationSpecialized

ApplicationsSteerable

Instruments Storage

Data Bases

ResourceAccess Services GRAM

Globus

SSH

Scheduler

LDM

OPenDAP GenericIngest Service

UserInterface

Desktop Applications• IDV• WRF Configuration GUI

LEAD Portal

PortletsVisualization Workflow Education

Monitor

Control

Ontology Query

Browse

Control

CrosscuttingServices

Authorization

Authentication

Monitoring

Notification

Con

figu

rati

on a

nd

E

xecu

tion

Ser

vice

s WorkflowService

ControlService

QueryService

StreamService

OntologyService

MyLEAD

WorkflowEngine/Factories

VO Catalog

THREDDS

Application ResourceBroker (Scheduler)

Host Environment

GPIR

Application Host

Execution Description

Applications (WRF, ADaM,IDV, ADAS)

Application Description

Application & Configuration Services

Client Interface

Observations• Streams• Static• Archived

Dat

a S

ervi

ces

Wor

kfl

ow S

ervi

ces

Cat

alog

Ser

vice

s

Decoder/ResolverService

RLSOGSA-

DAI


Key System Components and Key System Components and TechnologiesTechnologies

Capability/Resource Principal Technologies

Atmospheric, Oceanographic, Land-Surface Observations

CONDUIT, CRAFT, MADIS, IDD, NOAAPort, GCMD, SSEC, ESDIS,

NVODS, NCDC

Operational Model Grids CONDUIT, NOMADS

Data Assimilation Systems ADAS, WRF 3DVAR

Atmospheric Prediction Systems WRF, ARPS

Visualization IDV

Data Mining ADaM

NSF NMI Project Globus Tool Kit

Semantic Interchange and Formatting ESML, NetCDF, HDF5

Adaptive Observing Systems (Radars) CASA OK Test Bed, V-CHILL

LEAD Portal NSF NMI Project (OGCE)

Workflow Orchestration BPEL4WS

Monitoring Autopilot

Data Cataloging/Management THREDDS, MCS, SRB

Linked Environments for Atmospheric DiscoveryThe Driver: Canonical Research & Education Problems

The LEAD Research ProcessThe LEAD Research Process

Fundamental Scientific and Technological Barriers

System Functional Requirements and Capabilities

System Architecture and Definition of Services

Building Blocks

Technology Generations

End User Focus Group Testing and Deployment

The End Game: Canonical Research & Education Problems

Basic Research Prototypes Test Beds


Generation 2DynamicWorkflow

Generation 3AdaptiveSensing

Generation 3AdaptiveSensing


Generation 1Static

Workflow

Generation 1Static

Workflow


Generation 1Static

Workflow

Year 1 Year 2 Year 3 Year 4 Year 5

LEADLEAD Technology Generations Technology GenerationsT

echn

olog

y &

Cap

abil

ity

Generation 1Static

Workflow

Generation 1Static

Workflow

Look-Ahead Research

Look-Ahead Research


In LEAD, Everything is a ServiceIn LEAD, Everything is a Service• Finite number of services – they’re the “low-level” elements

but consist of lots of hidden pieces…services within services.

Service A(ADAS)

Service B(WRF)

Service C(NEXRAD Stream)

Service D(MyLEAD)

Service E(VO Catalog)

Service F(IDV)

Service G(Monitoring)

Service H(Scheduling)

Service I(ESML)

Service J(Repository)

Service K(Ontology)

Service L(Decoder)

Many others…


Start by Building Simple Prototypes to Start by Building Simple Prototypes to Establish the Services/Other Capabilities…Establish the Services/Other Capabilities…


Service F(IDV)

Service L(Decoder)

Prototype X




Service F(IDV)

Service L(Decoder)

Prototype Y

Service D(MyLEAD)





Service F(IDV)

Service L(Decoder)

Prototype Z

Service A(ADAS)

Service I(ESML)


Service D(MyLEAD)



Service B(WRF)

Service A(ADAS)


Service D(MyLEAD)

Service L(Mining)

Service L(Decoder)


……and then Solve General Problemsand then Solve General Problemsby Linking them Together in Workflowsby Linking them Together in Workflows


Service B(WRF)

Service A(ADAS)


Service D(MyLEAD)

Service L(Mining)

Service L(Decoder)


……and then Solve General Problemsand then Solve General Problemsby Linking them Together in Workflowsby Linking them Together in Workflows

Note that these servicescan be used as stand-alonecapabilities, independent of

the LEAD infrastructure(e.g., portal)







Feedback from Application ScientistsFeedback from Application Scientists

Benefits• Single sign-on feature is very handy• Secured access to compute resources from a

browser, is increasing productivity

Difficulties• Grid authentication is not trivial to use - important

feature needed by an application scientist• Hard to keep track of continuously evolving grid

middleware• System needs continuous development as

middleware on production machines moves forward and is not backward compatible

Canonical Problem #3

Problem #3: Dynamically Adaptive, High-Resolution Nested Ensemble Forecasts

Goal: For the continental United States (CONUS), automatically generate a 1-km grid spacing ADAS analysis every 30 minutes, and a 6-hour, 2-km grid spacing CONUS forecast every 3 hours. Automatically launch finer-grid spacing nested WRF ensemble forecasts when data mining algorithms – applied to both the CONUS analyses and forecasts – detect features indicative of storm potential (e.g., convergence lines, strong instability, incipient convection) or actual storm development. Conduct rigorous post-mortem assessment of statistical forecast skill and compare the high-resolution nested grid forecasts with the single-grid CONUS run at coarser resolution.


START

Define Data Requirements and Query for Desired Data


START


Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)


START



Allocate Computational Resources


START




DataSurface Observations

Upper-Air ObservationsCommercial Aircraft Data

NEXRAD Radar DataSatellite Data

Wind Profiler DataLand Surface Data

Terrain DataBackground Model Fields

and Previous Forecasts

ESML &Decoding


START










ESML &Decoding

Remapping, Gridding,

Conversion

ADAS Quality Control


START










ESML &Decoding


Conversion


ADAS Quality Control ADAS

AnalysisProcessing


START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields


START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields

ADAS-to-WRF Converter


START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields


3D Gridded Fields in WRF Mass Coordinate + Suite

of Ensemble Initial Conditions


START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




Multiple Copies of WRF Forecast Model Running Simultaneously


START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output



START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage



START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage



Meta Data Creation and Cataloging

START










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage




Visualization & Data MiningSTART










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage




Visualization & Data Mining STOPSTART










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage




Visualization & Data Mining STOPSTART










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage




Visualization & Data Mining STOP

Adjust ForecastConfiguration and

Schedule ResourcesSTART










ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage

















How Would One Go How Would One Go About Setting This About Setting This

Up in LEAD??Up in LEAD??• The “First LEAD Commandment”

– Thou shalt not use unintelligible computer science jargon in the portal for describing options/tasks to end users

– Foo, portlet, ontology, widget, daemon, worm, hash…




Select/Search for Data

Data Environment

Select Region of Interest



Tools Environment

Select Tools

IDV VisualizerADAS AssimilatorWRF PredictorADaM Data MinerDecoders



Experiments Environment

new load saved


Grid Resources Environment

Select Resource

ESML &Decoding


Conversion



AnalysisProcessing

ADAS Analysis (3D

Gridded Fields) +

Background Fields




WRF Gridded Output

myLEADStorage
















Documents

L inked E nvironments for A tmospheric D iscovery Linked Environments for Atmospheric Discovery (LEAD) Kelvin K. Droegemeier School of Meteorology and