Linked Environments for Atmospheric Discovery
Linked Environments for Linked Environments for Atmospheric Discovery Atmospheric Discovery
(LEAD)(LEAD)
Linked Environments for Linked Environments for Atmospheric Discovery Atmospheric Discovery
(LEAD)(LEAD)Kelvin K. Droegemeier
School of Meteorology and Center for Analysis and Prediction of Storms
University of Oklahoma
Jay AlamedaNational Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
Linked Environments for Atmospheric Discovery
Geosciences CI ChallengesGeosciences CI Challenges• Enormously complex human-natural system
– Vast temporal (sec to B yrs) and spatial (microns to 1000s of km) scales
– Highly nonlinear behavior
• Massive data sets– physical and digital– static/legacy and dynamic/streaming– geospatially referenced– multidisciplinary and heterogeneous– open access
Linked Environments for Atmospheric Discovery
Geosciences CI ChallengesGeosciences CI Challenges• Massive computation
– weather, space weather, climate, hydrologic modeling
– seismic inversion– coupled physical system models
• Inherently field-based, visual disciplines with the need to manage information for long periods of time
• Bringing advanced CI capabilities to education at all levels
• Connecting the last mile to operational practitioners
Linked Environments for Atmospheric Discovery
• Each year, mesoscale weather – floods, tornadoes, hail, strong winds, lightning, and winter storms – causes hundreds of deaths, routinely disrupts transportation and commerce, and results in annual economic losses > $13B.
Where Where ALLALL These Elements These Elements Converge: Mesoscale WeatherConverge: Mesoscale Weather
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
What Would What Would YouYou Do??? Do???
Linked Environments for Atmospheric Discovery
What What Weather TechnologyWeather Technology Does… Does…Forecast ModelsNEXRAD Radar
Decision Support Systems
Linked Environments for Atmospheric Discovery
What What Weather TechnologyWeather Technology Does… Does…Forecast ModelsNEXRAD Radar
Decision Support Systems
Absolutely Nothing!Absolutely Nothing!
Linked Environments for Atmospheric Discovery
The LEAD GoalThe LEAD GoalProvide the IT necessary to allowProvide the IT necessary to allow
PeoplePeople (scientists, students, (scientists, students, operational practitioners) operational practitioners)
andand
TechnologiesTechnologies (models, sensors, data (models, sensors, data mining)mining)
TO INTERACT WITH WEATHERTO INTERACT WITH WEATHER
Linked Environments for Atmospheric Discovery
The RoadblockThe Roadblock• The study of mesoscale weather is stifled by rigid
IT frameworks that cannot accommodate the – real time, on-demand, and dynamically-adaptive needs of
mesoscale weather research; – its disparate, high volume data sets and streams; and – its tremendous computational demands, which are
among the greatest in all areas of science and engineering
• Some illustrative examples…
Linked Environments for Atmospheric Discovery
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction/Detection
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
Traditional MethodologyTraditional Methodology
STATIC OBSERVATIONS
Radar DataMobile Mesonets
Surface ObservationsUpper-Air BalloonsCommercial Aircraft
Geostationary and Polar Orbiting Satellite
Wind ProfilersGPS Satellites
Linked Environments for Atmospheric Discovery
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction/Detection
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
Traditional MethodologyTraditional Methodology
STATIC OBSERVATIONS
Radar DataMobile Mesonets
Surface ObservationsUpper-Air BalloonsCommercial Aircraft
Geostationary and Polar Orbiting Satellite
Wind ProfilersGPS Satellites
The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!
The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!
Linked Environments for Atmospheric Discovery
The Consequence: Model Grids The Consequence: Model Grids Fixed in Time – No AdaptivityFixed in Time – No Adaptivity
Linked Environments for Atmospheric Discovery
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction/Detection
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
STATIC OBSERVATIONS
Radar DataMobile Mesonets
Surface ObservationsUpper-Air BalloonsCommercial Aircraft
Geostationary and Polar Orbiting Satellite
Wind ProfilersGPS Satellites
The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static
Models Responding to Observations
Linked Environments for Atmospheric Discovery
10 km
3 km
1 km
20 km
Model Dynamic AdaptivityModel Dynamic Adaptivityt = tt = too
Linked Environments for Atmospheric Discovery
t = tt = to o + 6 Hours+ 6 Hours
10 km
3 km
3 km
3 km3 km
10 km
20 km
Linked Environments for Atmospheric Discovery
Today’s Standard Computer ForecastToday’s Standard Computer Forecast
12-hour NationalForecast (coarse grid)
Radar
Radar(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Today’s Standard Computer ForecastToday’s Standard Computer Forecast
12-hour NationalForecast (coarse grid)
Radar
Radar(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Radar(Tornadoes
in Arkansas)
6-hour Mesoscale Forecast
(medium grid)
Radar
Experimental Experimental MesoscaleMesoscale Window Window
Radar
Linked Environments for Atmospheric Discovery
Radar(Tornadoes
in Arkansas)
6-hour Mesoscale Forecast
(medium grid)
Radar
Experimental Experimental MesoscaleMesoscale Window Window
Radar
Linked Environments for Atmospheric Discovery
Radar 6-hour LocalForecast (fine grid)
Experimental Experimental Storm-ScaleStorm-Scale Window Window
Xue et al. (2003)
Linked Environments for Atmospheric Discovery
Dynamic Adaptivity in ActionDynamic Adaptivity in Action
Linked Environments for Atmospheric Discovery
11 h Forecast 11 h Forecast 20 June 200120 June 2001
(6 km)(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
9 h Forecast 9 h Forecast 20 June 200120 June 2001
(6 km)(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
5 h Forecast 5 h Forecast 20 June 200120 June 2001
(6 km)(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
3 h Forecast 3 h Forecast 20 June 200120 June 2001
(6 km)(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
MesoscaleWeather
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Linked Environments for Atmospheric Discovery
MesoscaleWeather
NWS National Static Observations & Grids
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Linked Environments for Atmospheric Discovery
MesoscaleWeather
NWS National Static Observations & Grids
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Local Observations
Linked Environments for Atmospheric Discovery
MesoscaleWeather
NWS National Static Observations & Grids
UsersADaM ADAS Tools
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Local Observations
Linked Environments for Atmospheric Discovery
MesoscaleWeather
NWS National Static Observations & Grids
UsersADaM ADAS Tools
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
MyLEADPortal
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Local Observations
Linked Environments for Atmospheric Discovery
MesoscaleWeather
NWS National Static Observations & Grids
UsersADaM ADAS Tools
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
MyLEADPortal
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Interaction Level I
Local Observations
Linked Environments for Atmospheric Discovery
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction/Detection
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
Traditional MethodologyTraditional Methodology
STATIC OBSERVATIONS
Radar DataMobile Mesonets
Surface ObservationsUpper-Air BalloonsCommercial Aircraft
Geostationary and Polar Orbiting Satellite
Wind ProfilersGPS Satellites
Observing Systems OperateLargely Independent of theWeather – Little Adaptivity
Observing Systems OperateLargely Independent of theWeather – Little Adaptivity
Linked Environments for Atmospheric Discovery
NEXRAD Doppler Weather NEXRAD Doppler Weather Radar NetworkRadar Network
Linked Environments for Atmospheric Discovery
The Limitations of NEXRADThe Limitations of NEXRAD
Linked Environments for Atmospheric Discovery
The Limitations of NEXRADThe Limitations of NEXRAD#1. Operates largely independent#1. Operates largely independent
of the prevailing weather conditionsof the prevailing weather conditions
Linked Environments for Atmospheric Discovery
The Limitations of NEXRADThe Limitations of NEXRAD
#2. Earth’s curvature prevents 72% of the atmosphere below 1 km from being observed
#1. Operates largely independent#1. Operates largely independentof the prevailing weather conditionsof the prevailing weather conditions
Linked Environments for Atmospheric Discovery
The Limitations of NEXRADThe Limitations of NEXRAD
#2. Earth’s curvature prevents 72% of the atmosphere below 1 km from being observed
#1. Operates largely independent#1. Operates largely independentof the prevailing weather conditionsof the prevailing weather conditions
#3. Operates entirely independent from#3. Operates entirely independent fromthe models and algorithms that use its the models and algorithms that use its
datadata
Linked Environments for Atmospheric DiscoverySource: NWS Office of Science and Technology
The Consequence: 3 of Every 4 The Consequence: 3 of Every 4 Tornado Warnings is a False AlarmTornado Warnings is a False Alarm
Linked Environments for Atmospheric Discovery
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction/Detection
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static
DYNAMIC OBSERVATIONS
Models and Algorithms Driving Sensors
Linked Environments for Atmospheric Discovery
New NSF Engineering Research New NSF Engineering Research Center for Adaptive Sensing of the Center for Adaptive Sensing of the
Atmosphere (CASA)Atmosphere (CASA)• UMass/Amherst is lead institution• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
New NSF Engineering Research New NSF Engineering Research Center for Adaptive Sensing of the Center for Adaptive Sensing of the
Atmosphere (CASA)Atmosphere (CASA)• UMass/Amherst is lead institution• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
New NSF Engineering Research New NSF Engineering Research Center for Adaptive Sensing of the Center for Adaptive Sensing of the
Atmosphere (CASA)Atmosphere (CASA)• UMass/Amherst is lead institution• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
UsersADaM ADAS Tools
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
MyLEADPortal
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Linked Environments for Atmospheric Discovery
Experimental DynamicObservations
UsersADaM ADAS Tools
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
MyLEADPortal
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Linked Environments for Atmospheric Discovery
Experimental DynamicObservations
UsersADaM ADAS Tools
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
MyLEADPortal
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Interaction Level II
Linked Environments for Atmospheric Discovery
The LEAD Goal RestatedThe LEAD Goal Restated• To create an integrated, scalable framework
that allows analysis tools, forecast models, and data repositories to be used as dynamically adaptive, on-demand systems that can– change configuration rapidly and automatically in
response to weather;– continually be steered by new data (i.e., the
weather);– respond to decision-driven inputs from users;– initiate other processes automatically; and – steer remote observing technologies to optimize
data collection for the problem at hand;– operate independent of data formats and the
physical location of data or computing resources
Linked Environments for Atmospheric Discovery
CS Challenges/BarriersCS Challenges/Barriers• Workflow
– Dynamic/agile/reentrant
• Data– Synchronization, fault-tolerance, metadata, cataloging,
interchange, ontologies
• Monitoring and performance estimation– Detection of vulnerabilities, recovery, autonomy
• Mining– Grid functionality, scheduling, fault tolerance
Linked Environments for Atmospheric Discovery
Meteorology Challenges/BarriersMeteorology Challenges/Barriers
• “Packaging” of complex systems (WRF, ADAS)• Fault tolerance• Continuous model updating for effective use of truly
streaming observations• Storm-scale ensemble methodologies• Hazardous weather detections based upon gridded
analyses versus use of “raw” sensor data alone• Dynamically adaptive forecasting (models and
observations) – how good compared to current static methodologies?
Linked Environments for Atmospheric Discovery
LEAD ArchitectureLEAD Architecture
DistributedResources
ResourceAccess Services
UserInterface
Desktop ApplicationsLEAD Portal
Portlets
CrosscuttingServices
Con
figu
rati
on a
nd
E
xecu
tion
Ser
vice
s Application ResourceBroker (Scheduler)
Application & Configuration Services
Client Interface
Dat
a S
ervi
ces
Wor
kfl
ow S
ervi
ces
Cat
alog
Ser
vice
s
Linked Environments for Atmospheric Discovery
LEAD ArchitectureLEAD Architecture
DistributedResources
ComputationSpecialized
ApplicationsSteerable
Instruments Storage
Data Bases
ResourceAccess Services GRAM
Globus
SSH
Scheduler
LDM
OPenDAP GenericIngest Service
UserInterface
Desktop Applications• IDV• WRF Configuration GUI
LEAD Portal
PortletsVisualization Workflow Education
Monitor
Control
Ontology Query
Browse
Control
CrosscuttingServices
Authorization
Authentication
Monitoring
Notification
Con
figu
rati
on a
nd
E
xecu
tion
Ser
vice
s WorkflowService
ControlService
QueryService
StreamService
OntologyService
MyLEAD
WorkflowEngine/Factories
VO Catalog
THREDDS
Application ResourceBroker (Scheduler)
Host Environment
GPIR
Application Host
Execution Description
Applications (WRF, ADaM,IDV, ADAS)
Application Description
Application & Configuration Services
Client Interface
Observations• Streams• Static• Archived
Dat
a S
ervi
ces
Wor
kfl
ow S
ervi
ces
Cat
alog
Ser
vice
s
Decoder/ResolverService
RLSOGSA-
DAI
Linked Environments for Atmospheric Discovery
Key System Components and Key System Components and TechnologiesTechnologies
Capability/Resource Principal Technologies
Atmospheric, Oceanographic, Land-Surface Observations
CONDUIT, CRAFT, MADIS, IDD, NOAAPort, GCMD, SSEC, ESDIS,
NVODS, NCDC
Operational Model Grids CONDUIT, NOMADS
Data Assimilation Systems ADAS, WRF 3DVAR
Atmospheric Prediction Systems WRF, ARPS
Visualization IDV
Data Mining ADaM
NSF NMI Project Globus Tool Kit
Semantic Interchange and Formatting ESML, NetCDF, HDF5
Adaptive Observing Systems (Radars) CASA OK Test Bed, V-CHILL
LEAD Portal NSF NMI Project (OGCE)
Workflow Orchestration BPEL4WS
Monitoring Autopilot
Data Cataloging/Management THREDDS, MCS, SRB
Linked Environments for Atmospheric DiscoveryThe Driver: Canonical Research & Education Problems
The LEAD Research ProcessThe LEAD Research Process
Fundamental Scientific and Technological Barriers
System Functional Requirements and Capabilities
System Architecture and Definition of Services
Building Blocks
Technology Generations
End User Focus Group Testing and Deployment
The End Game: Canonical Research & Education Problems
Basic Research Prototypes Test Beds
Linked Environments for Atmospheric Discovery
Generation 2DynamicWorkflow
Generation 3AdaptiveSensing
Generation 3AdaptiveSensing
Generation 2DynamicWorkflow
Generation 1Static
Workflow
Generation 1Static
Workflow
Generation 2DynamicWorkflow
Generation 1Static
Workflow
Year 1 Year 2 Year 3 Year 4 Year 5
LEADLEAD Technology Generations Technology GenerationsT
echn
olog
y &
Cap
abil
ity
Generation 1Static
Workflow
Generation 1Static
Workflow
Look-Ahead Research
Look-Ahead Research
Linked Environments for Atmospheric Discovery
In LEAD, Everything is a ServiceIn LEAD, Everything is a Service• Finite number of services – they’re the “low-level” elements
but consist of lots of hidden pieces…services within services.
Service A(ADAS)
Service B(WRF)
Service C(NEXRAD Stream)
Service D(MyLEAD)
Service E(VO Catalog)
Service F(IDV)
Service G(Monitoring)
Service H(Scheduling)
Service I(ESML)
Service J(Repository)
Service K(Ontology)
Service L(Decoder)
Many others…
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to Start by Building Simple Prototypes to Establish the Services/Other Capabilities…Establish the Services/Other Capabilities…
Service C(NEXRAD Stream)
Service F(IDV)
Service L(Decoder)
Prototype X
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to Start by Building Simple Prototypes to Establish the Services/Other Capabilities…Establish the Services/Other Capabilities…
Service C(NEXRAD Stream)
Service F(IDV)
Service L(Decoder)
Prototype Y
Service D(MyLEAD)
Service E(VO Catalog)
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to Start by Building Simple Prototypes to Establish the Services/Other Capabilities…Establish the Services/Other Capabilities…
Service C(NEXRAD Stream)
Service F(IDV)
Service L(Decoder)
Prototype Z
Service A(ADAS)
Service I(ESML)
Service J(Repository)
Service D(MyLEAD)
Service E(VO Catalog)
Linked Environments for Atmospheric Discovery
Service B(WRF)
Service A(ADAS)
Service C(NEXRAD Stream)
Service D(MyLEAD)
Service L(Mining)
Service L(Decoder)
Service J(Repository)
……and then Solve General Problemsand then Solve General Problemsby Linking them Together in Workflowsby Linking them Together in Workflows
Linked Environments for Atmospheric Discovery
Service B(WRF)
Service A(ADAS)
Service C(NEXRAD Stream)
Service D(MyLEAD)
Service L(Mining)
Service L(Decoder)
Service J(Repository)
……and then Solve General Problemsand then Solve General Problemsby Linking them Together in Workflowsby Linking them Together in Workflows
Note that these servicescan be used as stand-alonecapabilities, independent of
the LEAD infrastructure(e.g., portal)
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Feedback from Application ScientistsFeedback from Application Scientists
Benefits• Single sign-on feature is very handy• Secured access to compute resources from a
browser, is increasing productivity
Difficulties• Grid authentication is not trivial to use - important
feature needed by an application scientist• Hard to keep track of continuously evolving grid
middleware• System needs continuous development as
middleware on production machines moves forward and is not backward compatible
Canonical Problem #3
Problem #3: Dynamically Adaptive, High-Resolution Nested Ensemble Forecasts
Goal: For the continental United States (CONUS), automatically generate a 1-km grid spacing ADAS analysis every 30 minutes, and a 6-hour, 2-km grid spacing CONUS forecast every 3 hours. Automatically launch finer-grid spacing nested WRF ensemble forecasts when data mining algorithms – applied to both the CONUS analyses and forecasts – detect features indicative of storm potential (e.g., convergence lines, strong instability, incipient convection) or actual storm development. Conduct rigorous post-mortem assessment of statistical forecast skill and compare the high-resolution nested grid forecasts with the single-grid CONUS run at coarser resolution.
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
START
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data MiningSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data Mining STOPSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data Mining STOPSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data Mining STOP
Adjust ForecastConfiguration and
Schedule ResourcesSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data Mining STOP
Adjust ForecastConfiguration and
Schedule ResourcesSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts
Linked Environments for Atmospheric Discovery
How Would One Go How Would One Go About Setting This About Setting This
Up in LEAD??Up in LEAD??• The “First LEAD Commandment”
– Thou shalt not use unintelligible computer science jargon in the portal for describing options/tasks to end users
– Foo, portlet, ontology, widget, daemon, worm, hash…
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Select/Search for Data
Data Environment
Select Region of Interest
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Tools Environment
Select Tools
IDV VisualizerADAS AssimilatorWRF PredictorADaM Data MinerDecoders
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Experiments Environment
new load saved
Linked Environments for Atmospheric Discovery
Grid Resources Environment
Select Resource
ESML &Decoding
Remapping, Gridding,
Conversion
ADAS Quality Control
ADAS Quality Control ADAS
AnalysisProcessing
ADAS Analysis (3D
Gridded Fields) +
Background Fields
ADAS-to-WRF Converter
3D Gridded Fields in WRF Mass Coordinate + Suite
of Ensemble Initial Conditions
WRF Gridded Output
myLEADStorage
Multiple Copies of WRF Forecast Model Running Simultaneously
Canonical Problem #3
Meta Data Creation and Cataloging
Visualization & Data Mining STOP
Adjust ForecastConfiguration and
Schedule ResourcesSTART
Define Data Requirements and Query for Desired Data
Allocate Storage and Move/Stream Data to Appropriate Location (e.g., PACI Center)
Allocate Computational Resources
DataSurface Observations
Upper-Air ObservationsCommercial Aircraft Data
NEXRAD Radar DataSatellite Data
Wind Profiler DataLand Surface Data
Terrain DataBackground Model Fields
and Previous Forecasts