Upload
yahoo-developer-network
View
1.069
Download
2
Tags:
Embed Size (px)
Citation preview
Yahoo! Hadoop India Summit, Indian Institute of Science
Middleware Frameworks for Adaptive Executions and
Visualizations of Climate and Weather Applications on Grids
Sathish VadhiyarGrid Applications Research LabSupercomputer Education and Research CentreIndian Institute of ScienceBangalore
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Outline Parallel Simulation and Visualization
◦Resource Constraints◦Impact on Climate Simulations
Adaptive Integrated Framework◦Framework◦Contradictory Objectives
Decision Algorithm Steering the Visualizations Results
◦Progress of Simulation and Visualization◦Adaptation of Parameters
Potential for Cloud Computing
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Parallel Simulation and Visualization
Critical climate applications like cyclone tracking require High-fidelity high-resolution simulation
◦ High-performance computations◦ Massive amount of output
On-the-fly remote visualization◦ Real-time guidance to policy and decision makers◦ Joint analysis by geographically distributed climate
scientists
High-performancesimulations
Remotevisualization
Figure: Simultaneous simulation and remote visualization using stable storage
Parallel I/O
NetworkDISK
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Resource Constraints
SIM VIS
Simulation
Process
Visualization Process
Stable Storage
Network
• High computation rate• High I/O bandwidth• Limited network
bandwidth• Limited storage space
Figure: Illustration of resource constraints on simulation
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Impact on climate simulations
Rapid accumulation of data in the stable storage
Eventual unavailability of storage Stalling of simulation Low temporal resolution Loss of visualization
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Adaptive Integrated Framework
APPLICATIONMANAGER
APPLICATIONCONFIG
JOB HANDLER
Output Frequency
# Processors
FR
AM
E S
EN
DER
FR
AM
E R
EC
EIV
ER
SIMULATIONPROCESS
ApplicationConfiguration
VISUALIZATIONPROCESS
Stall if no disk space
Network
o Invokes a decision algorithm periodicallyo Reacts to significantly low disk space
o Schedules climate simulation applicationo Starts, stops, restarts simulation process
o Simulates climate across time steps
o Outputs climate data to storage
o Visualizes simulation output
Periodic Invocation
DECISIONALGORITHM
o Adapts to resource and application dynamics
o Determine near-optimal parameters
Storage
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Decision Algorithm
Objectives1. Maximize rate of simulation2. Maximize temporal resolution3. Enable continuous visualization4. Ensure availability of storage
Contradictory Objectives
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Decision Algorithm
Input◦ Simulation resolution◦ Network bandwidth◦ Remaining disk space
Output◦ Number of processors for simulation◦ Output frequency
Optimization Based Algorithm
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Optimization-based Approach
Maximize simulation rate within the constraints related to continuous visualization, acceptable output frequency, I/O bandwidth, disk space and network bandwidth
Causes of faster consumption of storage space Faster execution time Limited network bandwidth High frequency of output
Objectives Optimal processor allocation Best possible output frequency Judicious use of storage
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Problem FormulationObjective function: minimize t
IO
OS t F T T
b (1)
Time Constraint: Time to solve + Time to output ≤ Time to transfer
t Time to solve one simulation time step
S Number of frames solved in an interval I
F Number of frames output in an interval I
T Number of frames transferred in an interval I
Table: Decision Variables
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Constraints
LBt T
LB z UB
Bound Constraints: Bounds for t and z
(4)
(5)
.in out
IO
Dn
R R
O F Db
S t F T n
Disk Constraint: Net input to the disk ≤ Remaining disk space
(2)
(3)
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Experiments Simulation: Weather Research and Forecasting Model v3.0.1 Visualization: VisIt v1.12.0
Climate Application: Tracking Cyclone Aila Modeled area: 32x106 sq. km. from 60ºE - 120ºE and 10ºS - 40ºN Formed: 23th May 2009, Dissipated: 26th May 2009
Figure: Visualization of Perturbation Pressure showing the track of Aila
Pressure (hPa) 995 994 992 990 988 986
Resolution (km) 24 21 18 15 12 10
Table: Resolutions for different Pressure Values February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Experiments
Configuration
Simulation Configuration
Maximu
mCores forSimulatio
n
MaximumDisk
SpaceUsed
AverageSim-Vis
Bandwidth
inter-department
fire: dual-core AMD Opteron 2218 (Gigabit Ethernet)
48 182 GB 56 Mbps
intra-country
gg-blr: Intel Xeon Quad Core Processor X5460 (Infiniband)
90 150 GB 40 Mbps
cross-continent
moria: dual-core AMD Opteron 265 (Gigabit Ethernet)
56 100 GB 60 KbpsTable: Simulation and Visualization Configurations
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Simulation Progress
Figure: For cross-continent configuration
Simulation stalls in Greedy-Threshold approach
Faster rate of simulation
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Visualization Progress
Figure: For intra-country configuration
Lags behind in attempt to
visualize every time step initially
Faster rate of visualization
INCREASING LAG
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Disk Space Utilization
Figure: For intra-country configuration
Higher rate of disk space
consumption
Less than 50% disk
space used
February 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
Adaptivity
Figure: For inter-department configurationFebruary 16, 2011
Yahoo! Hadoop India Summit, Indian Institute of Science
February 16, 2011
Steering the Visualization
Yahoo! Hadoop India Summit, Indian Institute of Science
February 16, 2011
Steering Across the Ocean!
Changing number of procs from 96 to 80
Changing Visualization Frequency
Changing Resolution of Simulation
Auto-changing number of procs to maintain QoS
Yahoo! Hadoop India Summit, Indian Institute of Science
Ship the simulations to a cloud Use resource management services of clouds to
find a “nearby” large storage This will eliminate the storage problem/constraint But new research challenges:
◦ Storage can spill over; Need to maintain metadata of storage repositories
◦ Simulation->Storage->Visualization will now involve multiple hops
◦ Hence added benefits due to large storage-as-service in cloud will have to balanced against loss in performance
February 16, 2011
Potential for Clouds
Yahoo! Hadoop India Summit, Indian Institute of Science
The infrastructure has to be expanded to include multiple simultaneous multi-user visualizations of multiple independent simulations
Such independent simulations are natural for executions on clouds.
February 16, 2011
Potential for Clouds
Yahoo! Hadoop India Summit, Indian Institute of Science
To minimize lag between simulation and visualization site – choosing representative frames
Multiple visualization-simulation framework Applying for other applications
February 16, 2011
Future Work
Yahoo! Hadoop India Summit, Indian Institute of Science
Preeti Malakar (Phd student) Dr. Vijay Natarajan (Co-researcher)
February 16, 2011
Acknowledgements
Yahoo! Hadoop India Summit, Indian Institute of Science
February 16, 2011