ARSC Initiatives in Weather, ARSC Initiatives in Weather, Climate and Ocean ModelingClimate and Ocean Modeling

Dr. Gregory Newby, Chief ScientistDr. Gregory Newby, Chief Scientist

Arctic Region Supercomputing CenterArctic Region Supercomputing Center

Presentation to the HPC User FormPresentation to the HPC User Form

September 9 2008September 9 2008

Common Themes with Other Common Themes with Other User Forum PresentersUser Forum Presenters

• Lots of work with WRFLots of work with WRF• Work with CCSMWork with CCSM• Mature community-developed applications that scale Mature community-developed applications that scale

reasonably well and are hungry for more CPU powerreasonably well and are hungry for more CPU power• Desire for higher resolution to better approximate Desire for higher resolution to better approximate

realityreality• Desire to add increasingly sophisticated processes to Desire to add increasingly sophisticated processes to

the phenomena under investigationthe phenomena under investigation– Therefore, these items will not be major themes of today’s Therefore, these items will not be major themes of today’s

talktalk

ARSC Themes of potential interest to ARSC Themes of potential interest to the HPC User Forumthe HPC User Forum

• Performance analysis of current (multicore) and Performance analysis of current (multicore) and forthcoming (Cell, GPU, FPGA, T2+) processors for WRFforthcoming (Cell, GPU, FPGA, T2+) processors for WRF

• Quasi-operational WRF with NWS/NOAA usersQuasi-operational WRF with NWS/NOAA users• Wildfire smoke prediction based on WRF/ChemWildfire smoke prediction based on WRF/Chem• Model couplingModel coupling• High resolution ocean forecastsHigh resolution ocean forecasts• Arctic ice tide impactsArctic ice tide impacts• WRF sensitivity analysisWRF sensitivity analysis

– These will be the major themes of today’s talkThese will be the major themes of today’s talk

Multicore Performance Analysis on Multicore Performance Analysis on ARSC’s SupercomputersARSC’s Supercomputers

• Mostly on Midnight (dual core)Mostly on Midnight (dual core)– Sun x2200m2 & x4600 cluster nodesSun x2200m2 & x4600 cluster nodes– 2312 Opteron cores, IB, 4GB/core, Lustre2312 Opteron cores, IB, 4GB/core, Lustre– 12.88 TFLOP theoretical peak12.88 TFLOP theoretical peak

• BenchmarksBenchmarks• Real-world applicationsReal-world applications• Forthcoming: Pingo (quad core)Forthcoming: Pingo (quad core)

– Cray XT5Cray XT5– 3456 cores, SeaStar, 4GB/core, Lustre3456 cores, SeaStar, 4GB/core, Lustre– About 30 TFLOP theoretical peakAbout 30 TFLOP theoretical peak

2-socket AMD64 topology2-socket AMD64 topology

One 1 GHz 16x16 HyperTransport link per supported processor with 8GB/second bandwidth

CPU-0,1 CPU-2,3

8 socket AMD64: Sun Fire 8 socket AMD64: Sun Fire X4600 ServerX4600 Server

8 socket AMD64 topology8 socket AMD64 topology

CPU0

CPU5CPU3

CPU6CPU4

CPU1

CPU2

CPU7

* All HT links are operating at 1GHz and 8GB/s

WRF Performance on WRF Performance on MidnightMidnight

Time required to compute 1 forecast hour on 3-nest domain

Next-Generation Processor Next-Generation Processor PerformancePerformance

• GPU, Cell, quad-core Xeon & Opteron, FPGA, CMTGPU, Cell, quad-core Xeon & Opteron, FPGA, CMT– Many benchmarks completed, some applicationsMany benchmarks completed, some applications

• Forthcoming: WRF test cases for AK domains (submitted Forthcoming: WRF test cases for AK domains (submitted for AGU 2008)for AGU 2008)– GPU (nVidia 9800 GTX), possibly FirestreamGPU (nVidia 9800 GTX), possibly Firestream– FPGA: WRF doesn’t have enough “hot spots,” porting BLAS FPGA: WRF doesn’t have enough “hot spots,” porting BLAS

too much work (but RapidMind might help)too much work (but RapidMind might help)– Cell: QS22 cluster being configured; should work wellCell: QS22 cluster being configured; should work well– UltraSPARC T2+: scaled well to 8 threads per socket; WRF UltraSPARC T2+: scaled well to 8 threads per socket; WRF

is too CPU-bound to benefit much from T2+ CMTis too CPU-bound to benefit much from T2+ CMT

Quasi-Operational and Research Quasi-Operational and Research Weather ModelingWeather Modeling

• Work has been evolving since 2005Work has been evolving since 2005• Initially, efforts were directed at quasi-Initially, efforts were directed at quasi-

operational, scheduled WRF forecasts for the operational, scheduled WRF forecasts for the Fairbanks Weather Forecast Office (WFO). The Fairbanks Weather Forecast Office (WFO). The goal was to leverage lots of CPUs to provide goal was to leverage lots of CPUs to provide high-resolution forecasts, and continually high-resolution forecasts, and continually assess & improveassess & improve

• The scheduled runs continue to dominate our The scheduled runs continue to dominate our efforts, but rigorous analysis and study has efforts, but rigorous analysis and study has yielded important resultsyielded important results

2-way Nested Domains for 2-way Nested Domains for ARSCwrfARSCwrf

200x200x75 cells 421x328x75 cells 151x151x75 cells

Operational WRF – AWIPSOperational WRF – AWIPS

This is the end product of our twice-daily WRF runs, as seen on the NWS Advanced Weather Interactive Prediction System (FAI WFO)

Value to WFO FAIValue to WFO FAI• Grid resolution is Grid resolution is

approximately 5km. approximately 5km. The ability to populate The ability to populate with a reasonably with a reasonably accurate, high-accurate, high-resolution weather resolution weather model saves the model saves the forecasters a lot of forecasters a lot of time. In the best case, time. In the best case, the model output the model output isis the NDFD product.the NDFD product.

Verification ProductsVerification Products• Four days after a Four days after a

forecast, we forecast, we automatically automatically retrieve all retrieve all available available observations and observations and tabulate a tabulate a comparison of the comparison of the WRF forecast for WRF forecast for each location vs. each location vs. the observationsthe observations

Data AssimilationData Assimilation

• Motivation – better set of initial conditionsMotivation – better set of initial conditions• Current initial conditions are interpolated from coarser resolution model runs – Current initial conditions are interpolated from coarser resolution model runs –

clearly much room for errorclearly much room for error• With data assimilation, we perturb our original input grid by carefully With data assimilation, we perturb our original input grid by carefully

assimilating available observations assimilating available observations • We are currently in the process of adding this capability to our operational runs, We are currently in the process of adding this capability to our operational runs,

and once implemented will perform assimilated and non-assimilated runs side-and once implemented will perform assimilated and non-assimilated runs side-by-side for comparisonby-side for comparison

Data AssimilationData Assimilation• Test Case – 48-hour forecast starting at 00Z on 01 May 2007Test Case – 48-hour forecast starting at 00Z on 01 May 2007

Surface obs – temperature, dewpoint,pressure, winds

Vertical soundings from raobs and satellite – temperature, dewpoint, pressure, winds

Alaska Air Quality Impacted by Alaska Air Quality Impacted by WildfiresWildfires

South Fairbanks, July 6, 2004. Air quality particulate level at approximately 10 micrograms/cubic meter.

South Fairbanks, June 28, 2004.Air quality particulate level at approximately 900 micrograms / cubic meter.

Photos courtesy of Dr. James Conner, FNSBhttp://www.dec.state.ak.us/air/am/2004_wf_sum.htm

Initial architecture: WRF/Chem Smoke Dispersion SystemInitial architecture: WRF/Chem Smoke Dispersion System

FIRE EMISSIONSFOFEM

MODIS FIRE DETECTION & BURN AREA

WRF-CHEM PARTICULATE, CHEMICAL &

METEOROLOGICAL FORECAST

WRF-MET METEOROLOGY FORECAST

Gridded Hourly Emissions

PLUME DYNAMICS

FUEL MOISTURE

FIRE SPREAD

Identical WRF domain initialization (SI/WPS)

WRF/Chem

POSTPROCESSINGWRF-Chem netCDF

Static Fuel DataEmission Factors

DEM, Static Fuel

Adapted from a scheme used by our partners from the US Forest Service Fire Science Lab in Missoula.

Fire Detection and Burn Area ExampleFire Detection and Burn Area Example

Example Product for Monday, 8th September 2008, 9:00 UTCExample Product for Monday, 8th September 2008, 9:00 UTC

A weak smoke concentration due to North-Easterly winds and extended fires North/East of the Yukon has been clearly confirmed in the Fairbanks area. Source: smoke.uaf.edu

Coupling Work: Regional Arctic Climate Coupling Work: Regional Arctic Climate Model & MoreModel & More

• Practical ongoing work, DoE funded. Wieslaw Maslowski, PIPractical ongoing work, DoE funded. Wieslaw Maslowski, PI– ARSC is also looking at some more general approaches to coupled climate ARSC is also looking at some more general approaches to coupled climate

models, including adding permafrost, ice sheets, hydrology, ecosystems, human models, including adding permafrost, ice sheets, hydrology, ecosystems, human infrastructure, and coastal erosion. Major partner: International Arctic Research infrastructure, and coastal erosion. Major partner: International Arctic Research Center, UAF. Center, UAF.

• RACM is a regional Arctic climate system model including WRF, RACM is a regional Arctic climate system model including WRF, VIC (land surface model), NAPC (ocean and sea ice model). VIC (land surface model), NAPC (ocean and sea ice model). Current emphasis is coupling WRF, VIC and NAPC via cpl7. Current emphasis is coupling WRF, VIC and NAPC via cpl7.

• Cpl7 is still in development. Our work will keep pace with cpl7 Cpl7 is still in development. Our work will keep pace with cpl7 progress. CCSM4 project groups have made early-release codes progress. CCSM4 project groups have made early-release codes available to RACM partners available to RACM partners

• Since cpl7 is designed for CCSM4, where the framework is Since cpl7 is designed for CCSM4, where the framework is different with WRF and VIC, some changes of cpl7 and WRF are different with WRF and VIC, some changes of cpl7 and WRF are necessary.necessary.

ATM

LND OCN

CPL

ICE

MPI_COMM_WORLD

CPLATM

CPLATMCPLLND

CPLICE

5 COMPONENTS10 COMMUNICATION GROUPS

Partitioning of RACM Processing

ROMS and BeyondROMS and Beyond

• ROMS is an ocean model that can operate at ROMS is an ocean model that can operate at both very large and very small scales, and both very large and very small scales, and takes shorelines into accounttakes shorelines into account

• At ARSC: Ongoing research for northwest At ARSC: Ongoing research for northwest Pacific, Gulf of Alaska, Beaufort Sea & Bering Pacific, Gulf of Alaska, Beaufort Sea & Bering SeaSea

• Also, partnering to couple ROMS with other Also, partnering to couple ROMS with other best-of-breed models for ocean / ice prediction best-of-breed models for ocean / ice prediction

CCSM3 Coupling for CCSM3 Coupling for Ocean / Ice ModelingOcean / Ice Modeling

The ROMS Regional SetupsThe ROMS Regional Setups

NutrientPhytoplanktonZooplankton models

Secondary Producers(Zooplankton)

Fish

NutrientsNO3, NH4…

Primary Producers(Phytoplankton)

Physical Forcing

(Wind, temperatureSunlight, mixing)

ROMS Driving ROMS Driving Ecosystem Model Ecosystem Model

Beaufort Sea WRF Beaufort Sea WRF Sensitivity AnalysisSensitivity Analysis

• Funded by MMSFunded by MMS• Emphasis includes near-coast oil Emphasis includes near-coast oil

spills in Beaufort Sea (north of AK & spills in Beaufort Sea (north of AK & Yukon Territory)Yukon Territory)

• Many scenarios, many WRF runsMany scenarios, many WRF runs– Reanalysis studyReanalysis study– Storm case studiesStorm case studies– Impact of winds on wavesImpact of winds on waves

MMS WRF Report: Higher Resolution not needed MMS WRF Report: Higher Resolution not needed for Effective Near-Shore Wind Fieldsfor Effective Near-Shore Wind Fields

From Field to Supercomputer: Designing Next-From Field to Supercomputer: Designing Next-Generation Sea Ice Modules for Very High Generation Sea Ice Modules for Very High

Resolution Ice-Ocean ModelsResolution Ice-Ocean Models

Arctic System Modeling

Tidal and wind forcing of the ice-ocean boundary layer verses simple wind forcing (14km resolution)

These animations demonstrate the difference between simulations being purely wind driven (right), and those that are both tidally and wind driven (left). Both animations span the same period, and have the same timestep, which is roughly 50 minutes)

ConclusionsConclusions

• Climate, weather and related phenomena (sea ice, land Climate, weather and related phenomena (sea ice, land surface, physical oceanography) models tend to be surface, physical oceanography) models tend to be community developed and supportedcommunity developed and supported– Scale fairly wellScale fairly well– Ported & maintained for new hardwarePorted & maintained for new hardware– Mostly Fortran + MPI, but very complex codesMostly Fortran + MPI, but very complex codes

• Scientists are hungry for higher resolution, longer runs, Scientists are hungry for higher resolution, longer runs, and additional runs to parameterize phenomenaand additional runs to parameterize phenomena

• The Arctic requires several adjustments to models for The Arctic requires several adjustments to models for optimal verisimilitude, versus mid-latitudesoptimal verisimilitude, versus mid-latitudes

Huge THANKSHuge THANKS• Don Morton (ARSC & U. Montana): WeatherDon Morton (ARSC & U. Montana): Weather• Abdullah Kayi (GWU): HyperTransportAbdullah Kayi (GWU): HyperTransport• Martin Stuefer (UAF), Georg Grell (NOAA) & Saulo Martin Stuefer (UAF), Georg Grell (NOAA) & Saulo

Freitas (CPTEC/INPE): Wildfire smokeFreitas (CPTEC/INPE): Wildfire smoke• Kate Hedstrom (ARSC): ROMSKate Hedstrom (ARSC): ROMS• Georgina Gibson (ARSC): NPZGeorgina Gibson (ARSC): NPZ• Wieslaw Maslowski (NPS): POP, RACMWieslaw Maslowski (NPS): POP, RACM• Andrew Roberts (ARSC), Jennifer Hutchings (UAF): Tidal Andrew Roberts (ARSC), Jennifer Hutchings (UAF): Tidal

iceice• Juanxiong He (ARSC): CouplingJuanxiong He (ARSC): Coupling• Jing Zhang, Jeremy Krieger (UAF), Don Morton: MMS Jing Zhang, Jeremy Krieger (UAF), Don Morton: MMS

WRF sensitivityWRF sensitivity– Many other partners & collaboratorsMany other partners & collaborators