WRF4G The Weather Research Forecasting model workflow for the GRID [email protected] Department of Applied Mathematics & Computer Sciences University of

WRF4GThe Weather Research Forecasting model workflow for the GRID

[email protected] of Applied Mathematics

& Computer SciencesUniversity of Cantabria, Spain

GRID developers: Valvanuz Fernández, Antonio S. Cofiño

Application developers: Jesús Fernández, Lluís Fita

Santander Meteorology GroupA multidisciplinary approach to weather & climate

http://www.meteo.unican.es

EGU General Assembly 2010, Wien, 2-7 May 2010

A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es

Numerical Weather Prediction model (regional)

Open source (Fortran90).

A world community with 6000+ registered users.

Application in a wide variety of studies: Weather forecasting (operational at NCEP)

Data assimilation studies

Regional climate studies

Idealized simulations

Educational applications,....

WRF Model


Computing and storage requirements: Intensive use of CPU

Supports a variety of parallel programming paradigms (OpenMP, MPI, serial,...) . It's a common benchmarking application for HPC.

Large amount of input, output and restart data.

Typical experiments last for days (even in parallel).

Application with a complex workflow (preprocessing, execution & postprocessing)

WRF Requirements

metgrid wrf

geogrid

ungrib

WRF Preprocessing System WRF ARW

WRFGEL

namelist.inputnamelist.wps

real

WR

F W

OR

KF

LOW


Reanalisys/Reforecasting

High number (~104) of independent simulations

High volume of output-data (>TB)

Requires scalability

Regional projections for climate change

Contiguous simulations (~10), weeks of walltime each.

High volume of output data (>10TB)

Recovering system for simulation restart

Weather Forecasting

QoS and optimal resources: deadline for delivering

Sensitive studies for Climate and Weather

Physical schemes, initial conditions and boundary conditions uncertainties sampling

Resource demanding experiments

Scientific experiments WRF


GOALS

Make easier for the user the process of design, execution and monitoring of the experiments with WRF.

Develop a framework that allow the user to use at the same time different computing resources in a transparent way:

Local clusters (PBS, SGE, LoadLeveler,…) Grid infrastructures

(gLite, Globus,…) Local resources

(SSH, fork, …).

Develop a set of command line tools and a Web portal for the WRF users.

WRF4G: Goals


CORDEX (COordinated Regional climate Downscaling EXperiment) is a framework to improve coordination of international efforts in regional climate downscaling research. CORDEX was initiated as a result of the Task Force on Regional Climate Downscaling, formed by the World Climate Research Program (WCRP).

A set of target regions has been proposed and modeling groups willing to contribute must comply with simulations specifications.

CORDEX will produce an ensemble of simulations sampling uncertainties related to: (i) varying Global Climate Model (GCM) simulations; (ii) varying greenhouse gas (GHG) concentration scenarios; (iii) natural climate variability; and (iv) different downscaling methods.

http://wcrp.ipsl.jussieu.fr/RCD_Projects/CORDEXhttp://www.meteo.unican.es/wiki/cordexwrf

There are currently 15 groups planning to contribute to CORDEX with WRF, 5 groups planning to contribute the African domain (key region for the AR5). All of them could benefit from WRF4G app.

Role of WRF4G in CORDEX


150.000 CPUs

70 PB

260 sites worldwide

Arquitectures: i386,x86_64

LRMS: torque, sge, lsf, bqs

Shared and not shared Home

Computing resources: OS: Debian, SL/Centos 4 and 5.

Memory: 250MB to 16GB

Processor: P4 to i7

Different queues limitations: walltime, memory & disk quotas…

Bandwidth in some sites is very small.

EGEE: Example of big Grid infrastructure


GRID Challenges:

Develop an application adapted to run in different arquitectures, OS, LRMS and parallel environments.

Repository of WRF binaries adapted to those environments

Detect sites that not accomplish application requirements: Memory, Processor, queue time limitations,…

Optimizing the data transfers and replication between resources distributed geographically.

Develop a monitoring tool for experiments status.

Create a checkpointing management system that allow to restart simulations.

Failure detection and recovery.

WRF4G: GRID Challenges


WRF4G is a port of the WRF Modeling System prepared to run in GRID environments.

WRF has been splitted in 2 layers to separate the scientific experiment from the infrastructure details (wrf.input & wrf4g.conf).

WRF4G supports several data transfer services (gsiftp, rsync,…) and execution systems (gLite, globus, PBS, SGE,…). It can be run in the user’s machine.

No deployment required on sites. The application is deployed by a pilot job.

WRF for GRID: WRF4G


metgrid real wrf

geogridStatic

GeographicalData

ungribGridded Data:

NAM, GFS, RUC,AGRMET, etc.



WRF4G Workflow


metgrid real wrf

geogridStatic

GeographicalData

ungribGridded Data:


WRF Preprocessing System WRF ARWProcessed locally

wrf4g.conf wrf.input


WRF4G

WRF4G Workflow


metgrid real wrf

geogridStatic

GeographicalData

ungribGridded Data:



WRFGEL

WRF4G.sh

Processed locally

WRF4G_ini.sh



wrf4g_submitter.sh

Job submission abstraction layer

WRF4G

UI

WN

Inp

ut

da

ta

ab

stra

ctio

n la

yer

WRF4G Workflow


metgrid real wrf

geogridStatic

GeographicalData

ungribGridded Data:



WRFGEL

WRF4G.sh

Processed locally

WRF4G_ini.sh



wrf4g_submitter.sh

Job submission abstraction layer

WRF4G

UI

WN

WRF4G-0.0.2.tgzWRF4Gbin-3.1r83.tgz

wrfout_*wrfrst_*

wrfinput_*wrfbdy_*

WRF4G_BASEPATHWRF4G_INPUT

WRF4G_APPS

Inp

ut

da

ta

ab

stra

ctio

n la

yer

geo_em.ncnamelist.wps

WRF4G_DOMAINS

WRF4G Workflow


${WRF4G_ROOT}

scripts

ui wn

WRFGEL WPS WRFV3bin lib

examples

WRF4G_ini.shwrf4g_submitter.shwrf4g_make_tarball.shwrf4g_make_tarball_bin.shwrf4g_submit.EELA_grid_jobwrf4g_submit.MDMclusterwrf4g_submit.MDMclusterIFBwrf4g_wrfgel_environment

create_output_structuredownload_fileexist_wpsget_date_restartregister_file

cccantabria wrf.input wrf4g.confdomain_partition wrf.input.in wrf4g.conf.innino50 wrf.input wrf4g.confoperativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf

vcpfortnmlncdumppreprocessor.*

WRF4G.sh

WRF4G structure


These are sample configuration files for a WRF4G experiment

${WRF4G_ROOT}

scripts

ui wn


examples





WRF4G.sh

WRF4G structure


These are user scripts launched from the UI

${WRF4G_ROOT}

scripts

ui wn


examples





WRF4G.sh

WRF4G structure


This is what the Worker Node sees just before runningWRF4G.sh

${WRF4G_ROOT}

scripts

ui wn


examples





WRF4G.sh

WRF4G structure


User Interface - my_experiments/exp

wrf4g.confwrf.input

Create and run an experiment


User Interface - my_experiments/exp

exp__... exp__reaN

0001 0002 ... 0001 0002 ... 0001 0002 ...

wrf4g.confwrf.input

sandbox.tgzwrf.chunkwrf.input

wrf4g.confWRF4G_ini

exp__rea1

realizations



time

Three realizations split into two chunks each:

exp__rea1

exp__rea2

exp__rea3

Storage Element - ${WRF4G_BASEPATH}/experiments/exp

exp__rea1 exp__rea2 exp__rea3

output restart wpsout output restart wpsout output restart wpsout

1 2 30 4 5 6



time

exp__rea1

exp__rea2

exp__rea3

Storage Element - METEO4G/WRF/experiments/exp



1 2 30 4 5 6

wrfinput_t0wrfbdy_t0




time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6

wrfout_t0wrfout_t1





time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6

wrfout_t0wrfout_t1wrfout_t2 wrfrst_t2





time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6

wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4

wrfrst_t2

wrfrst_t4





time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6


wrfrst_t2

wrfrst_t4






time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6


wrfrst_t2

wrfrst_t4






time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6

wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5

wrfrst_t2

wrfrst_t4






time

exp__rea1

exp__rea2

exp__rea3




1 2 30 4 5 6

wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5wrfout_t6wrfout_t7wrfout_t8

wrfrst_t2

wrfrst_t4

wrfrst_t6

wrfrst_t8






time

exp__rea1

exp__rea2

exp__rea3





wrfrst_t2

wrfrst_t4

wrfrst_t6

wrfrst_t8



1 2 30 4 5 6


wrfrst_t2

wrfrst_t4

wrfrst_t6

wrfrst_t8




wrfrst_t2

wrfrst_t4

wrfrst_t6

wrfrst_t8






To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:

Data Management vcp: Provides transparent copy between any of the

following protocols: gsiftp, LFC, rsync, local copies

vcp -r /local/dir gridftp://server:port/remote/pathvcp gsiftp://srv1:port/rmt/file gsiftp://srv2:port/other/filevcp /local/file1 rsync://server/other/local/path/vcp gsiftp://srv1:port/rmt/file ln:link

WRF4G architecture (I)


To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:

Execution Management Through the use of plugins, the user can submit jobs to a

different infrastructures: Grid (globus 4.2 and glite CE), local clusters (PBS, SGE, SLURM,…), local resources.

Prepared to use different execution environments: OpenMP and MPI.

Strong scheduling policy based in history records and resources characteristics.

Failure detection and recovery.

WRF4G architecture (II)


The application is been supported in EELA-2, EGEE & NGI-ES infrastructures and GT2.4 & GT4.

A release candidate (v1.0 RC1) has been launched to run realistic experiments. It consist on a virtual machine UI with all the application components.

Collaborations with end-users in LA (CETA-CIIFEN-

UPS), Europe (CESGA-MeteoGalicia) for feedback and Asia (HAII, Thailand).

Currently we are testing MPI support in the GRID.

Daily operational version running on the GRID for ensemble of weather forecasts.

Conclusions & Current Work


Develop a monitoring system oriented to app status.

Execution management works on Globus and gLite CE based Gridway plugins. PBS y SGE are used, but plugins will developed for better integration with Gridway.

Develop an scalable replica management integrated with Gridway. (bottleneck in GRID infrastructures).

Create a user portal that allow the users managing and monitoring their experiments.

Collaborate with EGI and other International GRID Initiatives like EUAsiaGrid

Incorporate users feedback

Future Work


Thanks [email protected]

Documents

WRF4G The Weather Research Forecasting model workflow for the GRID [email protected] Department of Applied Mathematics & Computer Sciences University of