1

Status Review of NOAA Implementation of ESMF Architecture (i.e. NEMS)

Mark IredellMarch 26 (#1-12) & April 2 (#12-32), 2008

• NEMS People• NEMS Projects• NEMS Issues

2

NEMS People

• Tom Black Atm, NAM• Huiya Chuang Post• Ed Colon Infrastructure• Mike Ek Land• Jim Geiger Land (NASA)• Bob Grumbine Sea ice• Henry Juang Atm, Dyn, GFS• Sujay Kumar Land (NASA)• Sarah Lu AQ• Avichal Mehra Ocean

• S. Moorthi Atm, Phy, GFS

• Ken Mitchell Land• Christa Peters Land (NASA)• Youhua Tang AQ• Hendrik Tolman Wave• Ratko Vasic Infrastructure• Jun Wang Infrastructure• Weiyu Yang Infrastructure• Mike Young Infrastructure• Shujia Zhou Land (NASA)• Yanqiu Zhu GSI

Biweekly UMIG (Unified Modeling Infrastructure Group) meeting

3

NEMS People

• V. Balaji GFDL• Brian Gross GFDL• Niki Zadeh GFDL

• Stan Benjamin GSD• Tom Henderson GSD• Jin Lee GSD• Jacques Middlecoff GSD

Monthly UMIT (Unified Modeling Infrastructure Telecon) meeting

Includes local UMIG group plus…

Occasionally the telecon invites a much wider group, including DTC and Navy.

4

NEMS Projects• ESMF• NEMS atmosphere• Write history and Post processor• Nesting• Aerosols and Chemistry• Land• Ocean, waves and sea ice• Ionosphere• Ensemble• Data assimilation

5

ESMF

• Earth System Modeling Framework• Community effort, partially supported by

NOAA• Proper ESMF Superstructure required for

all NEMS components• ESMF Infrastructure optional• NEMS will require ESMF 3.1.0r at this time

6

An ESMF Component

An ESMF “Component”“Parent” Driver (User Code)

SetServices()

Initialize()

Run()

Finalize()

ESMF library codeESMF library code

Run()

Import State

Export State

Internal State

Data flow in an ESMF Component Run method

7

ESMF Component rules• Only SetServices is public; Initialize, Run, and Finalize

are private but exposed by SetServices.• Import State is input; Export State is output. Both should

be fully described using ESMF metadata (names, grid, decomposition, etc).

• Coupler Component likely has Import and Export States on different grids. A Coupler Component is user code.

• Internal State is private to the component but persists across calls to Initialize, Run, and Finalize. Even other instances of the same component will have their own private internal state. An access exception may be made to other “friendly” components.

8

ESMF Infrastructure capabilities

ESMF offers or will offer capabilities for decomposed halo update and transposes as well as extensive parallel regridding tools (among other capabilities). The user is free to make use of these or not. Thus far NEMS has taken advantage of existing in-house capability and has minimally made use these ESMF capabilities. This is expected in such transition projects, and ESMF capability will be used as needed.

9

ESMF Portability, Conventions

• ESMF is designed to run on most systems Earth modeling runs on. Plus, ESMF infrastructure will support some ordinary functions that otherwise often hinder portability.

• ESMF supports but does not require the Climate-Forecast metadata conventions.Should NEMS adopt them?– Pro: wide community uses it, including NOAA labs– Con: very long names, but we can make local aliases

10

Units GRIB code

NCEP name NCEP CF

NCEP description CF Standard Name

1 PRES Pa Pa Pressure air_pressure

2 PRMSL Pa Pa Pressure reduced to MSL

air_pressure_at_sea_level

3 PTEND Pa/s Pa s-1

Pressure tendency

tendency_of_air_pressure

5 ICAHT M ICAO Standard Atmosphere Reference Hgt

6 GP m2/s2 m2 s-2 Geopotential geopotential

7 HGT gpm m Geopotential height

geopotential_height

8 DIST m m Geometric height

altitude

9 HSTDV m Standard deviation of height

10 TOZNE Dobson m Total ozone equivalent_thickness_at_stp_of_atmosphere_o3_content

CF convention examples fromhttp://cf-pcmdi.llnl.gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping

11

NEMS Component Library

• Create a NEMS component library (in SVN).• Each component will have some documentation.

Moreover, each component should have a sample MAIN to run it in stand-alone mode for testing. For instance, Dynamics components may have Held-Suarez, Physics may have a single-column model.

• Each type of component will have an assigned NCEP librarian to coordinate the sub-library.

12

NEMS Atmosphere

Atmosphere

Dynamics PhysicsDyn-PhyCoupler

NMM-B

Spectral

FIM

Color KeyComponent class

Coupler class

Completed Instance

Under Development

NAM Phy

GFS PhySimple

unified atmosphereincluding digital filter

Future Development

• The goal is one unified atmospheric component that can invoke multiple dynamics and physics.• At this time, dynamics and physics run on the same grid in the same decomposition, so the coupler literally is very simple.

ARW

FVCORE

FISL

NOGAPS WRF Phy

Navy PhyCOAMPS

Regrid,Redist,Chgvar,Avg, etc

13

Original FIM run() Call Order FIM run:

do timestep = first, last dyn_1 ! 1st half of dyn calls call physics() dyn_2 ! 2nd half of dyn callsend do

Must re-order operations without changing model results

Slide courtesy Stan Benjamin

14

FIM run() Call Re-Ordering Re-order time stepping loop:

do timestep = first, last+1 if (timestep > first) & dyn_2(timestep-1) if (timestep <= last) then dyn_1(timestep) call physics(timestep) endifend do

Slide courtesy Stan Benjamin

15

New FIM run() Call Order Combine dyn_1 + dyn_2 into dyn_run and

push “if” statements inside Push physics() and if statement into

phy_rundo timestep = first, last+1 call dyn_run() call phy_run()end do

Slide courtesy Stan Benjamin

MI: Note NMM and GFS have gone through the exact same process.

16

Write history and Post processor

Atmosphere

Dynamics Physics

Color Key

Completed Instance

Under Development

Future Development

Write

Binary file

NEMSIO

Post

WriteWrite

• The Write component transfers state from model tasks to “quilt” tasks and then writes out from there.• NEMSIO is intended to be a unified optimized parallelized I/O package that can write several formats. Ideally, NEMSIO would be used for all NEMS output data and metadata (not necessarily restart data).• The unified post-processor NCEP_POST will run on the quilt tasks.• Perhaps in the future, the Dynamics and Physics will write their own history files.

unified atmosphere

Component class

Coupler class

17

Nesting

Atmosphere

Color Key

Completed Instance

Under Development

Future Development

• Parent creates children and creates proper boundary conditions in their import states.• Recursively, children create grandchildren.• Children run on different tasks from parent in concurrent nesting, same tasks in sequential nesting.• Two-way concurrent nesting would require a different time integration scheme that needs to be tested.

Atmosphere

1-way conc. NAM-NAM

1-way conc. GFS-NAM

2-way seq.

2-way conc.

moving nest

AtmosphereAtmosphere Atmosphere

AtmosphereAtmosphere

Atmosphere

Component class

Coupler class

18

NEMS Nesting• Eventually we will have all forms.

► The parent domain can have any number of children.

• Begin with 1-way grid-associated static nests.

► Telescoping: Children can have any number of children.

► Domains can run concurrently on unique sets of processors.

These three criteria require the need for general andrepeated splitting of the MPI Communicators. --Done

► One-way / Two-way.

► Static / Moving.

► Grid-associated / Not grid-associated.

Slide courtesy Tom Black

19

ATM_RUN(Fcst_time)DO Timeloop over Fcst_time at interval Δtinternal

IF(My_Compute_Tasks) PREDICT(Δtinternal) IF N_CHILDREN>0

DO N= 1, N_CHILDREN IF(My_Compute_Tasks) ISend BC’s to child IF(Child_Compute_Tasks(N)) Recv from parent CALL ATM_RUN(Child(N), Fcst_time=Δtinternal)

ENDDOIF(Two-way) Send children’s state back to me and blend into

mineENDIF

ENDDOEND ATM_RUN

Nesting – ATM_RUN

Recursive call

“Mini” timeloop

Slide courtesy Tom Black

20

Aerosols and Chemistry

Dynamics Physics

Color Key

Completed Instance

Under Development

Future DevelopmentAerosols

GOCART

• GOCART is Goddard aerosol model. CBM is VOC-NOX-O3 mechanism (chemistry in CMAQ model).• Aerosol and chemistry components compute sources, sinks, and chemical transformation.• Aerosol and chemistry components may need to expose internal state to avoid memory copies.• Atmosphere must be ready to do convective fluxes and removal as well as advection and diffusion.

CBM-5

reduced chem

Atmosphere

WRF-chem

Component class

Coupler class

21

Land (alternative 0)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics Physics Land

LIS-Noah

others

Atmosphere

Component class

Coupler class

22

Land (alternative 0b)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics Physics

Land

LIS-Noah

others

Atmosphere

Component class

Coupler class

Main

23

Land (alternative 1)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics Radiation Land

LIS-Noah

others

PBL/Moist

Atmosphere

Component class

Coupler class

24

Land (alternative 1b)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics RadiationLand

LIS-Noah

others

PBL/Moist

Atmosphere

Component class

Coupler class

25

Land (alternative 2)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics

Atmosphere

Radiation Land PBL/Moist

Physics

Aerosols/Chemistry

Component class

Coupler class

26

Land (alternative 3)Color Key

Completed Instance

Under Development

Future Development

• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?

Dynamics

Atmosphere

Radiation PBL/Moist Aerosols/Chemistry

Land Ice Waves Ocean

Physics

Surface

Component class

Coupler class

27

GEOS-5 AGCM COMPONENT STRUCTURE

AGCM

DYNAMICS

GWDFVCORE

RADIATION

COLUMN PROCESSES

SOLAR

IR

MOIST

TURBSURFACE

CHEM

LAKE

SALTWATER

CAP

(qv, ql, qi, cl,…)

(u, v, T, p)

(Tc, qc, Td,...)

(Ts,Fi...)VEGDYNCATCH

HISTORY

ICELAND(Ts,Fi...)

(O3,…,Qdust,…)

(Tskin,Hskin,...)(Ci,...)

OGCM

Slide courtesy Max Suarez

28

Ocean, waves and sea ice

Atmosphere

Color Key

Completed Instance

Under Development

Future Development

• Under ESMF NEMS, there should be not much difference between SPMD and MPMD codes. However, the MPMD code might not be as portable. MPMD will allow more flexible optimizations though.• Due to some rapidly interacting physics (sea ice thermodynamics and wave roughness), some coupling would have to be frequent.

Atm-OcnCoupler Ocean

GFS Atm-OcnCoupler MOM4

NCEP CFS non-ESMF MPMD

Waves

GFS WW III MOM4

NEMS-GFS HYCOM

Ice

H-ice

N-ice

Earth system model Component class

Coupler class

29

Ionosphere

Atmosphere

Color Key

Completed Instance

Under Development

Future Development

• GFS-IDEA is the GFS extended up to 600 km (from 60 km). GIP is Global Ionosphere-Plasmasphere• The coupling will be frequent and three-dimensional, providing optimization challenges.

Atm-IonCoupler Ionosphere

GFS-IDEA GIPTBD

Whole atmosphereComponent class

Coupler class

30

Ensemble

Atmosphere

Color Key

Completed Instance

Under Development

Future Development

• The atmosphere ensemble coupler provides stochastic forcing for the ensemble.• The current non-NEMS GEFS capability will be converted to NEMS.

Atm ensCoupler

GFS

NEMS-GFS

Atmosphere Atmosphere

Ensemble

NEMS-any

Component class

Coupler class

31

Data assimilation

Model

Color Key

Completed Instance

Under Development

Future Development

• Frequent coupling of atmosphere and analysis will be necessary as hourly analyses are required.• Whole Earth System Nested Ensemble may include any components in NEMS from previous slides.• 4DVAR is supported in this scenario. The inner loop of the variational system could be entirely within the analysis component, or it could invoke adjoint methods with the NEMS atmosphere.• The model-analysis coupler may change grids and variables. When returning to the model state, only the analysis increments will be interpolated.

Data assimilation

Mdl-AnlCoupler Analysis

GFS GSI

NAM

Whole Earth System Nested

Ensemble

Component class

Coupler class

32

NEMS Issues• ESMF version

– NEMS will require ESMF 3.1.0r, which will be the latest public release in spring 2008. NEMS may later need ESMF 3.1.1.

– All components must use the same version• MAPL

– GSFC ESMF wrapper– Used by MOM4, GOCART, FVCORE– Compatible with NEMS?

• Code Repositories– Operationally secure– Collaboration friendly

• Experiment Launcher compatible with operations• Documentation and Support