www.esmf.ucar.edu

Cecelia DeLuca, Don Stark, Chris HillArctic System Model WorkshopMay 20, 2008

Earth System Modeling Framework

www.esmf.ucar.edu

Outline

• About ESMF• Best Practices, On and Off List• Governance• Personal History• Conclusion

www.esmf.ucar.edu

About ESMF

• ESMF provides component wrappers with standard interfaces, a set of data structures for data exchanges, and customizable drivers.

• ESMF provides common utilities, such as data communications, regridding, time management, configuration, and message logging.

• Goals are component interoperability and software reuse.

www.esmf.ucar.edu

GEOS-5

surface fvcore gravity_wave_drag

history agcm

dynamics physics

chemistry moist_processes radiation turbulence

infrared solar lake land_ice data_ocean land

vegetation catchment

coupler

coupler coupler

coupler

coupler

coupler

coupler

• Each box is an ESMF component• ESMF State objects carry data between components• Every component (including couplers) has a standard interface to facilitate exchanges• The ESMF architecture enables the assembly of many different systems

ESMF Application Example

GEOS-5 AtmosphericGeneral Circulation ModelApplication Example

www.esmf.ucar.edu

Standardization of Component APIs

• There are only three ESMF component methods: Initialize, Run, and Finalize (I/R/F)

• Users create a Component by assigning their user code I/R/F methods to an ESMF Component type

• The ESMF Component calls back into the specific user-assigned methods

• I/R/F methods cascade down the tree• Small set of standard arguments:

Child GridComp “Atmosphere”

Parent GridComp “Hurricane Model”

Finalize

Child GridComp “Ocean”

Finalize

Child CplComp “Atm-Ocean Coupler”

Finalize

Call Initialize Call FinalizeCall Run

Initialize Run Finalize

Initialize

Initialize

Initialize

Run

Run

Run

AppDriver (“Main”)

Call Initialize Call FinalizeCall Run

call ESMF_CompRun (myComp, importState, exportState, clock, phase, blockingFlag, rc)

www.esmf.ucar.edu

Standardization of Data Structures• ESMF State data structures contain all data exchanged between

components• Flexibility in data representation - States can contain lists of varied

data structures, including Arrays, ArrayBundles, Fields, FieldBundles, and other States

The ESMF philosophy is to constrain the number of component methods and their arguments, but retain flexibility in the data structures used in inter-component exchanges

www.esmf.ucar.edu

ESMF Adoption - PARSSSEPrepare user code• Split user code into initialize, run and finalize methods and decide on components and coupling

fields and control flowAdapt data structures• Wrap native data structures either as

◦ ESMF Arrays (represent data in index space only)Regrid via user-supplied weights + sparse matrix multiply

◦ ESMF Field objects (Arrays + coordinates and metadata) Regrid via on-line weight generation

Pack Arrays or Fields into States.Wrap time information in ESMF Clocks.

Register user methods• Attach user code initialize, run and finalize methods to ESMF Components through a registration

processSchedule, Synchronize, and Send data between components• Write couplers using ESMF redistribution, sparse matrix multiply, regridding, or user-specified

transformationsExecute the application• Run components using an ESMF driver

www.esmf.ucar.edu

ESMF Earth Science Components & Models

HAF

GAIM

www.esmf.ucar.edu

ESMF Timeline • First phase started in 2002 with NASA funding• Second phase began in 2005 with a transition to multi-agency support and

management – NASA, NOAA, NSF, and DoD sponsors• ESMF component interfaces are the basis of programs at multiple agencies:

◦ DoD Battlespace Environments Institute (BEI)◦ NOAA National Environmental Modeling System (NEMS) ◦ NASA Modeling Analysis and Prediction Program for Climate Variability

and Change (MAP)◦ Community Sediment Transport Model (ONR), others…

• In operational use since August 2006, initially at the National Weather Service, ongoing transition of other ESMF coupled systems to operations

• Scope of current work will be completed in 2010• Ongoing diversified funding expected

www.esmf.ucar.edu

Release Path2002 2003 2004 2005 2006 2007 2008 2009 2010

ESMF v1Prototype

ESMF v2Components, VM and UtilsESMF_GridCompRun()

ESMF v3Index Space OperationsESMF_ArraySparseMatMul()

ESMF v4Grid OperationsESMF_GridCreate()ESMF_FieldRegrid()

ESMFv5StandardizationBuild, init, data types, error handling, …

Public releaseESMF v3.1.0r

Full system prototype

Building bottom-up

Standardization

www.esmf.ucar.edu

ESMF Package Description, the Basics• Open source, open development (browsable repository)• Components can run concurrently, sequentially or in mixed

mode• Serial or parallel• Single or multiple executable or combinations• Shared or distributed memory or hybrid • Support for model ensembles, including execution of multiple

instances in the same address space• 2000+ unit tests, system tests, and examples regression tested

nightly on 26+ platform/compiler combinations• Reference Manual, Users Guide and the examples therein

updated automatically with code changes

www.esmf.ucar.edu

ESMF Package Description, Coupling• Data transformations can be executed within a coupler

component, or arranged in a coupler component and executed directly between model components

• Coupling can be done in index space or physical space• Performance: < 5% overhead in time to solution vs customized

native approaches, highly scalable in performance and memory (performance reports online)

www.esmf.ucar.edu

Upcoming ReleasesPublic Release 3.1.0r, May 2008

• Main capabilities added since last public release - 2.2.2rp3, December 2006:

◦ General grid and data representation and parallel regridding support through an Array index space layer (user supplied interpolation weights) – logically rectangular grids, multi-patch grids, unstructured grids

◦ Representation of rectilinear and curvilinear grids in a Grid class (preparation for framework-generated interpolation weights)

◦ New ports including gfortran and Solaris

• Compact, simpler interface: cut number of public data handling methods in half (304 -> 147, with caveats)

www.esmf.ucar.edu

Upcoming Releases (cont.)Internal Release 3.1.1, June 2008

• Initial implementation of observational data streams

• Generation of regridding interpolation weights for logically rectangular grids, bilinear and higher order method

• Attribute class can store standardized metadata packets and represent metadata hierarchies - e.g., State metadata includes the metadata of Fields that are stored in it

• Initial implementation of unstructured grid interface

Patch Release 3.1.0rp*, Summer 2008

• Petascale optimizations for Array and ArrayBundle sparse matrix multiply

www.esmf.ucar.edu

Relationship to CCSM

• Passed CCSM Stage 1 evaluation of ESMF in May 2007, showing < 5% overhead, comparable memory use, no change to component internals.

• Result: creation of cpl7 based on ESMF design• Currently working on Stage 2 evaluation:

Creation of an ESMF version of cpl7 called ESMCoupler• Expected advantages:

◦ Greater interoperability with other components◦ More flexible architecture that doesn’t rely on a single central coupler◦ On-line regridding◦ Integrated handling of metadata

www.esmf.ucar.edu

Team CompositionTeam composition (blue is off-site)

1. Manager (agency coordination and technical coordination)2. Operations Manager (website, metrics, space issues, local administration)3. Integrator/Test lead (regression testing, release management)4. Tester for numerical methods

7 developers:5. Systems level/porting/build6. Low level data structures and architecture 7. Structured grids 8. Meshes9. High level numerical data structures 10. Language interfaces11. Utilities including calendaring12. Metadata and attributes

External and related13. External: ½ FTE performance testing14. Related: FTE for component distribution portal

www.esmf.ucar.edu

Distributed Development

GOAL - Everybody on the team has access to all information, current and past

• Archived email list where all development correspondence gets cc’d• Frequent telecons with minutes• Web browsable repository, mail summary on check-ins• Daily archived test results• Monthly archived metrics• Public archived trackers (bugs, feature requests, support requests,

etc.)

www.esmf.ucar.edu

ESMF Governance

Executive BoardStrategic Direction

Organizational ChangesBoard Appointments

Interagency Working GroupStakeholder Liaison

Programmatic Assessment & Feedback

Advisory BoardExternal Projects CoordinationGeneral Guidance & Evaluation

Functionality Change Requests

Joint Specification TeamRequirements Definition

Design and Code ReviewsExternal Code Contributions

Implementation Schedule

Resource Constraints

Collaborative DesignBeta Testing

Working Project

ExecutiveManagement

Reporting

Reporting

weekly

Core Development TeamProject Management

Software DevelopmentTesting & Maintenance

Distribution & User Supportdaily

annually

Change Review BoardDevelopment Priorities

Release Review & Approvalquarterly

• Multiple timescales, all staff levels• Places for structured argument

www.esmf.ucar.edu

Values• Community driven development and community ownership• Openness of project processes, management, code and information• Correctness• Commitment to a globally distributed and diverse development and

customer base• Simplicity• Efficiency• Public storage of project records and other information• Engagement

Web link for detail: http://www.esmf.ucar.edu/about_us/values.shtml

www.esmf.ucar.edu

How to Get Started & Get Help• Web meeting with the development team• Support list – esmf_support@ucar.edu• Code Examples page and FAQ• Tutorials, on-line and in person• Annual Meeting (Fort Lauderdale, FL May 28-30, 2008)• Reference Manual and Users Guide

www.esmf.ucar.edu

Governance

Management of ESMF requires governance that recognizes social and cultural factors as well as technical factors

Main objectives of governance:• Enabling people to argue in a civilized, contained,

constructive way• Enabling people to set priorities and make decisions based

on resource realities• Coordinating at many levels, from developer to center director

to program manager

www.esmf.ucar.edu

Facilitating Science:Coupled Climate-Chemistry with ESMF

The image shows results from a version of the GEOS-5 atmosphericgeneral circulation model coupled to a stratospheric chemistry package(STRAT-CHEM), also developed at NASA, but independently of GEOS-5,which has now been made ESMF compliant.