Upload
andrew-harper
View
215
Download
0
Embed Size (px)
Citation preview
GEOS–CHEM,Transition to ESMF,and GEOS-5 met fields
Bob YantoscaSoftware EngineerAtmospheric Chemistry Modeling GroupHarvard University
Group Meeting / Telecon16 Nov 2005
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 2
Topics1. Quick GEOS–CHEM Status Report2. ESMF Overview3. ESMF Design Principles4. ESMF Installation on Your Platform5. GEOS–5 Met Fields: A Quick Look
NOTES: ESMF = Earth System Model Framework Some slides in this presentation were taken from
Introduction to ESMF Tutorial and Introduction to the Earth System Modeling Framework from www.esmf.ucar.edu
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 3
Part 1
GEOS–CHEM Status Report
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 4
GEOS–CHEM Status GEOS–CHEM v7–03–06 is the latest public release
v7–03–06 is now available for download from G–C web site. The manual pages have been updated.
Several additions, fixes, improvements have been incorporated into G–C from the last public version (v7–02–04, April 2005)
Here follows a laundry list of updates…
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 5
GEOS–CHEM Status Met field updates in v7–03–06 Now supports GISS met fields for GCAP simulations
(S. Wu) Added code to read GEOS–3 "XTRA" met fields (M.
Fu) i.e. PARDF, PARDR, SNOW
Snow depth is now used in GEOS–3 for dust emissions
New simulations in v7–03–06 CO2 (P. Suntharalingam) HCN / CH3CN (Y. Xiao)
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 6
GEOS–CHEM Status Convection updates in v7–03–06 Turn off scavenging in shallow convection for GCAP
(S.Wu)
Diagnostic updates in v7–03–06 Improvements to plane-flight diagnostic
Archives trop column AOD at location of aircraft Archives column AOD from the surface to the aircraft
location Improvements to timeseries diagnostics
Minor fixes for AOD output in ND49 diagnostic Timeseries diagnostics now can archive grid box
height
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 7
GEOS–CHEM Status Emissions updates in v7–03–06 EMEP (anthro) European Emissions (M. Auvray, B. Field)
MEGAN (biogenic) emissions (M. Fu)
Rescaling of Lightning NOx AND LNOx, DMS, sea-salt emissions now turned off @ poles
(R. Hudman, R. Yantosca, B. Alexander)
Uses Nightingale et al 2000b formula for sea-air exchange (S. Wu) For DMS, Acetone, CH3I, sea-salt, etc. emissions
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 8
GEOS–CHEM Status Aerosol updates in v7–03–06 Now uses ISORROPIA package to compute ATE
(B. Alexander)
Added SO4s, NITs tracers into std simulation (R. Park, B. Alexander)
New oxidant fields for offline aerosol simulations (C. Heald, R. Yantosca) From 1-yr benchmark v7–02–03
Now allows for hygroscopic growth of aerosols in drydep(R. Park, B. Alexander)
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 9
GEOS–CHEM Status Technical updates in v7–03–06 (R.Yantosca, B.Field) Now modified for Intel Fortran Compiler “ifort” v9.0 Can now to regrid emissions from GEOS 1x1 grid to:
1x1 GEOS nested grids 2x25 GEOS grid 4x5 GEOS grid 4x5 GCAP grid
Removed obsolete SLOW–J code Removed obsolete CO–OH parameterization code Removed several obsolete common block variables G–C now writes diaginfo.dat, tracerinfo.dat files for
GAMAP You no longer need to manually edit these files anymore!
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 10
GEOS–CHEM Status Benefits of v7–03–06 Intel v9 compiler optimizes much better on the Altix
1-month 4x5 fullchem run is ½ hr faster than Intel v7 MEGAN biogenic inventory will eventually supplant
GEIA But for the time being we are keeping both
Housekeeping Please migrate to v7–03–06 when possible! Also be sure to download new data directories
From geos.as.harvard.edu
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 11
Part 2
ESMF Overview
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 12
What is ESMF?What is the Earth Systems Modeling Framework (ESMF)? The ESMF is a focused community effort to tame the
complexity of models and the computing environment. It leverages, unifies and extends existing software frameworks, creating new opportunities for scientific contribution and collaboration.
Goals of ESMF:1. Increase scientific productivity by making modeling and
analysis software components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers.
2. Unify the national and international Earth system modeling community through a common modeling paradigm and regular interactions at all levels.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 13
What is ESMF?1. ESMF provides tools for
turning model codes into components with standard interfaces and standard drivers
2. ESMF provides data structures and common utilities that components use
i. to organize codesii. to improve performance
portabilityiii. for common services
such as data communications, regridding, time mgmt and message logging
ESMF InfrastructureData Classes: Bundle, Field, Grid, Array
Utility Classes: Clock, LogErr, DELayout, Machine
ESMF SuperstructureAppDriver
Component Classes: GridComp, CplComp, State
User Code
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 14
What is ESMF?
Diagnostics
With ESMF, you can reduce a CTM to a collection of “pluggable”
black box “Components” with standard inputs & outputs
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 15
ESMF is a Community Effort Collaborators and customers include:
NSF NCAR NOAA GFDL, NOAA NCEP DOE LANL, DOE ANL NASA GMAO, NASA Land Information Systems, NASA GISS DoD Navy, Air Force, and Army University of Michigan, UCLA, MIT
Development & testing is open-source
Website: www.esmf.ucar.edu
Email: [email protected]
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 16
Why Should I Adopt ESMF if I already
have a working model? There is an emerging pool of other ESMF-based science
components that you will be able to interoperate with to create applications - ESMF has a broad customer base.
It will reduce the amount of infrastructure code that you need to maintain and write, and allow you to focus more resources on science development.
ESMF provides solutions to two of the hardest problems in model development: structuring large, multi-component applications so that they are easy to use and extend, and achieving performance portability on a wide variety of parallel architectures.
It may be better software (?!) than the infrastructure software that you are currently using.
Community development and use means that the ESMF software is widely reviewed and tested, and that you can leverage contributions from other groups.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 17
Why Should I Adopt ESMF if I already
have a working model? NASA now “requires” that all of its Earth
sciences modeling assets conform to the Earth System Modeling Framework
This is being done so that “pieces” of GEOS–CHEM (or other models) can be inserted into larger Earth System Models
Future funding may be dependent on the ability to make models ESMF-compliant
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 18
ESMF Overview Summary
What is ESMF? It’s a software framework for earth science models
that NASA et al is espousing
What ESMF will do: It will make it easier to connect “pieces” of the same
model model together It will make it easier to take “pieces” of one model
and mix them with “pieces” from other models. It will probably eventually reduce model
development time (however, some up-front development will be needed)
What ESMF will not do: It will NOT improve the quality of the science of a
model
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 19
Part 3
ESMF Design Principles
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 20
Classes and Objects in ESMF The ESMF Application Programming Interface (API) is based
on the object-oriented programming notion of a class. A class is a software construct that’s used for grouping a
set of related variables together with the subroutines and functions that operate on them.
A particular instance of a class is called an object. Think of a class as being a “data type” and an object as
being a specific variable of that particular data type.
Our goal: convert existing model code into ESMF Components by using the ESMF class structure.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 21
Classes and Objects in ESMF
There are two software layers in ESMF Infrastructure Superstructure
Infrastructure classes define the data types which are ESMF’s “building blocks”
Array, Field, Bundle, etc …
Superstructure classes define the overall function of ESMF Components
Gridded Component, Coupler Components, “main”
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 22
ESMF Infrastructure Data Classes
ESMF Grid
ESMF Array
ESMF Array = a data container (similar to an F90 array) It can contain data and may be up to 7 dimensions.ESMF Grid = definition of a physical grid(latitude, longitude, and vertical dimensions)
ESMF Array
ESMF Grid
An ESMF Field is the combination of an ESMF array with an ESMF grid.
Therefore, an ESMF field is like a “smart” array. It can tell you not only about its data but also about what kind of grid it is defined on.
ESMF Fields can be used to store physical quantities (i.e. Pressure, Temperature, Winds, Humidity, etc.)
ESMF Field
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 23
ESMF Infrastructure Data Classes
ESMF Field #1
An ESMF Bundle is the combination of more than one ESMF Field.
Bundles make it easy to group fields together in a logical fashion
ESMF Field #2 ESMF Field #3
. .
.
ESMF Bundle
ESMF Bundle #1 ESMF Bundle #2 ESMF Bundle #3
. .
.ESMF State
An ESMF State is the combination of one or more bundles.
States are passed between ESMF Gridded Components.
Each Gridded Component takes one Input State and one Output State as arguments.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 24
ESMF Infrastructure Data Classes
Other Infrastructure utilities: Time Manager and Calendar Functions Configuration Attributes (replaces namelists) Message logging Communication libraries Regridding library (parallelized, on-line SCRIP) IO (barely implemented) Performance profiling
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 25
ESMF Superstructure ClassesSchematic of an ESMF Gridded Component
Input State
Output State
ESMF Clock
Return code
ESMF Superstructure Layer
User Code Init Routine
ESMF Infrastructure Layer
User Code Run Routine
User Code Finalize Routine
ESMF library function calls
Internal F90 code routines
Internal F90 code routines
Internal F90 code routines
Each ESMF Gridded Component must have init, run, finalize routines.
Init takes data from the Input state and allocates all internal data storage
Run is used to invoke the user source code
Finalize deallocates all internal storage.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 26
Design Strategy: Modularity
Gridded Components don’t have access to the internals of other Gridded Components, and don’t store any coupling information. Gridded Components pass their States to other components through their argument list.Since components are not hard-wired into particular configurations and do not carry coupling information, components can be used more easily in multiple contexts.
atm_comp
NWP application
Seasonal prediction
Standalone for basic research
NOTE: the same ESMF Component can be “plugged” into several different types of simulations
No more global variables!
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 27
ESMF Superstructure ClassesSchematic of an ESMF Coupler Component
Comp #1
Comp #2
Comp #3
Input States
Output States
Internal Linkages
ESMF Coupler Component
In the Coupler, the programmer is responsible for connecting data (e.g. P,T) from the output states of one or more Components with the input state of another Component (and vice versa).
P
T
P, T
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 28
Design Strategy: Hierarchical ApplicationsSince each ESMF application is also a Gridded Component, entire ESMF applications can be nested within larger applications. This strategy can be used to systematically compose very large, multi-component codes. Example: GEOS–5 assimilation system.
GEOS-5
surface fvcore gravity_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 29
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct, etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 30
Things to be happy about
GEOS–CHEM is in good shape for ESMF conversion
Most new G–C code is in F90 module formEach F90 module has INIT and CLEANUP routines
G–C is highly “methodized”Many quantities are accessible via function callsComplexity is moved to functions rather in main code
G–C has consistent file I/O and error handling G–C code is very well documented
We may be able to leverage other efforts GMI is also converting to the ESMF framework Tom Clune is writing an ESMF replacement for SMVGEAR
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 31
Things to be aware ofESMF conversion will require time and effort
ESMF library function calls are very “wordy”You have to use a lot of function calls to do stuff
G–C’s legacy code routines will have to be rewritten:Full-chemistry emissionsDrydep ?FAST–J photolysis (this is 3rd party code)Diagnostics
Conversion of G–C to ESMF will have to be in parallel with:
Normal G–C user supportGEOS–5 met field implementation
We are looking to hire a 2nd FTE for programming support
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 32
Part 4
ESMF Installation
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 33
ESMF Platform SupportSystem requirements include: A Fortran 90 or later compiler
C and C++ compilers (proprietary or gcc/g++)
a MPI implementation compatible with these compilers OR an MPI-bypass library (which is bundled with the ESMF source distribution)
The gmake utility (for building the code)
The tar and gzip utilities (for unpacking data files)
The Perl programming language, for running test scripts
Latex, latex2html, and the Unix/Linux dvipdf utility are needed to build the ESMF documentation (not required to build code)
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 34
ESMF Platform Support IBM AIX (32 and 64 bit) SGI IRIX64 (32 and 64 bit) SGI Altix (64 bit) Cray X1 (64 bit) Compaq OSF1 (64 bit) Linux Intel (32 and 64 bit) Linux PGI (32 bit) Linux NAG (32 bit) Linux Absoft (32 bit) Linux Lahey (32 bit) Mac OS X with xlf (32 bit)
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 35
ESMF Platform SupportCAVEATS: Only the most recent compilers may be
supported SGI: Probably need at least SGI MIPS 7.4 compiler Intel: Need at least Intel v8; it works w/ Intel v9 *** BE PREPARED TO UPDATE YOUR COMPILERS ***
Installation is heavily weighted towards proprietary C and C++ compilers Bob Y. had difficulties trying to install with the “free”
gcc/g++ compilers at Harvard
If your system is unsupported, you’re out of luck
Complain to [email protected]
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 36
ESMF Platform Support My ESMF Installation story – version 2.2
I couldn’t install on SGI with v7.2.1.3 compiler So I gave up on that platform
I couldn’t install on Altix with Intel v7.1 compiler Intel v7.1 compiler was “too old” for ESMF
I bugged Jack to install Intel Compiler v9.0 Intel 9.0 compiler installation took a while (licensing
issues)
My 1st attempt to install w/ Intel v9 didn’t work ESMF source code had bugs
ESMF issued a software patch, I recompiled w/ that Finally, success!!!!
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 37
Part 5
GEOS–5: A Quick Look
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 38
GEOS–5 Quick Look GEOS–5 is the next met field product from GMAO Slated to become operational after INTEX-B mission
Resolution will be significantly higher 0.5 (lat) x 0.625 (lon) x 72 vertical layers
Grid will be in Lagrangian control volume coords Different than GEOS–4 You don’t use a formula to compute P(I,J,L) You carry around a 3-D pressure field to compute P(I,J,L)
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 39
GEOS–5 Quick Look Concerns GEOS–5 “raw” data volume per day is huge!! We may have to get an account on NASA machines for
data processing LCV coordinates are not compatible with existing G–C code LCV coordinates are not compatible with existing TPCORE
May need to modify TPCORE S–J Lin is out of the TPCORE business Not sure what the ramifications for mass-conservation are
when we switch from ETA to LCV coordinates It would be easier for us if GMAO made GEOS–5 data on
the same grid as GEOS–4 This may be possible if we squawk loud enough
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 40
GEOS–5 Quick Look GEOS-5 A-3 (3hr time-avg'd) MET FIELDS REQUIRED FOR GEOS-CHEM===============================================================================
Name Description Units Where it's used---- ----------- ----- ---------------ALBEDO Visible surface albedo [unitless] drydep + as a snow/ice flagCLDTOT 2-D column cloud fraction [unitless] drydep GWETTOP Topsoil wetness [unitless] dust mobilization HFLUX Sensible heat flux [W/m2 ] drydep LAI GMAO leaf area index [m2/m2 ] for comparison purposes PARDF Diffuse PAR [W/m2 ] new isoprene emission code PARDR Direct PAR [W/m2 ] new isoprene emission code PBLH PBL heights [m ] BL mixing, drydep, etc. PRECTOT Total accum. precip [kg/m2/s ] wetdep PRECCON Total conv. precip [kg/m2/s ] wetdep RADLWG Net upward LW rad @ ground [W/m2 ] dust mobilization RADSWG New downwd SW rad @ ground [W/m2 ] dust mobilization SNOWMAS Snow depth -- H2O equiv. [kg/m2 ] dust mobilization T2M Temp @ 2m altitude [K ] proxy for surface air. tempTSKIN Ground or Sea Surf. Temp. [K ] new isoprene emission code U10M U-wind @ 10m altitude [m/s ] drydep, NOx em, acetone em.USTAR Friction velocity [m/s ] drydep V10M U-wind @ 10m altitude [m/s ] drydep, NOx em, acetone em.Z0M Roughness Height [m ] drydep
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 41
GEOS–5 Quick Look GEOS-5 A-6 (6hr time-avg'd) FIELDS REQUIRED FOR GEOS-CHEM===============================================================================
Name Description Units Where it's used---- ----------- ----- ---------------CMFMC Cloud mass flux [kg/m2/s ] convectionDTRAIN Cloud detrainment flux [kg/m2/s ] convectionCLOUD 3-D cloud fraction [unitless] diagnostic output DQVDTMST Tendency in sp. humidity [g/kg/day] wetdepOPTDEP Total grid box opt. depth [unitless] photolysis (J-values)QV Specific Humidity [kg/kg ] full chemistry + wetdepT Temperature [K ] everywhere U Zonal winds [m/s ] TransportV Meridional winds [m/s ] Transport
We will probably also pull OMEGA (vertical velocity) Pa/s
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 42
GEOS–5 Quick Look GEOS-5 I-6 (6-hr inst) FIELDS REQUIRED FOR GEOS-CHEM===============================================================================
Name Description Units Where it's used---- ----------- ----- ---------------LWI Land-water flags [unitless] to denote water/land/icePS Surface pressure [Pa] used everywhere TROPP Tropopause pressure [Pa] for trop heightSLP Sea-level pressure [Pa] for diagnostics
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 43
Conclusions v7–03–06 is now the latest std code release Please upgrade!!!!
GEOS–CHEM will be converted to ESMF This will make things easier in the long run Conversion process will take time
GEOS–5 will be the operational data product in 2006 However there are several questions we still have Not sure to what extent G–C may need to be modified A thorough validation will be necessary
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 44
Extra Slides
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 45
Design Strategy:Intracomponent CommunicationAll communication in ESMF is handled within components. This allows the architecture of the framework to be independent of the communication strategy. The result is that there is flexibility in implementation of communications and component drivers are straightforward.
climate_comp
ocn_comp atm_comp
atm_phys
phys2dyn_coupler
atm_dyn
atm2ocn _coupler
As a consequence, Coupler Components must be defined on the union of all processors and all the Gridded Components that they couple.
In this example, in order to send data from the atm Component (blue) to the ocean Component (green), the atm2ocn_coupler mediates the send.
16 Nov 2005 GEOS-CHEM, ESMF and GEOS-5 46
Design Principle: Scalable Applications
Since each ESMF application is also a Component, entire ESMF applications can be treated as Gridded Components and nested within larger applications.
climate_comp
ocn_comp atm_comp
ocn2atm_coupler
atm_phys
phys2dyn_coupler
atm_dyn
Example: atmospheric application (in blue) itself composed of multiple Components may be run standalone, or nested within a larger climate application