CSEG UpdateCSEG Update

Mariana VertensteinCCSM Software Engineering

Group

Mariana VertensteinCCSM Software Engineering

Group

22

Improvements to CCSM scriptsImprovements to CCSM scripts

Addition of timing tool that provides automated information to help determine load balance, throughput and cost of a run.

Improvements made to CCSM script infrastructure Provide the ability to define new “component

sets/modes”. and as command line options. Makes it easier to run and test new CCSM science (such as addition of component biogeochemistry and atmospheric chemistry).

Simplified user specification of non-default task/thread settings.

Changes improve extensibility and robustness of scripts.

Addition of timing tool that provides automated information to help determine load balance, throughput and cost of a run.

Improvements made to CCSM script infrastructure Provide the ability to define new “component

sets/modes”. and as command line options. Makes it easier to run and test new CCSM science (such as addition of component biogeochemistry and atmospheric chemistry).

Simplified user specification of non-default task/thread settings.

Changes improve extensibility and robustness of scripts.

33

Improvements to CCSM scripts (cont)

Improvements to CCSM scripts (cont)

CCSM test framework rewritten using new functionality. Introduced new testing functionality (e.g. auto-

promotion test). Introduced new “test suites” that now permit different

testing levels:development, pre-tag, post-tag, monthlycurrently in process of defining contributions to each

test suite Benefits:

Regression testing will be easier to perform.Straightforward addition of tests for new CCSM

scenarios. Easier testing of new scenarios (e.g. CLM-CN with

CO2 exchange).More frequent testing of new science.More robust CCSM code base.

CCSM test framework rewritten using new functionality. Introduced new testing functionality (e.g. auto-

promotion test). Introduced new “test suites” that now permit different

testing levels:development, pre-tag, post-tag, monthlycurrently in process of defining contributions to each

test suite Benefits:

Regression testing will be easier to perform.Straightforward addition of tests for new CCSM

scenarios. Easier testing of new scenarios (e.g. CLM-CN with

CO2 exchange).More frequent testing of new science.More robust CCSM code base.

44

Porting and PerformancePorting and Performance ORNL Cray X1 (phoenix)

Both release and development code base have been ported. CAM FV dycore ported in development code base. Porting this dycore in CCSM proved to be very challenging.

30 years/day on 200 procs (ignorning performance fluctuations)

13 years/day on 108 procs on NCAR bluevista 400 year FV control simulation carried out

ORNL Cray XT3 (jaguar) Port of development code base is starting now. Release

code base is running - will require a validation. NCAR IBM (bluevista)

Major new OS upgrade will permit over-subscription of nodes. Provides an opportunity for major performance improvements - but will also result in significantly larger parameter space for performance optimization.

ORNL Cray X1 (phoenix) Both release and development code base have been

ported. CAM FV dycore ported in development code base. Porting this dycore in CCSM proved to be very challenging.

30 years/day on 200 procs (ignorning performance fluctuations)

13 years/day on 108 procs on NCAR bluevista 400 year FV control simulation carried out

ORNL Cray XT3 (jaguar) Port of development code base is starting now. Release

code base is running - will require a validation. NCAR IBM (bluevista)

Major new OS upgrade will permit over-subscription of nodes. Provides an opportunity for major performance improvements - but will also result in significantly larger parameter space for performance optimization.

44

55

Porting and Performance (future)Porting and Performance (future)

NCAR IBM BluegeneWill require single executable

concurrent systemWill require parallel I/O (NetCDF and

binary)

NCAR IBM BluegeneWill require single executable

concurrent systemWill require parallel I/O (NetCDF and

binary)

66

Data Model ProjectData Model ProjectSerial version rewrite of all data models

(datm7, dlnd7, dice7, docn7) completed!datm7 can now duplicate stand-alone CLM

functionality (in serial mode).docn7 now has both DOM and SOM

functionality (SOM scientific verification is close to complete).

All data components now have same functionality.Can perform spatial interpolation from input

data resolution to model resolution.Can data cycle over subset of years.

Serial version rewrite of all data models (datm7, dlnd7, dice7, docn7) completed!datm7 can now duplicate stand-alone CLM

functionality (in serial mode).docn7 now has both DOM and SOM

functionality (SOM scientific verification is close to complete).

All data components now have same functionality.Can perform spatial interpolation from input

data resolution to model resolution.Can data cycle over subset of years.

77

Data Model Project (what is next)

Data Model Project (what is next)

Parallelize the new serial data models.Incorporate parallel data models into

currently developed sequential CCSM.Replace component specific data models

with CCSM data models. As examples:Replace SOM/DOM in CAM with DOCN7Replace forcing atm driver in CLM with DATM7

Use of only one set of data models will result in more consistent science and remove existing code duplication.

Parallelize the new serial data models.Incorporate parallel data models into

currently developed sequential CCSM.Replace component specific data models

with CCSM data models. As examples:Replace SOM/DOM in CAM with DOCN7Replace forcing atm driver in CLM with DATM7

Use of only one set of data models will result in more consistent science and remove existing code duplication.

77

88

Single Executable Concurrent CCSM

Single Executable Concurrent CCSM

CSEG is leveraging the work that Helen He and Chris Ding (SciDAC) have done in order to create a single executable concurrent implementation of CCSM development code.

A somewhat different implementation will be produced to satisfy current CCSM requirements.

Aim to have development CCSM tag (including all CCSM components) by mid-July.

Aim to have a CCSM3.0 release based version released by later in the summer.

Existence of single executable system should improve CCSM portability and debugging.

CSEG is leveraging the work that Helen He and Chris Ding (SciDAC) have done in order to create a single executable concurrent implementation of CCSM development code.

A somewhat different implementation will be produced to satisfy current CCSM requirements.

Aim to have development CCSM tag (including all CCSM components) by mid-July.

Aim to have a CCSM3.0 release based version released by later in the summer.

Existence of single executable system should improve CCSM portability and debugging.

99

Single Executable Sequential CCSM

Single Executable Sequential CCSM

Major progress made in replacing stand-alone CAM with “pseudo-sequential CCSM”.

Both ESMF and MCT will be examined as coupling frameworks. We are committed to creating a sequential interoperable CCSM utilizing the ESMF framework.

ESMF superstructure coding for Stage 1 Evaluation Plan is beginning now. Superstructure design is already in place. Estimated completion date is end of summer. Plan is to incorporate ESMF coupling framework on CAM trunk upon successful completion of Stage 1 evaluation effort.

Major progress made in replacing stand-alone CAM with “pseudo-sequential CCSM”.

Both ESMF and MCT will be examined as coupling frameworks. We are committed to creating a sequential interoperable CCSM utilizing the ESMF framework.

ESMF superstructure coding for Stage 1 Evaluation Plan is beginning now. Superstructure design is already in place. Estimated completion date is end of summer. Plan is to incorporate ESMF coupling framework on CAM trunk upon successful completion of Stage 1 evaluation effort.

1010

Original stand-alone CAM architecture (CAM3.0)

Original stand-alone CAM architecture (CAM3.0)

CAM Driver

Physics

CLM CAM-ICE CAM-OCN

Dynamics

Note that surface models are invoked from within CAM

physics - not from a top level driver

Pseudo-Sequential CCSM Top level Architecture

Pseudo-Sequential CCSM Top level Architecture

Application Driver(FD)

CLM CAM-ICE CAM-OCNCAM

Phys Dyn

Couplers (FD)

ATM merger(FD)

OCN merger(FD)

LND merger(FD)

ICE merger(FD)

Thin coupling layer (FD)

Thin coupling layer (FD)

Thin coupling layer (FD)

Thin coupling layer (FD)

Introduce top level framework dependent

(FD) application driver to replace CAM driver

Introduce toplevel ESMF clock tocoordinate time evolution of all

components

Introduce new flexible andextensible “thin”

coupling layer design

1212

Sequential CCSM (cont)Sequential CCSM (cont)

New top level based application driver. Independent of CAM data structures Time evolution based upon ESMF general time

management utilities Utilizes CCSM share code for reading input Initial implementation: MCT Near term implementation: ESMF

New surface coupling layer. Initial implementation: MCT Near term implementation: ESMF

New directory structure for sequential CCSM. Applies to both MCT and ESMF

Inter-component domain checking in coupling layer. Initial implementation: MCT Near term implementation: ESMF

New top level based application driver. Independent of CAM data structures Time evolution based upon ESMF general time

management utilities Utilizes CCSM share code for reading input Initial implementation: MCT Near term implementation: ESMF

New surface coupling layer. Initial implementation: MCT Near term implementation: ESMF

New directory structure for sequential CCSM. Applies to both MCT and ESMF

Inter-component domain checking in coupling layer. Initial implementation: MCT Near term implementation: ESMF

1313

CAM Update CAM Update Implemented support for non lat-lon grids (Jim Edwards

and Pat Worley). Non lat-lon support added to Boundary data and aerosol data interpolation code.Physics/Dyamics coupling layer. Only aqua-planet mode is currently supported.

Significant FV dycore interface refactoring implemented:Only data on the XY decomposition outside the

portable dycore is now exposed.Added dynamics import/export states, and dynamics

component module with init, run, final methods. CAM testing framework extended to add new platforms

(e.g. phoenix) and to perform overnight regression testing.

New features added to CAM tropospheric MOZART.

Implemented support for non lat-lon grids (Jim Edwards and Pat Worley). Non lat-lon support added to Boundary data and aerosol data interpolation code.Physics/Dyamics coupling layer. Only aqua-planet mode is currently supported.

Significant FV dycore interface refactoring implemented:Only data on the XY decomposition outside the

portable dycore is now exposed.Added dynamics import/export states, and dynamics

component module with init, run, final methods. CAM testing framework extended to add new platforms

(e.g. phoenix) and to perform overnight regression testing.

New features added to CAM tropospheric MOZART.

1414

CAM Update (what is next)CAM Update (what is next) Refactoring of CAM to run with new dycores.

Incorporation of HOMME dycore into CAM.Refactoring CAM’s history module to

encompass non lat-lon output.Ability to run pseudo-sequential CCSM (not

just aqua-planet) with non lat-lon CAM. Incorporation of parallel I/O into CAM.

Replace all binary I/O with NetCDF I/O.Creation of new tool to generate CAM

namelists (tool should easily extend to other models if desired).

Refactoring of CAM to run with new dycores.Incorporation of HOMME dycore into CAM.Refactoring CAM’s history module to

encompass non lat-lon output.Ability to run pseudo-sequential CCSM (not

just aqua-planet) with non lat-lon CAM. Incorporation of parallel I/O into CAM.

Replace all binary I/O with NetCDF I/O.Creation of new tool to generate CAM

namelists (tool should easily extend to other models if desired).

1515

CAM Update (what is next) CAM Update (what is next)

SCAM (Single Column Atmosphere Model) refactoring Goal is to produce a more flexible and maintainable

interfaces. , Adoption of new components

Incorporation of new ice sheet model (GLIMMER).Primary work will be done by Bill Lipscomb.GLIMMER has already been incorporated into CAM

build system.Replacement of CAM-CSIM with CICE and

CAM-SOM/DOM with DOCN7.

SCAM (Single Column Atmosphere Model) refactoring Goal is to produce a more flexible and maintainable

interfaces. , Adoption of new components

Incorporation of new ice sheet model (GLIMMER).Primary work will be done by Bill Lipscomb.GLIMMER has already been incorporated into CAM

build system.Replacement of CAM-CSIM with CICE and

CAM-SOM/DOM with DOCN7.

1616

CLM UpdateCLM UpdateFinemesh grids have been implemented in

CLM. This gives CLM the new capability to run on its own independent grid. The implementation follows the scheme of Hahmann and Dickinson.CLM still couples to CAM or CCSM via the CAM coarse

grid. Mapping is done within the CLM code base.The implementation does not change answers when

the CLM fine grid is identical to the CAM grid.Stand-alone CAM runs have been run with a T42

coarse grid and a half degree finemesh grid. Results are encouraging.

New downscaling and upscaling interactions are being implemented.

Finemesh grids have been implemented in CLM. This gives CLM the new capability to run on its own independent grid. The implementation follows the scheme of Hahmann and Dickinson.CLM still couples to CAM or CCSM via the CAM coarse

grid. Mapping is done within the CLM code base.The implementation does not change answers when

the CLM fine grid is identical to the CAM grid.Stand-alone CAM runs have been run with a T42

coarse grid and a half degree finemesh grid. Results are encouraging.

New downscaling and upscaling interactions are being implemented.

1717

CLM Update (what is next)CLM Update (what is next) A nested grid capability is being added to CLM, so the

model can be run at spatially varied resolution to optimize cost.

A new prognostic canopy air space scheme is being implemented. Scientific formulation is being finalized. Software implementation (Forrest Hoffman) will start as

soon as the scientific formulation is completed. CLM offline testing framework will be rewritten in order

to test the numerous new features that are being introduced into the system.

Urban code will be incorporated into the main development line of development.

A nested grid capability is being added to CLM, so the model can be run at spatially varied resolution to optimize cost.

A new prognostic canopy air space scheme is being implemented. Scientific formulation is being finalized. Software implementation (Forrest Hoffman) will start as

soon as the scientific formulation is completed. CLM offline testing framework will be rewritten in order

to test the numerous new features that are being introduced into the system.

Urban code will be incorporated into the main development line of development.

1818

POP2 UpdatePOP2 UpdateCompleted incorporation of major CCSM

features into the LANL pop2.1.alpha code. CCSM POP2 code is now in CCSM SVN repository and current CCSM scripts.

Added New tracer advection scheme (1D Lax-

Wendroff with 1D flux limiters).Near-surface eddy flux, enhanced deep

mixing. Support for new 1-degree grid with

Galapagos Islands topography (gx21v4).

Completed incorporation of major CCSM features into the LANL pop2.1.alpha code. CCSM POP2 code is now in CCSM SVN repository and current CCSM scripts.

Added New tracer advection scheme (1D Lax-

Wendroff with 1D flux limiters).Near-surface eddy flux, enhanced deep

mixing. Support for new 1-degree grid with

Galapagos Islands topography (gx21v4).

1919

POP2 Update (what is next)

POP2 Update (what is next)

Inclusion of a more efficient barotropic solver designed by John Dennis.

Reduced equatorial viscosity.A new vertical grid.Systematic exploration of the above

features. Plans for incorporating the ecosystem

model into POP2 will be developed during the CCSM workshop.

Inclusion of a more efficient barotropic solver designed by John Dennis.

Reduced equatorial viscosity.A new vertical grid.Systematic exploration of the above

features. Plans for incorporating the ecosystem

model into POP2 will be developed during the CCSM workshop.

2020

CICE UpdateCICE UpdatePCWG has decided to incorporate CICE as the

new standard CCSM ice model. New name will be “Community Ice CodE”.

Testing has been done using CICE 3.1 to establish main differences with CSIM 5.0.

CICE 3.1 changes answers significantly, but still within the realm of the same climate.

CICE 3.14 has additional bug fixes and a few new physical parameterizations. This is the version that will be adopt after more testing is completed.

Goal is to incorporate CICE 4 (with new data structures) into CCSM and into pseudo-sequential CCSM.

PCWG has decided to incorporate CICE as the new standard CCSM ice model. New name will be “Community Ice CodE”.

Testing has been done using CICE 3.1 to establish main differences with CSIM 5.0.

CICE 3.1 changes answers significantly, but still within the realm of the same climate.

CICE 3.14 has additional bug fixes and a few new physical parameterizations. This is the version that will be adopt after more testing is completed.

Goal is to incorporate CICE 4 (with new data structures) into CCSM and into pseudo-sequential CCSM.