www.metoffice.gov.uk © Crown Copyright 2020, Met Office

GungHo: Designing a next generation dynamical core for weather & climate prediction

Thomas Melvin

Overview

Part I

• Gungho Dynamical Core

Part II

• LFRic Atmosphere model

• Code generation

• Next generation modelling system

Acknowledments

Nerc: Gungho partner Universities

STFC: PSyClone development

Met Office: LFRic development team

Dynamics Research

UM Physics developers

NGMS programme

• Move to quasi-uniform mesh to remove polar singularity

• Maintain ‘good’ aspects of current model• No computational modes

• Accurate wave dispersion

• Semi-Implicit timestepping

• Reuse subgrid parametrizations

• Improve inherent conservation

• Improve scalability

What we are trying to do

• Gungho: Mixed finite element dynamical core

• LFRic: Model infrastructure for next generation modelling

• PSyclone: Parallel Systems code generation used in LFRic and Gungho

• UM: Current modelling environment ( UM parametrisations are being reused in LFRic)

Model Components

Mixed Finite Element method gives

• Compatibility: 𝛻 × 𝛻𝜑 = 0, 𝛻 ∙ 𝛻 × 𝒗 = 0

• Accurate balance and adjustment properties

• No orthogonality constraints on the mesh

• Flexibility of choice mesh (quads,triangles) and accuracy (polynomial order)

Mixed Finite Elements

Mixed Finite Element Method

Pointwise scalars

Circulation Vectors

Flux Vectors Velocity

Volume integrated Scalars Pressure, Density

Pointwise scalars Potential Temperature, moisture

• Switch from vector invariant form to advective form momentum equation• Allows use of same explicit upwind advection scheme as for scalars: improves stability

& accuracy

• Switch from Krylov (BiCGStab) to Multigrid preconditioner for the pressure solver (improves scalability)

• Move to 2x2 (advection, nonlinear) iterations in the timestep from 4x1 (reduces cost)

• Over relaxation (𝜏𝜌 = 1) in continuity equation (improves stability)

New developments

• Inspired by iterative-semi-implicit semi-Lagrangian scheme used in UM

• Transport uses high-order, upwind, explicit Eulerain FV scheme

• Wave dynamics use iterative-semi-implicit, lowest order mixed finite element method (equivalent to C-grid/Charney-Phillips staggering)

Gungho Discretisation

[1] A mixed finite-element, finite-volume, semi-implicit discretization

for atmospheric dynamics: Cartesian geometry, Melvin et al QJRMS19

Timestep

Compute Slow Physics: 𝑆𝑛(𝑞𝑛)

Compute dynamics forcing: 𝐹𝑛(𝑞𝑛)

Advect fields: A 𝑞𝑝, 𝑢 =A(𝑞𝑛 + 𝑆𝑛 + 𝐹𝑛, 𝑢)

Compute Fast Physics: 𝑆𝑛+1(𝑞𝑛+1)

Compute dynamics forcings:

𝐹𝑛+1(𝑞𝑛+1)

Solve linear system: L𝑞′ = 𝑅

2x

outer

loop2x

inner

loop

𝐿𝑞′ = 𝑅 = 𝐹𝑛+1 + 𝐹𝑛 + 𝑆𝑛+1+ 𝑆𝑛+ 𝐴(𝑞𝑝, 𝑢)

Linear operator Fast (convection, bl)

and Slow (radiation,

gwd) subgrid terms

Dynamics forcing (pgf, Coriolis) Transport

• Advection equation solved using a multi-stage scheme

• Field ( 𝑞) is reconstructed at each stage using a high order upwind polynomial

• Flux (ℱ) or advective (𝒜) increment used to advect field is diagnosed and used in iterative-implicit scheme

• Wind field is fixed during the advection step

• Moisture is transported on a shifted vertical mesh using flux form scheme moisture mass is conserved

Transport

Linear Solver

Iteratively (GCR) solve mixed system

Solve (Multigrid) Helmholtz system

Apply smoother on each level

Error is determined by convergence of mixed

system. Multigrid v-cycle used to speed up

convergence

Form approximate Schur complement≈ 10 ×

≈ 2 ×

• Helmholtz system 𝐻Π′ = 𝑅 solved using a single Geometric-Multi-Grid V-cycle with block-Jacobi smoother

Multigrid Preconditioner

• Block-Jacobi smoother with small number

(2) of iterations on each level

• Smoother: Exact vertical solve: 𝐻𝑧−1 C1152 (~9km)

• No global sums

• Reduced number of operator (H)

applications

www.metoffice.gov.uk © Crown Copyright 2020, Met Office

Questions?

Results

2D Schar hill

Gungho ENDGame

EGGH

Vertical velocity

BalanceDCMIP2012 Test 200: Balanced atmosphere over a large mountain

Gungho C96 ENDGame 1degree

Vertical velocity

Baroclinic Wave: Day 10

Gungho C96 ENDGame (1deg)

Surface Pressure850hPa Temperature

Grid Imprinting

C96, Dt=900s C192, Dt=450s C384, Dt=225s

Vertical velocity (with zonal mean removed) after 1 day of

baroclinic wave simulation

Cube corner

Code Generation: PSyclone

• LFRic [3] uses the PSyclone [4] code generation tool to create the parallel layer

• PSyclone reads the code and generates interfaces based upon defined rules

• Parallelisation & optimisation strategies are defined via a simple script

[3] LFRic: Meeting the challenges of scalability and performance

portability in Weather and Climate models, Adams et al J. Par. Dist. Comp. 19[4] github.com/stfc/PSyclone

Code generation: User codeCompute: 𝑢 + ∆𝑡∇𝑝

Algorithm

Kernel

Code generation: Auto generationPsyclone parses:

Algorithm

Kernel metadata

and produces:

New algorithm

& Psy Layer

Code generation: Auto generation

Built-in

MPI

MPI

Kernel Call

Horizontal

Looping

OpenMP

All parallel directives

appear in the Psylayer

LFRic Atmosphere

• Gungho dynamical core

• Unified model subgrid physics

• Within the LFRic infrastructure

+

• LFRic uses existing UM Physics ( + code repository)

• Can run in single column model (SCM) mode to compare output

• LFRic uses k-first indexing while UM uses i-first

• i-first allows physics to work on a large segment of data giving a performance boost. LFRic has to use single columns as a segment

• Investigating transposing data in LFRic

UM Physics

LFRic UM

Aquaplanet & beyond

• Couple UM physics to Gungho dynamical

core

• Read in reconfigured UM start dump

• I/O using XIOS

• Parallel code layer automatically generatedDynamical core + subgrid physics

No land surface or orography

Proscribed surface temperature

Dynamical

CoreAquaplanet

Basic GA

Proto GA

Aquaplanet

• Comparison to current ENDGame (SISL) dynamical core over 2yr simulation

• Same subgrid physics, vertical mesh & timestepLFRic C48

Reversed jet in UM, not present in LFRic

LFRic jets extend higher

Aquaplanet

850hPa Cloud Ice 850hPa Vertical velocity

Time averages over 800 days of simulation

Next Generation Modelling System (NGMS)

"Reformulating and redesigning our complete weather and climate research and operational/production systems, including oceans and the environment, to allow us to fully exploit future generations of supercomputer for the benefits of society"

NGMS ProjectsGungho Atmospheric

Science Project

• Develop atmospheric

science aspects &

deliver model

scientifically as good as

UM

LFRic Infrastructure

Development• Deliver infrastructure to

replace the UM scalable

for future platforms

LFRic Inputs• Tools to ingest fixed &

time-varying fields.

• Include initial conditions,

ancillary fields and

LBCs

LFRic Diagnostic

Infrastructure• Development of

research diagnostics

and research workflow

capabilities

NG-Marine Systems• Deliver scalable

marine systems

including ocean, sea-

ice & wave models

NG-Coupling• OASIS3-MCT coupled

components

NG-DA• NGMS-ready coupled

atmos/ocean DA

• JEDI as a DA

framework

NG-OPS• Processing of NWP

observational data for

NG-DA

NG-VAT

(Visualisation

Analysis Tools)• Support for visualisation

and evaluation tools

used by scientists

FAB Build System• Development of new

build systems for NGMS

components

NG-R2O• Support transition of

NGMS capability from

research to NWP

operations

NG-Composition• Coordination of aerosol

& chemistry

development within

NGMS

NG-R2C• Support transition of

capability from research

to climate production

NG-ADAQ• Development of

dispersions models (e.g.

NAME) for

next generation

computing

NG-VER• Development of NWP

verification capability for

NGMS

Summary

• Gungho uses a mixed finite element + finite volume transport on a cubed sphere

• Shows similar accuracy to ENDGame

• Has been coupled to UM subgrid physics• Run in aquaplanet mode

• Functionality to run in global atmosphere more exists but needs to be tested

• Only one part of NGMS

www.metoffice.gov.uk © Crown Copyright 2020, Met Office

Questions?

• Quasi-Newton Method:

• Linearized around reference state (previous timestep state) 𝒙∗ ≡ 𝒙𝒏

• Solve for increments on latest state: 𝒙′ ≡ 𝒙 𝒌+𝟏 − 𝒙 𝒌

• Semi-Implicit system contains terms needed for acoustic and buoyancy terms

Linear Solver

[2] Multigrid preconditioners for the mixed finite element dynamical core of

the LFRic atmospheric model, Maynard et al QJRMS20

• Solver Outer system with Iterative (GCR) solver

Linear Solver

• Contains all couplings

• Preconditioned by approximate Schur complement for the pressure increment

• Velocity and potential temperature mass matrices are lumped

• Drops non-orthogonal terms (cubed-sphere, terrain following)