Four-Dimensional Ensemble-Variational Data Assimilation

4DEnVar

Colloque National sur l'Assimilation de données

Andrew Lorenc, Toulouse France. 1-3 décembre 2014

4DEnVar: Topics in Talk

• Design

• Comparison with 4DVar

• Localisation & filtering of ensemble covariance

• Plans

Data Assimilation for Numerical Weather Prediction

Design of a Modern System

Bayesian combination of observations and prior (background)

Incremental formulation – find best correction to prior

Cycled DA – each cycle assimilates a batch of observations

spread over a time-window

Gaussian assumptions about Probability Distribution Functions

“Errors Of The Day” information in prior (background) PDF

comes from an ensemble of forecasts.

Possible Methods

• [ Hybrid / Ensemble ] - 4DVarUse ensemble to augment 3D covariance at beginning of window,

then use linear & adjoint model for time-dimension.

• [ Hybrid ] - 4DEnVarUse ensemble trajectories to determine 4D covariance directly.

• EnKF – e.g. 4D-LETKFLocalised Ensemble Transform Kalman Filter – transforms background

ensemble perturbations to sample the analysed PDF from Kalman eqn.

All need to use [ localisation / smoothing / hybridisation ]to get usable covariances from a small ensemble!!

Incremental 4D-Ensemble-Var

Statistical 4D-Var approximates entire PDF by a Gaussian.

4D analysis is a (localised) linear combination of nonlinear

trajectories. It is not itself a trajectory.

Comparison with 4DVar

Simplified

Gaussian

PDF t1Simplified

Gaussian

PDF t0Full model evolves mean of PDF

PF model evolves any simplified perturbation,

and hence covariance of PDF

Statistical, incremental 4DVar

Statistical 4DVar approximates entire PDF by a

4D Gaussian defined by PF model.

4D analysis increment is a trajectory of the PF model.

Lorenc & Payne 2007

Hybrid

A new form of linear model

Test with a single wind ob, in a jet, at the start of the window

100% ensemble1200km localization scale

4DEnVar

En-4DVarerror

Met Office trial of 4DEnVarLorenc et al. (2014)

Our first trial copied settings from the hybrid-4DVar:

• C with localisation scale 1200km,

• hybrid weights βc2=0.8, βe

Results were disappointing:

The reason was the large weight given to the climatological

covariance, which is treated like 3DVar in 4DEnVar

hybrid-3DVar = hybrid-3DEnVar

hybrid-4DVar 3.6% better

hybrid-4DEnVar 0.5% better

50-50% hybrid1200km localization scale

4DEnVar

Relative “Strong Constraint Errors”

We ran similar tests on a Hurricane Sandy case.

Here the ensemble covariances dominated, making hybrid-4DEnVar perform better.

1200km localization scale

Jet case Hurricane

4DEnVar 51% 57%

En-4DVar 54% 69%

Hybrid-4DEnVar 78% 66%

Hybrid-4DVar 66% 75%

When the ensemble covariances dominated the increments,

and the horizontal localisation was not too severe,

4DEnVar had better consistency with the strong constraint than 4DVar.

Runs with smallest deviation from model constraint

Conclusions from 4D analysis increment study

1. The main error in our hybrid-4DEnVar (v hybrid-4DVar) is that the climatologicalcovariance is used as in 3DVar.

2. 3D localisation not following the flow is not an important error for our 1200km localisation scale and 6hour window, but does become important for a 500km scale.

Improving 4DEnVar

The maintenance and running costs of hybrid-4DVar are larger, so there is an incentive to improve hybrid-4DEnVar.

We need to reduce the weight on climatological B relative to the ensemble covariance.

We must first improve the ensemble covariances:

• a bigger ensemble;

• better ensemble generation;

• better filtering of ensemble covariance, e.g. localization.

Encouraging progress has been made in all of these.

Localisation

Ensemble covariance filtering

Covariance B is big! We need a large ensemble

PLUS clever filtering to reduce sampling noise,

based on 2 ideas:

Assume some correlations are near zero, & localise:

horizontal, vertical, spectral, transformed variables;

Assume local homogeneity − apply smoothing:

horizontal, rotational, and time.

Horizontal localisationErrors in sampled ensemble covariances

0 500 1000 1500 2000 2500 3000

distance (km)

From Lorenc (2003)

The Schur Product

Curve C chosen such that covariances go to zero at distance. e.g. compactly supported (4.10)

from Gapsari and Cohn (1999)

This gives:Ensemble covariances modified to be 0 at distance.Covariance function slightly narrower than ideal.

n=100 * compact support

0 500 1000 1500 2000 2500 3000

distance (km)

From Lorenc (2003)

100% ensemble500km localization scale

4DEnVar

4D-Var

Spectral [waveband] localisation,of pressure at 4km

Input spectrum, 6 bands, sampled spectrum Implied global covariance functions

Ensemble spread in pressure at level 24 (~4km)

Spread, after localisationusing 6 wavebands (~4km)

Comparison with hybrid-4DVar

Hybrid-4DEnVar Hybrid-4DVar

4D via use of ensemble trajectories.

Only 3D use of climatological B.

4D via use of M & MT.

No forecast model inside algorithm,

easy to add variables.

Needs special software for M & MT,

effort needed to add variables.

Needs memory to store trajectories. M & MT have to be run each iteration.

Needs large ensemble and

good localisation.

Needs good covariance model.

The Met Office’s operational hybrid-4DVar works well on current & next computer.

But M & MT are struggling with model changes and resolution increases.

hybrid-4DEnVar will be the first method coded for use with the planned GungHo

model on the massively parallel computer expected next decade.

for global NWP

Parallelisation

4DVar has several potential problems looming in

the next decade - their timing for each centres will

depend on their computers and models:

1. Need new design to use millions of parallel

threads, especially in sequential runs of linear (PF)

and Adjoint models.

2. Forecast models are being redesigned to

address this − a maintenance issue for the PF and

Adjoint models.

4DEnVar is a simple solution, using the ensemble

trajectories, pre-calculated in parallel, instead of

the models inside 4DVar.

If Fourier filters and Poisson solvers are not

available then the LETKF is an easier approach.

Comparison with EnKF

Hybrid-4DEnVar EnKF

Needs transforms and filters of model

fields, each iteration.

Works locally, using ensemble

predictions of observations H(xbk).

Easy to build & cheap to run; cost small

compared to ensemble forecasts.

Using wavebands, better for analysing a

wide range of scales, including large

scales from global ensemble.

Can use hybrid with climatological B.

We cannot yet run a big enough ensemble of good enough forecasts to effectively

sample background error covariance. Priority is on improving model & computer.

Meanwhile Met Office is running 3DVar and developing 4DVar, with a small

forecast ensemble via downscaling of the global ensemble.

We will start experimenting with En-DA with a simple EnKF (concentrating on

model forecasts and infrastructure). 4DEnVar is not yet planned.

for Convective−scale NWP

How to create the ensemble?

• From a separate EnKF system. E.g.:

Canada uses an independent EnKF (Houtekamer et al. 2014)

Met Office MOGREPS (Bowler et al., 2008; Flowerdew and Bowler, 2011)

uses a Localised ETKF re-centred on a deterministic analysis.

• From an Ensemble of 4DEnVar (EDA: Bonavita et al., 2012)

We are experimenting with this, including a MeanPert algorithm to

reduce cost.

From a single EnVar’s Hessian information: EVIL (Auligné, 2012)

• Downscaling from a global ensemble

Met Office experience with regional MOGREPS this is key process

Canada’s regional 4DEnVar uses the global ensemble (Caron et al., 2015)

Trials of increased ensemble size and weight

Modest improvement when increasing ensemble size

Much larger improvement when ensemble weight is high

4DVar performs worse with high ensemble weight, 4DEnVar performs better

Using ensemble modes from the wrong time brings a small benefit

Adam Clayton

4DEnVar: Summary of Talk

• Design

4D using ensemble hybrid-4DVar / hybrid-4DEnVar / EnKF

• Comparison with 4DVar

4DEnVar time propagation OK except in hybrid.

• Localisation & filtering of ensemble covariance

Space, transformed, spectral (wavebands), time, how.

• Plans

Method (global & convective), MPP, getting ensemble, size

Questions and answers

References (2)

My related talks:

Comparing ensemble-variational assimilation methods for NWP: Can hybrid-4DEnVar match hybrid-4DVar? WWOSC, Montreal, August

2014. https://www.wmo.int/pages/prog/arep/wwrp/new/wwosc/documents/Lorenc_4DEnVar_4DVar.pdf

Advances in data assimilation techniques and their relevance to satellite data assimilation. ECMWF Seminar on Use of Satellite

Observations in NWP,8-12 September 2014. http://www.ecmwf.int/sites/default/files/Lorenc_AdvancesDAsatellites.pdf

Four-Dimensional Ensemble-Variational Data Assimilation

Documents

Ensemble Kalman Filter: Current Status and Potentialekalnay/pubs/PartA_Ch4_Springer_fin… · variational assimilation) can be adapted to the LETKF without requiring an adjoint model

Variational Assimilation of combined satellite retrieved ...earth.esa.int/workshops/atmos2006/participants/440/pres_440_nieradzik.pdf · Variational Assimilation of combined satellite

Ensemble Data Assimilation of GSMaP …...2015/12/25 · Ensemble Data Assimilation of GSMaP Precipitationintothe Nonhydrostatic Global Atmospheric Model NICAM Data Assimilation Seminar,

4D Variational Data Assimilation Observation Operators 4D Variational Data Assimilation Observation Operators Hernan G. Arango

Variational Data Assimilation : Optimization and Optimal ...inavon/pubs/SasakiNovember15(5).pdf · Variational Data Assimilation : Optimization and Optimal Control Franc¸ois-Xavier

Summer School on Data Assimilation and Inverse Problems ...potthast... · Variational Assimilation Summer School on Data Assimilation and Inverse Problems Lecture 3: Variational Data

Scale-dependent localization in ensemble- variational data … · 2017. 9. 1. · Scale-dependent localization in ensemble-variational data assimilation: Application in global and

Hans Huang: Variational Data Assimilation, SNU lecture. June 9th, 20091 DA Variational Data Assimilation Hans Huang NCAR Acknowledge: Dale Barker, The

Ensemble Data Assimilation for Reanalysis

Introduction to Variational Data Assimilation...Introduction to Ensemble Data Assimilation Steven J. Fletcher Cooperative Institute for Research in the Atmosphere, Colorado State University

Hybrid Variational Ensemble Data Assimilation for Tropical ...rammb.cira.colostate.edu/research/tropical_cyclones/hfip/ensemble... · NOAA/ESRL: Jeff Whitaker, Stan Benjamin, Steve

Improving High Resolution Tropical Cyclone Prediction Using a Unified GSI-based Hybrid Ensemble-Variational Data Assimilation System for HWRF Xuguang Wang

Hybrid Variational-Ensemble Data Assimilation at NCEP

Ensemble-variational sea ice data assimilation Anna Shlyaeva, Mark Buehner, Alain Caya, Data Assimilation and Satellite Meteorology Research Jean-Francois

Assimilation of Supplemental Observations during Tropical ... · (GSI) data assimilation scheme was upgraded to use a hybrid ensemble-variational approach with characteristics of

Hybrid Variational/Ensemble Data Assimilation for the NCEP GFS

A Three-Dimensional Variational Data Assimilation System

Model reduced variational data assimilation: An ensemble ... · Motivation 4DVar or the adjoint method • Numerically efficient way to calculate the gradient (one gradient calculation

Hybrid Variational-Ensemble Data Assimilation · 2015. 10. 29. · ensemble data assimilation approach, including details of the application of traditional variational covariance

A comparison of Ensemble-Variational data assimilation and 4D-Var