High-Dimensional Nonlinear Data Assimilation with the ... · High-Dimensional Nonlinear Data Assimilation with the Nonlinear Ensemble Transform Filter (NETF) and its Smoother Extension

High-Dimensional Nonlinear Data Assimilation with the Nonlinear Ensemble Transform Filter (NETF)

and its Smoother Extension

Paul Kirchgessner, Lars Nerger Alfred Wegener Institute, Bremerhaven, Germany

Julian Tödter, Bodo Ahrens University of Frankfurt, Frankfurt, Germany

5. International Symposium on Data Assimilation, Reading, UK, July 18 – 22, 2016

Nonlinear Ensemble Transform Filter & Smoother

Overview

!  Study new Nonlinear Ensemble Transform Filter – NETF (Tödter & Ahrens, MWR, 2015)

!  Extend NETF for smoothing

!  Test filter and smoother in realistic high-dimensional idealized ocean data assimilation experiments


•  represent state and its error by ensemble of m states

•  Forecast: •  Integrate ensemble with numerical model

•  Analysis: •  update ensemble mean

•  update ensemble perturbations

(both can be combined in a single step)

•  Ensemble Kalman filters & NETF: Different definitions of

•  weight vector

•  Transform matrix

Ensemble filters – ensemble Kalman filters & NETF

X

w

W

x

a = x

f +X

0fw

X0a = X0fW


•  Ensemble Kalman: •  Transformation according to KF equations

•  NETF (Tödter & Ahrens, MWR, 2015) !  Mean update from Particle Filter weights: for all particles i

Nonlinear ensemble transform filter - NETF

!  Ensemble update •  Transform ensemble to fulfill analysis covariance

(like KF, but not assuming Gaussianity) •  Derivation gives

( : mean-preserving random matrix; useful for stability) (Almost same formulation: Xiong et al., Tellus, 2006) ⇤

W =pm

⇥diag(w)� wwT

⇤1/2⇤

wi ⇠ exp

⇣�0.5(y �Hxf

i )TR�1

(y �Hxfi )

⌘


•  Smoother: Update past ensemble with future observations •  Rewrite ensemble update as

•  Filter:

Ensemble Smoothers – ETKS & NETS

Xak|k = Xf

k|k�1Wk

analysis time Observations used up to time

•  Smoother at time

!  works likewise for ETKS and NETS !  also possible for localized filters

Xai|k = Xf

i|k�1Wk

i < k

See, e.g., Nerger, Schulte & Bunse-Gerstner, QJRMS 140 (2014) 2249–2259


•  Performance for small model (Lorenz-96) •  In Tödter & Ahrens (MWR, 2015)

•  NETF beats ETKF for m=20 and larger

Performance of NETF – Lorenz-96

How do NETF and NETS perform in a more realistic case?

update mechanism seems to be more beneficial than itsstochastic counterpart.The next experiment concerns the L2005 model,

which exhibits a distinct spatial structure compared tothe L96 model. Figure 5 shows the analysis error withrespect to ensemble size. Concerning the relative per-formances of the KF-based filters, the general structureis very similar to the L96 experiment. The most re-markable, seemingly counterintuitive difference is thathere the NLEAF1 performs better than the NLEAF2,except whenm5 100. Lei and Bickel (2011) do not showthe NLEAF2 for their larger-dimensional experiments,hence, we cannot directly compare these findings totheir results. A possible explanation could be revealedby the update formalism of theNLEAF2 [Lei andBickel2011, their Eq. (3)], which requires the estimation of anindividual analysis covariance matrix for each ensemblemember, based on each of the m perturbed obser-vations. These low-rank approximations of the d 3 dcovariance matrix with m members are subject to sam-pling error, and it seems that the stochastic errors of theperturbed observations amplify the errors in the

estimation of the covariances, which may lead to theseunexpected results in certain larger-dimensional cases.This hypothesis is supported by the fact that in the low-dimensional L63 experiments the NLEAF2 consistentlyoutperformed the NLEAF1. Additionally, in the L2005system, spatial correlations are more important than inthe L96 system, hence a reliable estimation of the co-variances is of greater relevance here. The NETF, whichalso focusses on the second-order statistics, does notsuffer from this issue. Again, it exhibits the smallestanalysis error form$ 20. We conclude that, particularlyin larger-dimensional scenarios, the benefits of the de-terministic update mechanism of the NETF becomemore apparent.To strengthen these findings, Table 3 gives an overview

of more diagnostic measures for m 2 f10, 25, 50, 100g.As in the L63 case, the comparison of RMSE and spreadas well as of innovation variance and expected innovationvariance indicate that, thanks to the inflation procedure,the filters behave reasonably well in both state and ob-servation space. Except for m5 10, all scores reveal thatthe NETF performs best. The CRIGN shows that in

FIG. 4. Result of the L96 experiment with double exponential observation errors withs2obs 5 1. Shown is the average RMSE for the six ensemble filters against ensemble size, i.e.,

NETF (black line), ETKF (black, dotted), ETKFrot (black, dashed), EnKF (gray line),NLEAF1 (gray, dotted), and NLEAF2 (gray, dashed).

FIG. 5. As in Fig. 4, but for the L2005 experiment with double exponential observation errorswith s2

obs 5 1.

1360 MONTHLY WEATHER REV IEW VOLUME 143


Assimilation into NEMO

European ocean circulation model

Model configuration

•  box-configuration SEABASS

•  ¼o resolution

•  121x81 grid points, 11 layers (state vector ~300,000)

•  wind-driven double gyre (a nonlinear jet and eddies)

•  medium size SANGOMA benchmark

True sea surface height at 1st analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6

True sea surface height at last analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6

www.data-assimilation.net


PDAF: A tool for data assimilation

PDAF - Parallel Data Assimilation Framework "  a program library for data assimilation "  provide support for ensemble forecasts "  provide fully-implemented filter and smoother algorithms

(LETKF, LSEIK, LESTKF, …) "  easily useable with (probably) any numerical model

(applied with NEMO, MITgcm, FESOM, MPI-ESM, HBM) "  makes good use of supercomputers "  first public release in 2004; continued development

Open source: Code and documentation available at

http://pdaf.awi.de

L. Nerger, W. Hiller, Computers & Geosciences 55 (2013) 110-118


Online coupling: Minimal changes to NEMO

Add to mynode (lin_mpp.F90) just before init of myrank #ifdef key_USE_PDAF CALL init_parallel_pdaf(0, 1, mpi_comm_opa) #endif

Add to nemo_init (nemogcm.F90) at end of routine #ifdef key_USE_PDAF CALL init_pdaf() #endif

Add to stp (step.F90) at end of routine #ifdef key_USE_PDAF CALL assimilate_pdaf() #endif

Modify dyn_nxt (dynnxt.F90) #ifdef key_USE_PDAF IF((neuler==0 .AND. kt==nit000).OR.assimilate) #else

Aaaaaaaa

Aaaaaaaa

aaaaaaaaa

Start

Stop

Initialize Model generate mesh Initialize fields

Time stepper consider BC

Consider forcing

Post-processing

init_parallel_pdaf

Do i=1, nsteps

init_pdaf

assimilate_pdaf


Observations and Assimilation Configuration

Observations

•  Simulated satellite sea surface height SSH (Envisat & Jason-1 tracks), 5cm error

•  Temperature profiles on 3ox3o grid, surface to 2000m, 0.3oC error

Data Assimilation

•  Ensemble size 120

•  ETKF and LETKF

•  Localization: weights on matrix R-1 (Gaspari/Cohn’99 function, 2.5o radius)

•  Assimilate each 48h over 360 days −60 −55 −50 −45 −40 −35 −30

2428

3236

4044

(a) SSH [m]

Longitude [°]

Latit

ude

[°]

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● −0.6

−0.4

−0.2

0.0

0.2

0.4

● ● ● ● ● ● ● ●

5000

4000

3000

2000

1000

0

(b) T [°C]

Latitude [°C]D

epth

[m]

24 26 29 32 35 38 41 44

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

0

2

4

6

8

10

12

FIG. 3. Observation characteristics on day 8: (a) The horizontal domain is shown, together with the Argo

profiler locations (crosses) and the synthetic SSH observations (colored) on the Envisat tracks (thin lines). (b)

The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles along the

� = �50� longitude line. Note that at � = 44�, the true temperature field is zero due to the lateral boundary

conditions.

774

775

776

777

778

41



The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles (46 values

each) along the � = �50� longitude line. At � = 44�, the true temperature field is zero due to the lateral

boundary conditions.

817

818

819

820

821

42

−60 −55 −50 −45 −40 −35 −30

2428

3236

4044

(a) SSH [m]

Longitude [°]

Latit

ude

[°]

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● −0.6

−0.4

−0.2

0.0

0.2

0.4

● ● ● ● ● ● ● ●

5000

4000

3000

2000

1000

0

(b) T [°C]

Latitude [°C]

Dep

th [m

]

24 26 29 32 35 38 41 44

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

0

2

4

6

8

10

12



The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles along the

� = �50� longitude line. Note that at � = 44�, the true temperature field is zero due to the lateral boundary

conditions.

774

775

776

777

778

41



The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles (46 values

each) along the � = �50� longitude line. At � = 44�, the true temperature field is zero due to the lateral

boundary conditions.

817

818

819

820

821

42


Application of LETKF

True sea surface height at 1st analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6

Initial guess of sea surface height

Longitude (degree)

Latit

ide

(deg

ree)

Estimated SSH at 1st analysis time

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6


Application of LETKF (2)

Estimated SSH at last analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6

True sea surface height at last analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6

Final SSH without assimilation True sea surface height at last analysis time

Longitude (degree)

Latit

ide

(deg

ree)

−60 −55 −50 −45 −40 −35 −30

24

28

32

36

40

44−0.6

−0.4

−0.2

0

0.2

0.4

0.6


•  RMS errors reduced to 10% (velocities to 20%) of initial error •  Slower convergence for NETF, but to same error level as LETKF •  CRPS (Continuous Rank Probability Score) shows similar behavior

Filter performances in NEMO

0 50 100 150 200 250 300 350

010

2030

4050

(a) for SSH

Time [days]

rela

tive

RM

SE /

CR

PS [%

]

0 50 100 150 200 250 300 3500

1020

3040

50

(a) for T

Time [days]

rela

tive

RM

SE /

CR

PS [%

]

RMSE_NETFRMSE_LETKFCRPS_NETFCRPS_LETKF

0 50 100 150 200 250 300 350

010

2030

4050

(a) for U

Time [days]

rela

tive

RM

SE /

CR

PS [%

]

0 50 100 150 200 250 300 350

010

2030

4050

(a) for V

Time [days]

rela

tive

RM

SE /

CR

PS [%

]

FIG. 9. Comparison of NETF and LETKF in terms of RMSE (black/gray) and CRPS (red/orange). The lines

represent the field-averaged relative RMSE and CRPS, respectively, for all prognostic variables, i.e., (a) SSH ,

(b) T , (c) U and (d) V , which are defined in Sec. 5.b. The legend in (b) is valid for all panels.

841

842

843

49

Tödter, Kirchgessner, Nerger & Ahrens, MWR 144 (2016) 409 – 427

SSH: Relative error reduction T: Relative error reduction


•  Smoother reduces filter errors by ~10% •  Can be useful as smoothing is cheap to compute

Applying the smoother

day120 240 360 480 600

RM

S er

ror i

n SS

H [m

]

0

0.01

0.02

0.03

0.04

0.05

0.06FilterSmoother lag 30d

RMS errors

lag [days]20 40 60 80 100 120

Ro

10 -5

10 -4

10 -3

10 -2

SSH [m²/day]T [(°C)²/day]U [(m/s)²/day]V [(m/s)²/day]

Roughness

Ro =

Z ✓dRMSE

dt

◆2

dt

•  Roughness of estimated trajectory is strongly reduced (smoothed)


•  Consider relative improvement

Different smoothing impact

lag [days]0 20 40 60 80 100 120

Ri [

%]

0

2

4

6

8

10

12

14

SSHTUV

0 20 40 60 80 100 120lag [days]

0

2

4

6

8

10

12

14

Ri [

%]

sshtuv

L-NETF LETKF

Ri = 100 ·✓1� RMSE

smoother

RMSEfilter

◆

•  Similar behavior for ssh (sea surface height) •  Distinct for T

!  Effect of distinct update schemes (NETF uses observation values for both state and ensemble update)

optimal lag

optimal lag optimal lag

for T


Summary

!  Nonlinear ensemble transform filter/smoother (NETF/S)

"  Update state estimate as particle filter

"  Transform ensemble using covariance matrix

!  NEMO ocean test case

"  NETF filtering performance similar to LETKF

"  Slower convergence

"  Sensitive on ensemble size

"  Successful smoothing

•  Dependence on lag distinct for LETKS & NETS (due to different update schemes)

Thank you!

Documents

High-Dimensional Nonlinear Data Assimilation with the ... · High-Dimensional Nonlinear Data Assimilation with the Nonlinear Ensemble Transform Filter (NETF) and its Smoother Extension