Data Assimilation in Meteorology

Chris Budd

Joint work with Chiara Piccolo, Mike Cullen (Met Office)Melina Freitag, Phil Browne, Emily Walsh, Nathan Smith and Sian Jenkins (Bath)

Understanding and forecasting the weather is essential to the future of planet earth and maths place a central role in doing this

Accurate weather forecasting is a mixture of

• Careful modelling of the complex physics of the ocean and atmosphere

• Accurate computations on these models

• Systematic collection of data

• A fusion of data and computation

Data assimilation is the optimal way of combining a complex model with uncertain data

Observations from space

Upper-air observations Weather radar

Surface observations

Integrated forecasting process

NWP forecasts

Analysis Intervention

Fine-tuning

Forecast productsand guidance

Modelling the global atmosphere and ocean

Vertical exchange between layers of momentum, heat and moisture

Horizontal exchange

between columnsof momentum,

heat and moisture

Vertical exchange between layers of momentum, heat and salts by diffusion, convection and upwelling

Orography, vegetation and surface characteristics included at surface on each grid box

Vertical exchange between layersby diffusion and advection

15° W60° N

3.75°

2.5°

11.25° E47.5° N

Met Office Current Configurations

North Atlantic and Europe (NAE)

Met. Office Global/Regional Ensemble Prediction System (MOGREPS) became fully operational in Sep 2008 after 3 years of trials. Focus on short-range out to 72hr.

UK

• Global 25km L70 model (was 40km L50)

• Incremental 4D-Var Data Assimilation

• 60km 24m ETKF ensemble (was 90km L38)

• Regional NAE 12km L70 model

• Incremental 4D-Var DA

• 16km 24m L70 ETKF ensemble (was 24km L38)

• UK 1.5km model (stretched) (was 4km)

• Incremental 3D-Var DA

Data: Sources of observation

Observation Volumes in 6 hours

CategoryCount %

usedCategory Count %

used

TEMPs 637 99% Satwinds: JMA 26103 4%

PILOTs 307 99% Satwinds: NESDIS 142478 3%

Wind Profiler1355 39% Satwinds:

EUMETSAT220957 1%

Land Synops

16551 99% Scatwinds: Seawinds

436566 1%

Ships 3034 84% Scatwinds: ERS 27075 2%

Buoys 8727 63% Scatwinds: ASCAT 241626 4%

Amdars 64147 23% SSMI/S 532140 1%

Aireps 7144 12% SSMI 698048 1%

GPS-RO 776 99% ATOVS 1127224 3%

AIRS 75824 6%

IASI 80280 3%

Performance Improvements

Met Office RMS surface pressure error over the N. Atlantic & W. Europe

“Improved by about a day per decade”

Andrew Lorenc

DA Introduced

What are the causes of improvements to NWP systems?

1. Model improvements, especially resolution and sub grid modelling

2. Better observations

3. Careful use of forecast & observations, allowing for their information content and errors.

Achieved by variational data assimilation

e.g. of satellite radiances.

Basic Idea of Data Assimilation

Numerical Weather Prediction NWP calculation gives a predicted state with an error

True state of the weather is

Make a series of observations y of some function of the true state

Eg. Limited set of temperature measurements with error€

H(x t )€

xb

Now combine the prediction with the observations

€

x t

Both the NWP prediction and the data have errors.

Can we optimally estimate the atmospheric state which is consistent with both the prediction and the data and estimate the resulting error?

NOTE: Approximately

10^9 degrees of freedom

10^6 data points

So significantly underdetermined problem

Assume initially:

1. Errors are unbiased Gaussian variables

2. Data and NWP prediction errors are uncorrelated

3. H(x) is a linear operator

yBest state estimate

analysis

Use to produce forecast

6 hours later

Data

NWP prediction

€

xa

€

xa

€

x f

€

x f

€

xB

€

xB

Can estimate using Bayesian analyis:

Maximum likelihood estimate of data y given

Best RMS unbiased estimate of the true state: BLUE

Minimum error variance

€

P(x t y)

P(x t )P(y) = P(y x t )

LikelihoodPrior

Posterior€

xa

€

x t

Assumptions about the error

Data error: Gaussian, Covariance R

Background (NWP) error: Gaussian, Covariance B

BLUE: Find which minimises

€

J(xa ) =1

2xa − xb( )

TB−1 xa − xb( ) +

1

2(Hxa − y)T R−1(Hxa − y)

€

xa

NOTE:

€

P(x y) /P(x) = e−J (x )

€

xB

If R and B are known then the best estimate of the analysis is

€

xa = xb + K(y − Hxb ), K = BHT (R + HBHT )−1

A = KRKT + (I − KH)B(I − KH)T

Kalman filter: Continuously updates the forecast and its error given the incoming data.

Covariance of the analysis error

€

J(xa ) =1

2xa − xb( )

TB−1 xa − xb( ) +

1

2(Hxa − y)T R−1(Hxa − y)

Implementation:

In the context of minimising the functional

This is implemented as 3D-VAR (since 1999 in the Met Office)

: Background, derived from 6 hour NWP forecast

: Analysis

: NWP forecast using as initial data

€

xB

€

xa

€

x f

€

xa

Ensemble Kalman Filter EnKF

This is a widely used Monte Carlo method that uses an ensemble of forecasts to estimate the terms in the Kalman filter

Idea: Take a large number of initial states and estimate the resulting background states

Estimate

€

x =1

Nxb,i,∑ , B =

1

N −1xB ,i − x ( ) xB ,i − x ( )

T∑

€

x i

€

x i€

x i

€

x i

€

x i

€

x i

€

xB ,i

€

xB ,i

€

xB ,i

€

xB ,i

€

xB ,i

€

xB ,i

y

Basic Filtering Idea

Advantages: Works well with high dimensional systems

Disadvantage: EnKF inaccurate with strong nonlinearity eg. Shear flow [Jones]

€

xa

€

xB

€

xa

€

xB

Correctedforecast

Previousforecast

4D VAR … Preferred variational method

Use window of several observations over 6 hours

Assimilation Window

9 UTC 12 UTC 15 UTC

Obs.

x

Time

xb

xa

Jo

Obs.

Jo

Jb

Obs.

Jo

Obs.

Jo

4D-VAR idea: Evolutionary model M (nonlinear)

Unknown initial state

Times Over a time window

Leads to state estimates

Data yi over window

Find so that the

estimates fit the data

Smoothing

€

x0

€

x1,x2,...

€

x0

€

t = t0, t1, t2,L

€

J(x0) =1

2x0 − xb( )

TB−1 x0 − xb( ) +

1

2(Hx i − y i)

T R−1(Hx i − y i)i= 0

N

∑

Minimise

Subject to the strong model constraint

€

x i+1 = M i(x i)

At present assume perfect model, but can also deal with certain types of model error (both random and systematic) by using a weak constraint instead

€

J(xa ) =1

2x0 − xb( )

TB−1 x0 − xb( ) +

1

2(Hx i − y i)

T R−1(Hx i − y i)i= 0

N

∑

+ λ i

i= 0

N−1

∑ (x i+1 − M x i)

Usually solved by introducing Lagrange multipliers

And solving the adjoint problems

€

0 =∇Ji = HT R−1(Hx i − y i) + λ i−1 − λ i mi x i

€

0 =∇J0 = B−1(x0 − xb ) + HT R−1(Hx0 − y0) − λ 0 m0 x0

Solution: Find to minimise nonlinear function J

Need forward calculation to find and backward solve to find

VERY expensive for high dimensional problems!!! Only have limited time to do the calculation (20 mins)

Incremental 4D-Var: Cheaper!

1. Assume is close to

2. Linearise J about and minimise this function using an iterative method eg. BFGS

3. Repeat if needed (not usually)

BUT: Relies on assumption of near linearity to work well

€

x0

€

x0

€

x i

€

xB

€

xB

€

λi

Very effective method!!

Met Office operational in 2004

[Lorenc, …. ]

Used by many other centres

Estimation of the background and covariance errors

Good estimates of the covariance matrices R and B are important to the effectiveness of 4D-VAR

1. To get the physics correct

2. To avoid spurious correlations between parameters

3. To give well conditioned systems

NOTE: B is a very large matrix, difficult to store and very difficult to update. Impractical to calculate using the Fokker-Plank equation

R Different instrument error characteristics and errors of

representativeness

B Enormous: 10^8 x 10^8

Deduce structure from:

Historical data

Known dynamical and physical structure of the atmosphere eg. Balance relationships

[Bannister]€

1 5 6

2 9 4

8 0 3

⎛

⎝

⎜ ⎜ ⎜

⎞

⎠

⎟ ⎟ ⎟

Build meteorology into the calculation of B through

Control Variable Transformations (CVTs)

IDEA: Choose more ‘natural’ physical variables which have uncorrelated errors so that the transformed covariance matrix is block diagonal or even the identity

Set€

χ

€

δx = Uχ = U pUvUhχ , B = UUT

Reduces the complexity of the system AND gives better conditioning for the linear systems

€

U p−1

€

Uv−1

€

Uh−1

Reduces vertical correlations by projecting onto empirical orthogonal vertical modes

Separates physical parameters into uncorrelated ones eg. temperature, wind, balanced and unbalanced

Effective, but errors arise due to lack of resolution of physical features leading to spurious correlations.

Reduces horizontal correlations by projecting onto spherical harmonics

Eg. Problems with stable boundary and inversion layers and assimilating radiosonde data

Poor resolution leads to inaccurate predictions of fog and ice

Solution one: increase global resolution

VERY EXPENSIVE!!!

Solution two: locally redistribute the computational mesh to resolve the features

Cheap and effective! [Piccolo, Cullen, B,Browne, Walsh]

€

δx = Uχ = U pUaUvUhχ , B = UUT

Adjust the vertical coordinates to concentrate points close to the inversion layer and reduce correlations

Introduce an extra transformation

€

Ua−1

Adaptive mesh transformation applied to latitude-longitude coordinates

Do this by using tools from adaptive mesh generation methods for PDES

€

dz

dξ= M(z)

Set: z original height variable

new ‘computational’ height variable

€

ξ

Relate these via the equation

M called the ‘monitor function’ [B, Huang, Russell, Walsh]

Take M large if there is active meteorology

Eg. High potential vorticity

€

M = 1+ c 2 ∂θ

∂z

⎛

⎝ ⎜

⎞

⎠ ⎟

2

Initially use background state estimate, then update

© Crown copyright Met Office

Monitor function and the Adaptive Grid

Piccolo&Cullen

QJR Met Soc 2011

© Crown copyright Met Office

First calculation

UK4 domain: 3 Jan 2011 00z

Updated calculation

Applied to the Met Office UK4 model

RMS T (K) RH (%) u (m/ s) v (m/s)Control 0.76 0.045 1.32 1.16

Test 0.64 0.045 1.29 1.16

Nobs 1011 901 819 819

Test case: 8th Feb 2010.

Significant reduction in RMS error especially for temperatures Piccolo&Cullen, QJR Met Soc 2011

Particularly effective for the 2m temperatures

Used together with Met Office Open Road software to advise councils on road gritting over Christmas

Adaptive mesh implemented operationally in November 2010.

Now extending it to a fully three dimensional implementation [B,Browne,Piccolo]

Other refinements to 4D-Var

Change in the background norm [Freitag, B, Nichols]

Total variation: Gives significantly better resolution of fronts, shocks and other localised features

But .. Hard to implement in high dimensions!!

€

J(x0) =1

2TV (x0 − xb ) +

1

2(Hx i − y i)

T R−1(Hx i − y i)i= 0

N

∑

Dealing with model error

If model has random errors with Covariance C can extend

4D-Var to find the minimiser of

€

J(x0) =1

2x0 − xb( )

TB−1 x0 − xb( ) +

1

2(Hx i − y i)

T R−1(Hx i − y i)i= 0

N

∑ +

1

2∑ x i+1 − M i(x i)( )

TC−1 x i+1 − M i(x i)( )

€

x0, x1, x2,L

However, most model errors eg. Numerical errors are systematic. Dealing with these and quantifying the uncertainty in DA is an area of active research

[Jenkins, B, Smith, Freitag], [Cullen&Piccolo], [Stuart]

Dealing with nonlinearity

Better use of appropriate (eg. Lagrangian) data

Tuning method to data [Jones, Stuart, Apte]

Use of particle filters

and MCMC methods [Peter Van Leeuwan]

Lot of research into finding a compromise between dealing with the high dimensionality and nonlinearity in the system

Conclusions

Data assimilation is an optimal way of

merging models with data

Useful for model tuning, validation,

evaluation, uncertainty quantification and reduction

Very effective in meteorology

Many other applications to Planet Earth

eg. Climate change, oil reservoir modelling, geophysics, energy management and even crowd behaviour

Problems with the simple Kalman Filter

• Assumption of Gaussian random variables

• Assumption of linearity

• Assumption of known covariances

• Covariance matrix B is VERY large for meteorological problems

• Minimisation is a large complex problem

Problems with 4D Var Accuracy

• Estimation of the background covariance matrix

• Ill conditioning of the linear systems

• Reliance of near linearity

• Inappropriate use of background data

• Incorrect covariance between data

• Unresolved random and systematic model error

• Poor resolution in the model

Improvements subject to much research