Craig H. Bishop Elizabeth A SatterfieldKevin T. Shanley, David Kuhl, Tom Rosmond, Justin

McLay and Nancy BakerNaval Research Laboratory

Monterey CA

November 2, 2012

Hidden Error Variancesand the optimal combination of

static and flow dependent variances

1

Introduction: Definitions• Error Variance: Mean of a large number of squared forecast

errors.• Flow Dependent Error Variance: Mean of a large number of

squared forecast errors given a particular flow. (In order to obtain a large number of errors the “flow” or “condition” must repeat itself).

• Hidden Error Variance: A flow dependent error variance that is formally unobservable because the particular flow does not repeat itself.

“A conundrum of predictability research is that while the prediction of flow dependent error distributions is one of its main foci, chaos hides flow dependent

forecast error distributions from empirical observation.”

Bishop and Satterfield (2012a,b, MWR, in press), Satterfield and Bishop (2012ab, to be submitted)

Binned Ensemble variance (mm^2)

Binned Ensemble variance0 2 4 5 6

45o

0

1

2

3

4

5

6

7

ET-ctlET-ens

731

a) 850-hpa T (oC)

Binned Ensemble variance0 20 40 50 60

45o

0

10

20

30

40

50

60

3010

b) 850-hpa Td (oC)

Binned Ensemble variance0 20 40 50

45o

0

10

20

30

40

50

3010

c) 850-hpa u (m s-1)

Binned Ensemble variance0 20 40

45o

0

10

20

30

40

3010

d) 850-hpa v (m s-1)

Spread-skill plot for COAMPS simulations relative to the control (solid line) and Mean (dashed line)

0 200 400 500 600

45o

0

100

200

300

400

500

600ET-ctl

ET-ens

300100

48-h total precipitation (mm)

Binned squared

Error (mm)^2

Mean ET-ctl = 119.41

Mean ET-ens = 100.82

Spread-skill plot for COAMPS simulations similar to for 48-h total accumulated precipitation (mm)^2.

Significant spread-skill relationships were found for all variables – including precipitation

Previous work: spread-skill diagramsCollect innovations (ob – fcst) corresponding to similar ensemble variances into a bin. Compute bin averaged squared innovation. It should increase with ensemble variance.

These diagrams do not reveal(a) the climatological range of true error variances, nor (b) the degree of variation of ensemble variance given a true error variance.

3

Overview

• Observations of hidden error variances using replicate systems.• Empirical determination of key pdfs.• Analytic model of statistical relationships of ensemble variances and

true error variances. • Use of (innovation, ensemble-variance) pairs to estimate

parameters of analytic model.• Estimation of optimal weights for Hybrid.• Comparison of performance of Hybrid DA with weights from brute

force tuning and weights from hidden error variance theory.• Other Hybrid results• Conclusions

4

What is the true flow dependent error variance ?

• Imagine an unimaginably large number of quasi-identical Earths.

Each Earth has one true state and one prediction but these differ from one Earth to another.

Collect all Earths having the same true state but differing forecasts of this state to

defin . T|e

t f

f t x x

x x

he error distribution is the distribution of differences

between individual forecasts and single truth within this set.

Collect all Earths having the same historical observati

fixed-tr

ons b

u

u

th

f

t diy

1 2

fering true atmospheric

states to define . The differences between individual truths and the

mean truth within this

| , ,...

fixed-obs set define the forecast error distribution. i

t ti x x y y

(Slartibartfast – Magrathean designer of planets, D. Adams, Hitchhikers …)

5

25000 Lorenz Model ReplicatesReveal Hidden Error Variance

• Using a 10 variable Lorenz ’96 model with additive model error and a 20 member Ensemble Transform Kalman Filter (ETKF) data assimilation scheme, we created 25,000 independent time series of analyses and forecasts, each having the same true state but differing random draws of observation error.

• True Error Variances were then obtained for each spatio-temporal point by averaging the squared forecast error for this point across the 25,000 replicates.

2( | ) ~ ( , )L s k 2 1~ ,prior prior

First demonstration of ETKF accurately predicting true flow dependent error variance in non-linear system.

Scatter plot of ETKF ensemble variance from a single replicate system as a function of true error variance. The true error variance is estimated from all 25,000 replicate systems. The linear fit to the points on the scatter plot is governed by the equation .

6

Controlled accuracy of ensemble variances by degrading ETKF variances

• A primary objective is to show how pdf of true error variances given an imperfect ensemble variance changes as the accuracy of the ensemble variance changes.

• To do this, we created degraded ensemble variances by sampling a Gamma

distribution with mean equal to the ETKF variance and relative variance determined by an “effective ensemble size” M.

2( | ) ~ ( , )L s k 2 1~ ,prior prior

2 2 1| ~ ,s

(b) M=4 (a) M=8

Examples of assumed likelihood gamma pdfs of ensemble variances with a mean of unity. Panel (a) is for an effective ensemble size of M=8, or equivalently, a relative variance of 2/7. Panel (b) is for an effective ensemble size of 4, or equivalently, a relative variance of 2/3. 7

Histograms of true error variance given an imperfect ensemble variance

2( | ) ~ ( , )L s k 2 1~ ,prior prior

2 2 1| ~ ,s The histograms give an empirical estimate of the pdf of true error variances given a constrained range of sample variances for an 8 member ensemble. The ranges are given on each figure; they correspond to the 2nd and 34th bins, respectively, of 35 bins of true error variance. The solid lines give the fit of an inverse-gamma function to the distribution of true error variances in each bin.

Inverse-gamma distribution is a very good fit to empirically derived histogram of true error variances given an ensemble variance for all ensemble variance categories.

M=8

M=8

8

Climatological pdf of true error variances

2( | ) ~ ( , )L s k 2 1~ ,prior prior

2 2 1| ~ ,s

Prior climatological distribution of true error variances. Bars show the probability density histogram of forecast error variances. Solid line shows the fit of the pdf (eq 4) to the data. The thick dashed line marks the mean of both the pdf and the data.

M=8

M=8

Inverse-gamma distribution gives a reasonable fit to empirically derived prior climatological pdf of true error variances.

9

Empirical estimation of pdf of true error variance given ensemble variance from 25000 trials

2 1prior ( ) ~ ( , )

2( | ) ~ ( , )L s k

(a) M=8, empirical (b) M=2, empirical

Red lines depict empirical estimate of pdf of true error variance (ordinate axis) given fixed values of ensemble variance (abscissa axis). Thin green and blue lines give

the mode and mean of the empirical estimates of the mode and mean of these estimates. Panels (a) and (b) show the empirical estimates for random sample

ensembles of sizes M=2 and M=8, respectively. The grey shading gives an inverse-gamma pdf fit to the climatological pdf of true error variances. 10

An analytic model of hidden error variance

Assumption 1: The error of the deterministic forecast is a random draw from a Gaussian distribution, whose true variance i

2 is a random draw from a prior climatological inverse gamma pdf of error variances.

2

2 2 1 2 2 2min

, ~ 0,

~ ( ) , var ,

f t f fi

i prior

x x N

An analytic model of hidden error variance

Assumption 2: Ensemble variances are drawn from a likelihood gamma pdf of ensemble variances with mean a(i

2 - min2)+s2

min

2 2 2 2min

2 2 2 2 2min

2 2 2 2 2 2 2 2 2

( )

| ( )

| ~  | | ,var | ,

i i min

i i i min

i i i i i i i i min

s a s

s a s

s L s s s s

stochastic

(b) M=4 (a) M=8

An analytic model of hidden error variance

Bayes’ Theorem defines the posterior inverse gamma pdf of error variances given an imperfect ensemble variance si

2

2 2 2prior2 2

post2 2 2

prior0

( | ) ( )( | )

( | ) ( )

ii

i

L ss

L s d

Climatological Prior

Distribution

Likelihood distribution of s2

given a particular 2

13

It can be shown that using assumptions 1 and 2 in Bayes’ theorem gives a that is

itself an inverse gamma distribution.

(a) M=8, empirical

(c) M=8, analytic (d) M=2, analytic

(b) M=2, empirical

2 2

2 2

Analytic model of | gives good match

to empirically estimated |

s

s

Green lines give modeBlue lines give mean

Given an ensemble variance, there are a broad range of possible true error variances.

Current DA schemes require a single value.

For the minimum error variance estimate, use the posterior mean.

For the maximal likelihood estimate, …

For QC, … 14

Posterior mean error variance is a Hybrid combination of static and ensemble variances

2 22 2 2 2min

min

1 1

| 1 1 1 1R R

R R R R

s sR Psa

P R P R

Flow dependent ensemble variance

2

2

 : true error variance

: ensemble variances

Static climatological mean error variance

2 2

: Relative variance of prior climatological pdf of error variances

: Relative variance of likelihood pdf of ensemble variances |

R

R

P

R L s

i. As the stochastic variation of ensemble variance about the true variance goes to zero, the weight on the ensemble variance goes to 1.

ii. If there is any imperfection in the flow-dependent ensemble variance, the optimal error variance estimate gives weight to the climatological covariance.

iii. If there is no variance of the true error variance, the weight on the static variance goes to 1.

Purely flow dependent error variance models are sub-optimal

Implications for Ensemble DA?Implications for 4DVAR?

15

Problem

How can the climatological relative variance of true variance and/or the

relative variance of ensemble variances given a true variance be determinedwhen the true variance is hidden from direct e

R

R

P

Rmpirical observation?

16

Solution: Equations that define hidden parameters from data assimilation output

17

2

2 2

New equations have been derived that estimate hidden error variance parameters

from a long time series , , 1, 2,..., of (innovation, ensemble-variance) pairs

, where is the observation err

i iv s i n

v R R

4222 2 2 2

min

2 22 2 2 2 2

min min2

2min

or vriance (1)

var var (2)3

covar ,, recall that | = (3)

var

-

Rv

P R R

v sa s a s

s

2 2min

2 2min2 2

min

2 2

2

assigned by designer or min over all if sample is large (4)

(5)

var |

i

R

s s i

s s

as

Rs

2 2 2

2 22 2 2 22 2minmin

var var 2 (6)1var|

s a

Mas

• Retrieved hidden parameters, var(s2), s2, a and M are shown in plots (a), (b), (c) and (d), respectively

• “Light grey bars: M=2, Dark grey bars: M=8• The “given” ensemble sizes in (d) are the random sample ensemble sizes used to

degrade the quality of the ETKF ensemble variance.

Equations recover hidden parameters “observed” by replicate systems

• “Observed” values are obtained from 175 DA cycles of the 25,000 “replicate systems”

• Minimum, mean and maximum of the values retrieved from 21 single system independent time series of

• with n=2,000,000. 2, , 1,2,...,i iv s i n

Each plot summarizes information from 60 independent retrievals. The values marked as min, mean, max and std are the minimum, mean, maximum and standard-deviation of the values retrieved from 60 completely independent synthetically generated data sets.

Recovery of min(sigma^2) is inaccurate when min(sigma^2) is small

These tests pertain to synthetic data generated using the analytical model of hidden error variance

The “specified” were set equal to values previously retrieved from Lorenz model experiments with a “given” M=8 and differing values of the model error q.

Each retrieval is from 2,000,000 (innovation, ensemble-variance) pairs synthetically generated from specified distributions.

Variation of weights for mean of posterior distribution of true error variances with model error q and given effective ensemble size M. Black bars give the weights for the de-biased flow-dependent ensemble variance while grey bars give the corresponding

weights for the static mean of the climatological error variances.

Variation of optimal weights withmodel error and ensemble size, M

Ensemble variance weight in dark grey.Static variance weight in light grey.q gives model error variance parameterM gives an “effective ensemble size” corresponding to the relative variance of a random normal ensemble of size M.

The weight on the ensemble variance increases with ensemble size

The weight on the static variance increases as model error variance increases20

Use of recoveries in Hybrid DA

2 22 2 2 2min

min

22 minEnsemble minmin climatology climatol

1

| 1 , where 1 1

This is a Hybrid of the form

1

R

R R

ff

Hybrid

s s Rs Ra

P R

sa a

PP Q Pogy

f

Flow-dependent ensemble prediction

Covariance matrix of unavoidable errors(?)

Static covariance

21

As a start, let’s guess that the optimal weights for error variance prediction are “useful” weights for error covariance prediction.

{In the following 3 examples, we assume that =0}

Application to Hybrid DA: Lorenz model 1, perturbed observations.

Analysis

0.08

0.085

0.09

0.095

0.1

0.105

0.11

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Alpha

RM

S

best practice hybrid

standard hybridoptimal weights hybrid - mean

• A suboptimal M=32 member ensemble is generated using a perturbed observations update.

• A climatological error covariance matrix (Pf

climatology) is formed by collecting forecast errors for 100,000 time steps (using an 100% ensemble based error covariance matrix)

• Pfhybrid is computed at each time step and

used in the ETKF DA scheme to obtain an analysis, which is cycled.

• We compute the “best practice” hybrid and the “standard” hybrid for all alpha values for comparison.

Forecast: 2 time steps

0.095

0.1

0.105

0.11

0.115

0.12

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Alpha

MSE

“Best Practice” hybrid: The ensemble based Pf is corrected by a factor of

2

2s

Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.

• The eq’s include a kurtosis term which is likely to be sensitive to data QC decisions based on the size of innovations.• Fortunately, it may be shown that the weight for the ensemble

variances is entirely independent of this term .

• The weight for the static term can then be obtained by insisting that the average of Hybrid variance be consistent with innovation variance.

23

Possible approaches to concerns in application of theory to Hybrid DA

2 2

2

covar ,

varens

v sw

a s

2 2 2

2 2

2.

ens c guess

ensc

guess

w s w

w sw

NAVDAS-AR-Hybrid ResultsLow resolution results

Hybrid weights computed for 6 distinct regions using new theory

Alpha=0.5

Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.

NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0

RMS wind error radiosonde verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)

NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0

RMS wind error self-analysis verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)

NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0

RMS error global radiosonde verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)

NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0

Geopotential Height Anomaly Correlation (verification against self-analysis). Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)

Conclusions1. A simple theory of the relationships between ensemble variances and true

error variances has been developed.2. This theory provides a new method for estimating from an archive of

(innovation, ensemble-variance) pairs a. the prior climatological pdf of true error variances, b. the likelihood pdf of ensemble variances given a true error variance,c. the posterior pdf of true error variances given an ensemble variance, andd. the mean of (c) as a weighted sum of a static and an ensemble variance.

3. The pdfs 2a and 2c are well approximated by inverse-gamma distributions for the Lorenz 96 system.

4. Result (2d) provides a theoretical justification for Hybrid error covariance models that linearly combine static and flow-dependent covariances.

5. The ansatz that the optimal weights (2d) for error variances are useful for error covariances is true for the Lorenz ’96 system with perturbed obs Hybrid DA and a low resolution Hybrid-4DVAR version of the Navy’s operational DA scheme.

6. Enables Hybrid weights to be defined regionally at a fraction of the cost of weights obtained via trial and error.

7. QC and ensemble post-processing applications are also possible.29