The  Na'onal  Center  for  Atmospheric  Research  is  sponsored  by  the  Na'onal  Science  Founda'on.  Any  opinions,  findings  and  conclusions  or  recommenda'ons  expressed  in  this  publica'on  are  those  of  the  author(s)  and  do  not  necessarily  reflect  the  views  of  the  Na'onal  Science  Founda'on.  

 ©UCAR  2014  

Ensemble  Verifica/on:  Part  II  Tim Hoar, NCAR �

Ensemble Verification: definition?�

The atmospheric science literature is rich with ways to quantify the accuracy of a forecast. I am not going to discuss any of these. Whole conferences could be organized around

this topic.��

All too often the poster/presentation states “I pushed the EnKF button and here are the results.” with little to no

regard as to whether or not the assimilation effectively used the information in the observations or whether the ensemble

was useful.��

I am going to focus on metrics that inform about the effectiveness of an ensemble forecast assimilation system.�

�

ICAP  tjh  2  of  21  

The Big Questions: �

1.  Are you looking at the prior (Good!) or the posterior (nearly useless for assimilation assessment – think ‘direct replacement’)?�

2.  Do you have an appropriate ensemble?�•  Underdispersed …. filter divergence�•  Overdispersed … uninformative�•  Biased … �

3.  Are your metrics skewed by observation rejection?�

4.  Where and why are observations being rejected?�ICAP  tjh  3  of  21  

Introduction to DART�

DART is the Data Assimilation Research Testbed, an �open source, freely available, ensemble data assimilation system that works with many geophysical and dynamical models and many types of observations.��DART is Fortran-based, scales well into the thousands of processors, has tutorials and documentation, and a small support staff.��We also have tools to evaluate the performance of an assimilation system– as opposed to a forecast – I’m going to use those tools to demonstrate what we believe to be important considerations.�

ICAP  tjh  4  of  21  

DART “home page”: �http://www.image.ucar.edu/DAReS/DART �

Overview article of DART:

Anderson, Jeffrey, T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, A. Arellano, 2009: The Data Assimilation Research Testbed: A Community Facility. Bull. Amer. Meteor. Soc., 90, 1283–1296. doi:10.1175/2009BAMS2618.1

Diagnostics�•  Was Assimilation Effective?�•  Observation-Space Diagnostics�•  ... �•  Histograms�•  State-Space Diagnostics�•  …�

ICAP  tjh  5  of  21  

Schematic for any ensemble system: �

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

1. Models advance�

2. Compare the model states to the observation.�

3. Compute increments�

4. Update model states�

1. Models advance�

ICAP  tjh  6  of  21  

Intro to DART-centric view and tools  

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

Stat

e-"

"Obs

erva

tion

-�sp

ace"

""s

pace�

ICAP  tjh  7  of  21  

Intro to DART-centric view and tools  

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

Prior" " " " "Posterior�

ICAP  tjh  8  of  21  

Intro to DART-centric view and tools  

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

Prior�state-space�diagnostics�

Posterior�state-space�diagnostics�

Prior observation-space�diagnostics�

Posterior observation-space�diagnostics* �

*Not really done as pictured, must calculate posterior state and reapply h.�

ICAP  tjh  9  of  21  

Intro to DART-centric view and tools  

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

Prior�state-space�diagnostics�

Posterior�state-space�diagnostics�

Can always do these.�

Need some model ‘truth’ to do these.�

Prior observation-space�diagnostics�

Posterior observation-space�diagnostics�

ICAP  tjh  10  of  21  

Some DART tools for STATE-space diagnostics: �

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tk

Some of these aggregate over a region, �some calculate metrics for each level, �some only make sense if you know the Truth …�

rou/ne   purpose  

plot_bins.m   Plots  the  rank  histograms  (usually  at  a  loca'on)  

plot_correl.m   plot  the  spa'al  correla'on  of  the  ensemble  against  a  single  loca'on  

plot_ens_'me_series.m   Plots  the  evolu'on  of  the  ensemble  (1  loca'on,  1  variable)  

plot_total_error.m   Plots  the  RMSE  of  all  variables,  all  loca'ons*  (if  Truth  is  available)  

plot_ens_err_spread.m   Plots  the  evolu'on  of  the  ensemble  error  and  spread    

ICAP  tjh  11  of  21  

Some DART tools for OBSERVATION-space diagnostics: �

Anderson: Ensemble Tutorial 14 9/8/06

How an Ensemble Filter Works for Geophysical Data Assimilation

6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...

y

****

h hh

y

tk+2

tkrou/ne   purpose  

obs_diag.f90   Calculates  diagnos'cs  from  observa'ons,  writes  out  a  netCDF  file.  

plot_evolu'on.m   plot  the  'me-­‐evolu'on  of  the  diagnos'c  

plot_profile.m   plot  a  'me-­‐averaged  ver'cal  profile  

plot_rank_histogram.m   plot  rank  histograms  (can  also  be  done  with  ncview)  

obs_seq_to_netcdf.f90   creates  a  netCDF  file  of  the  observa'ons  

link_obs.m   Creates  graphics  to  explore  loca'ons/values/QC  etc.  

Can always do these.�

Prior observation-space�diagnostics�

Posterior observation-space�diagnostics�

ICAP  tjh  12  of  21  

an example of plot_evolution �

08/01 08/06 08/11 08/160

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

month/day − Aug.01,2005 06:00:00 start

rmse

Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa

forecast: mean=1.1971 analysis: mean=0.98162

forecastanalysis

data file: /Users/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc

0

200

400

600

800

1000

1200

1400

1600

1800

2000

# of

obs

: o=

poss

, +=u

sedCan plot:�

RMSE�Bias �Spread�Totalspread�Ens mean �QC value�

prior�

posterior�

# of observations�possible – ‘o’�

# of observations�used – ‘+’�

Simple mean �region �

scal

e fo

r #

of

obse

rvat

ions�

This experiment started from nearly identical conditions and the model dynamics caused it to diverge over time.�

ICAP  tjh  13  of  21  

evolution of two quantities�

08/01 08/06 08/11 08/160

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

month/day − Aug.01,2005 06:00:00 start

rmse a

nd tota

lspre

ad

Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa

rmse pr=1.1971, po=0.98162 totalspread pr=0.91985, po=0.81559

rmse

totalspread

data file: /Users/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc

0

200

400

600

800

1000

1200

1400

1600

1800

2000

# o

f obs : o

=poss, +

=used

Prior and posterior plotted for each timestep … sawtooth.�

Totalspread has a minimum corresponding to observation error.�

RMSE & totalspread similar�

ICAP  tjh  14  of  21  

compare multiple experiments�

12/26 01/05 01/15 01/25 02/04 02/14 02/24 03/06 03/160

0.1

0.2

0.3

0.4

0.5

0.6

0.7

02−Jan−1998 12:00:01 through 15−Mar−1998 12:00:00

fore

cast

spr

ead

12/26 01/05 01/15 01/25 02/04 02/14 02/24 03/06 03/16

0

7.1

14.3

21.4

28.6

35.7

42.9

50

# of

obs

(o=p

oss,

+=u

sed)

AtlanticXBT_TEMPERATURE @ 20 m

DARTPOP23DARTPOP48

DART also has the ability to calculate the metrics for ‘trusted’ observations, i.e. even when the observation is rejected.�

ICAP  tjh  15  of  21  

observation-space rank histograms�

plot_rank_histogram�?biased�

ncview can plot rank histograms�right from obs_diag.f90 output.�Can plot several levels on same axis.�

Underdispersed @ 200 hPa �?Overdispersed @ 850 hPa �

This requires the user to save the results of the forward observation operator for all ensemble members during the assimilation. Cannot calculate this ‘after the fact’.�

ICAP  tjh  16  of  21  

an example of plot_vertical�

Shaded areas correspond to vertical aggregation.�

# of observations�possible – ‘o’�

# of observations�used – ‘+’�

scale for # of obs�

Important not to consider the ‘spin up’ period.�

Priors are solid lines, �posteriors are dashed.�

ICAP  tjh  17  of  21  

an example of link_obs  

ICAP  tjh  18  of  21  

an example of link_obs  

Rotate view angle�

Enable observation selection or ‘brushing’�

e.g. observation �e.g. p

rior

mea

n�

DART QC�

Observa'on  processing  for  forecast  metrics:  

VariableX: � (analysisT, station, level, copy, ensemble, forecast_lead) | | | | | | | | | | | +- forecast length : 0,3,6,9,12,... | | | | +----------- ensemble member index | | | +------------------ obs value, prior, obs_err | | +------------------------- vertical level index | +--------------------------------- (horizontal) station index +------------------------------------------- analysis time/date

Anything goes: unsorted, unordered observations �

obs_seq_coverage: determines uniform observation network � obs_seq_verify:"

creates a netCDF file with variables ordered to make analysis easy�

ICAP  tjh  20  of  21  

www.image.ucar.edu/DAReS/DART  dart@ucar.edu  

For more information: �

MITgcm_ocean �

NAAPS�NCOMMAS�

PBL_1d�

POP�

AM2�BGRID �

CAM�

CLM�

COAMPS�

COAMPS_nest �MPAS_OCN �

MPAS_ATM�

NOAH �

PE2LYR �SQG�

WRF�

TIEGCM�

GITM�

CABLE�

WACCM�ROMS� wrfHydro �

ICAP  tjh  21  of  21