Hybrid Variational-Ensemble Data Assimilation at NCEP

Hybrid Variational-Ensemble Data Assimilation at NCEP

Daryl Kleist

1Workshop on Probabilistic Approaches to Data Assimilation for Earth Systems

Banff, Alberta, Canada – February 2013

with acknowledgements to Kayo Ide, Dave Parrish, Jeff Whitaker, John Derber, Russ Treadon, Wan-Shu Wu, Jacob Carley, and Mingjing Tong

NOAA/NWS/NCEP/EMC

Outline

• Introduction– (Brief) background on hybrid data assimilation

• Hybrid Var/Ens at NCEP

• OSSE-based hybrid experiments

• Future Work and Summary

2

f & e: weighting coefficients for fixed and ensemble covariance respectively

xt’: (total increment) sum of increment from fixed/static B (xf’) and ensemble B

k: extended control variable; :ensemble perturbations

- analogous to the weights in the LETKF formulation

L: correlation matrix [effectively the localization of ensemble perturbations]3

Hybrid Variational-Ensemble(ignoring preconditioning for simplicity)

• Incorporate ensemble perturbations directly into variational cost function through extended control variable– Lorenc (2003), Buehner (2005), Wang et. al. (2007), etc.

ekx

yxHRyxH

LxBxx

t1T

t

1

1T

ef1

fT

fff

2

1

2

1

2

1 N

n

nnββ,J ααα

N

n

nn

1eft xxx α

Single Temperature Observation

4

3DVAR

f-1=0.0 f

-1=0.5

Outline

• Introduction– (Brief) background on hybrid data assimilation

• Hybrid Var/Ens at NCEP

• OSSE-based hybrid experiments

• Future Work and Summary

5

6

Air Quality

WRF NMM/ARWWorkstation WRF

WRF: ARW, NMMNMM-BGFS, Canadian Global Model

Satellites99.9%

Regional NAMNMM-B

North American Ensemble Forecast System

Hurricane GFDLHWRF

GlobalForecastSystem

Dispersion

ARL/HYSPLIT

Forecast

Severe Weather

*Rapid Updatefor Aviation

ClimateCFS

1.7B Obs/Day

Short-RangeEnsemble Forecast

NOAA’s NWS Model Production Suite

MOM3

NOAH Land Surface Model

CoupledOceansHYCOM

WaveWatch III

NAM/CMAQ

Regional DataAssimilation

Global DataAssimilation

GDAS Hybrid: 22 May 2012

7

• Package included other changes• NPP ATMS (MW): 7 months after launch

• This is by far the fastest we have ever begun assimilating data from a new satellite sensor after launch

• GPS RO Bending Angle replaced Refractivity

• Summary of pre-implementation retrospective testing• Improved Tropical winds• Improved mid-latitude forecasts• Fewer Dropouts• Improved fits to observations of forecasts• Some improvement in NA precip. in winter• Increased bias in NA precip. – decreased rain/no rain skill in summer

(Improved by land surface bug fix)• Overall significant improvement of GFS forecasts

TC Track Error Reduction

8

HYBRID TEST

GFS OPERATIONAL

NHC/JTWC OFFICIAL

93DHYB-3DVAR

1000mb

500mb

NH

SH

Impact on Geopotential Height

NAM vs NAM parallels upper air stats vs raobs

10

Height RMS error

Day 1 = BlackDay 2 = RedDay 3 = Blue

Vector Wind RMS error

Ops NAM = Solid ; NAMB (with Physics changes) = Dashed ; NAMX (with physics changes and using global EnKF in GSI) = Dash-Dot

Ops NAM

NAMB (improvedphysics)

NAMX (improvedphysics + EnKF)

Thanks to Eric Rogers and Wan-Shu Wu

RAP GSI-hybrid vs. RAP GSI-3dvar

upper-air verification

3dvar

hybrid

Temp Rel Hum

Wind

28 Nov – 3 Dec 2012

+ 6 h forecast RMS Error

3dvar

hybrid

RAP hybrid DA using global ensemble

3dvarhybrid

Assimilation of NOAA-P3 Tail Doppler Radar (TDR) Data using GSI hybrid method for HWRF

• HWRF Model: 3 domains with 0.18-0.06-0.02 degree (27-9-3 km) horizontal resolutions, 43 vertical levels with model top at 50 hPa, with ocean coupling• TC environment cold start from GDAS forecast, TC vortex cycled from HWRF forecast• GSI hybrid analysis using GFS EnKF ensemble 80% of background error covariance from ensemble B. Horizontal localization 150 km, vertical localization 10 model levels for weak storm and 20 model levels for strong storm• Conventional data plus TDR data• Modified gross error check, re-tuned observation error and rejected data dump with very small data coverage for TDR data• 19 TDR missions for Hurricane Isaac, Leslie and Sandy

TDR data for Isaac 2012082712

Outline

• Introduction– (Brief) background on hybrid data assimilation

• Hybrid Var/Ens at NCEP

• OSSE-based hybrid experiments

• Future Work and Summary

13

14

Observing System SimulationExperiments (OSSE)

• Typically used to evaluate impact of future observing systems– Doppler-winds from spaced-based lidar, for example

• Useful for evaluating present/proposed data assimilation techniques since ‘truth’ is known– Series of experiments are carried out to test hybrid variants

• Joint OSSE– International, collaborative effort between ECMWF, NASA/GMAO,

NOAA (NCEP/EMC, NESDIS, JCSDA), NOAA/ESRL, others– ECMWF-generated nature run (c31r1)

• T511L91, 13 month free run, prescribed SST, snow, ice

Availability of SimulatedObservations [00z 24 July]

15

AMSUA/MSU AIRS/HIRS

AMSUB GOES SOUNDER

SURFACE/SHIP/BUOY

AIRCRAFT

SONDES

AMVS

SSMI SFC WIND SPDPIBAL/VADWND/PROFLR

3D Experimental Design

16

• Model• NCEP Global Forecast System (GFS) model (T382L64; post May 2011 version – v9.0.1)

• Test Period• 01 July 2005-31 August 2005 (3 weeks ignored for spin-up)

• Observations• Calibrated synthetic observations from 2005 observing system (courtesy Ron Errico/Nikki

Privi)

• 3DVAR• Control experiment with standard 3DVAR configuration (time mean increment compared to

real system and found to be quite similar)

• 3DHYB• Ensemble (T190L64)

• 80 ensemble members, EnSRF update, GSI for observation operators• Additive and multiplicative inflation

• Dual-resolution, 2-way coupled• High resolution control/deterministic component• Ensemble is recentered every cycle about hybrid analysis

– Discard ensemble mean analysis

• Parameter settings• f

-1=0.25, e-1=0.75 [25% static B, 75% ensemble]

• Level-dependent localization

Time Series of Analysis andBackground Errors

17

500 hPaU

850 hPaT

Solid (dashed) show background (analysis) errors

3DHYB background errors generally smaller than 3DVAR analysis errors (significantly so for zonal wind)

Strong diurnal signal for temperature errors due to availability of rawinsondes

3DVAR and 3DHYBAnalysis Errors

183DVAR 3DHYB 3DHYB-3DVAR

U

T

Q

Zonal Wind BackgroundErrors

19

3DVAR 3DHYB

Bf BEN

3DHYB_(Retuned Spread)

20

U

T

Q

New (RS) hybrid experiment almost uniformly better than 3DVAR

3DHYB_RS-3DVAR

21

4D-Ensemble-Var[4DENSV]

As in Buehner (2010), the H-4DVAR_AD cost function can be modified to solve for the ensemble control variable (without static contribution)

Where the 4D increment is prescribed exclusively through linear combinations of the 4D ensemble perturbations

Here, the control variables (ensemble weights) are assumed to be valid throughout the assimilation window (analogous to the 4D-LETKF without temporal localization). Note that the need for the computationally expensive linear and adjoint models in the minimization is conveniently avoided.

K

kkkkkkkk

N

n

nnJ1

1T

1

1T

2

1

2

1yxHRyxHL ααα

N

n

n

kn

k1

exx α

22

Hybrid 4D-Ensemble-Var[H-4DENSV]

The 4DENSV cost function can be easily expanded to include a static contribution

Where the 4D increment is prescribed exclusively through linear combinations of the 4D ensemble perturbations plus static contribution

Here, the static contribution is considered time-invariant (i.e. from 3DVAR-FGAT). Weighting parameters exist just as in the other hybrid variants.

K

kkkkkkkk

N

n

nn,J

1

1T

1

1T

ef1

fT

fff

2

1

2

1

2

1

yxHRyxH

LxBxx ααα

N

n

n

kn

k1

ef xxx α

Single Observation (-3h) Examplefor 4D Variants

23

4DVAR

H-4DVAR_ADf

-1=0.25H-4DENSVf

-1=0.25

4DENSV

Time Evolution of Increment

24

t=-3h

t=0h

t=+3h

H-4DVAR_AD H-4DENSV

Solution at beginning of window same to within round-off (because observation is taken at that time, and same weighting parameters used)

Evolution of increment qualitatively similar between dynamic and ensemble specification

** Current linear and adjoint models in GSI are computationally unfeasible for use in 4DVAR other than simple single observation testing at low resolution

25

4D OSSE Experiments

• To investigate the use of 4D ensemble perturbations, two new OSSE based experiments are carried out.

• The original (not reduced) set of inflation parameters are used.

• Exact configuration as was used in the 3D OSSE experiments, but with 4D features

– 4DENSV• No static B contribution (f

-1=0.0)

• Analogous to a dual-resolution 4D-EnKF (but solved variationally)

• To be compared with 3DENSV

– H-4DENSV • 4DENSV + addition of time invariant static contribution (f

-1=0.25)

• This is the non-adjoint formulation

• To be compared with 4DENSV

Impact on Analysis Errors

26

U

T

Q

4DENSV-3DENSV H-4DENSV-4DENSV

27

OSSE-based comparison of 3DHYB andH-4DENSV (with dynamic constraints)

U

T

Q

4DHYB-3DHYB

Something in the 4D experiments is resulting in more moisture in the analysis, triggering more convective precipitation

28

Summary of 4D Experiments

• 4DENSV seems to be a cost effective alternative to 4DVAR

• Inclusion of time-invariant static B to 4DENSV solution is beneficial for dual-resolution paradigm

• Extension to 4D seems to have larger impact in extratropics (whereas the original introduction of the ensemble covariances had largest impact in the tropics)

• Increased convection in 4D extensions remains a mystery (it is not the weak constraint on unphysical moisture as I originally hypothesized)

• Original tuning parameters for inflation were utilized. Follow-on experiments with tuned parameters (reduced inflation) and/or adapative inflation should yield even more impressive results.

29

Scale-Dependence Motivation(Courtesy: Tom Hamill)

Spectrum of Dual-Resolution Increment without SD-weighting

30

Spectrum of Dual-Resolution Increment with SD-weighting

31

Initial scale-dependent tests inand OSSE

32

Outline

• Introduction– (Brief) background on hybrid data assimilation

• Hybrid Var/Ens at NCEP

• OSSE-based hybrid experiments

• Future Work and Summary

33

Summary

34

• NCEP successfully implemented hybrid variational-ensemble algorithm into GDAS

• NCEP aggressively pursuing application of hybrid to other systems– Mesoscale (NAM), HWRF, Rapid Refresh (and HRRR follow on), storm scale ensemble

– Future Reanalysis• Have already run preliminary tests for 1981-1983 periods, attempting to capture QBO

transitions (a difficult problem for reduced observing system periods)

• Extensions to the GDAS hybrid are ongoing, including 4DEnsVar

GDAS Hybrid

35

• EMC targeting T1148 SL GFS implementation within next year– How to configure hybrid? What can be afforded computationally on new

machine?

• Merging DA and EPS efforts– Currently have separate EnKF (DA) and BV-ETR (EPS) cycles/perturbations

• Improving the ensemble for the hybrid– Stochastic physics (Jeff Whitaker)

– TC Initialization (Yoichiro Ota)

• Hybrid developments– Improved localization, flow-dependent (and/or scale-dependent) weighting

– Testing and preparing 4d-ensemble-var for implementation

– Helping efforts toward cloudy/precip. radiances

TC relocation for EnKF(work done by Yoichiro Ota)

1. Update TC center position (latitude and longitude) by the EnKF

2. Use updated positions as inputs to the TC relocation

3. Apply this procedure before the EnKF analysis and GDAS analysis

Apply TC relocation used in deterministic analysis to each ensemble member, but allowing TC structure perturbations and some TC position spread.

Blue: first guess positionRed: Updated positionGreen: TC vital position

The idea is to separate linear problem (TC location space) and nonlinear problem (actual relocation of fields).

Example: spaghetti diagram

Before relocation After relocation

TC relocation of this method can reduce the uncertainty on the TC position, maintaining the TC structure perturbations and some of the position uncertainty.

Courtesy Yoichiro Ota

Comparison with GEFS TC relocation

EnKF analysis with TC relocation EnKF 6 hour forecast perturbation + GEFS TC relocation

GEFS operational TC relocation scheme destroyed almost all initial position uncertainty and create very small spread around TC.Courtesy Yoichiro Ota

6th WMO Symposium on DA

39

• NCEP will be hosting the next WMO DA Symposium from 7-11 October 2013 at NCWCP

• Call for papers will be out soon

• BACKUP SLIDES

40

Spectral Cost Function

41

• Assume increment cost function is in spectral space

s1se

s αβν L

osTssTs

fss

2

1

2

1J,J ναν zBxz

yxHRyxH

LxBxx

st

1Tst

s1Tsse

sf

1Tsf

sf

ssf

2

12

1

2

1ααββα,J

sf

1sf

s xBz β

• Perform variable transforms

New Control Variables andSpectral-dependent Weights

42

• New spectral control variables then become

s1se

s νβα L

s1sf

sf Bzx

β

• GSI control variables are in physical space, however, so we introduce spectral transform operator, S

BzSSx1s

f1

f β

νβα LSS1s

e1

Extension for Dual-Resolution

43

N

n

nn

1e

1se

1e xSSx αβ

να L

• The extension to dual-resolution requires a further modification to apply the scale dependence to the ensemble-based increment, and not the control variable itself