Poor Man's EPS experiments and LAMEPS plans at the Met Office

1 LAM EPS Workshop, Madrid, 3-4 October 2002

Ken Mylne and Kelvyn Robertson

Met Office

Poor Man's EPS experiments and LAMEPS plansat the Met Office


Why PEPS (Poor Man’s EPS)? Storms of Dec 1999 over Europe were poorly

forecast by most deterministic models, even at 24h– Need for effective short-range ensemble to reduce risk of

missing severe weather events

Existing operational ensembles (eg ECMWF) designed for medium-range (3-10days)

– some evidence of poor performance for severe events in short-range

PEPS is an ensemble formed by combining the operational output from several NWP centres

– provides a relatively cheap way of obtaining short-range ensemble forecasts


Why PEPS might work Multi-model multi-analysis ensemble

– experiments in USA have shown this is important (eg Hou et al, 2001; Stensrud et al, 1999)

Random sampling of initial condition errors– may be important for estimating probabilities

at short-range

Previous studies (eg Ziehmann, 2000) have shown encouraging results


Preliminary system 9 models Low-res (5x5°) H500 and pmsl

only Output every 24h Data stored and

used by VT, not DT


Verification - Brier Skill

Brier Skill Scores, using the ECMWF EPS as ref.

Several PEPS configurations– all available models

– one model removed (all versions)

– all plus 6 members of EPS

– reduced combinations

Range of PMSL thresholds 126 days from 7th Feb to 12th

June 2001


Hi-Res PEPSSuccess of the preliminary system

encouraged us to set up a much larger PEPS system:

Larger ensemble – around 15 members from 9 models

Higher resolution– tests at 1.25x1.25°

– output every 12h

More fields– PMSL

– H500

– T850

– 2m Temp

– 10m Windspeed

– Precipitation


Data Exchange 9 centres agreed to supply forecast data

Data are pulled from FTP sites in near-real time– European data via ECMWF fast link

– Other centres via the internet

– Met Office UM

– ECMWF

– DWD

– Meteo-France

– BoM

– JMA

– KMA

– CMC

– NCEP

– Russia


Brier Skill - Winter DJF 2001/02 Results similar

to preliminary experiments

Reference EPS is 12 hours older due to late data cut-off

– provides the gain which could be achieved operationally


Effect of 12h Advantage Re-ran verification

without giving PEPS the 12h advantage

Apparent PEPS skill mostly comes from the 12h advantage

Without:– No skill at T+24

– Slight advantage at T+84

With 12h Without 12h


BSS - Different Weather Parameters

PMSL H500 T850 T 2m 10m WS

T+24

T+72

Results similar for all weather parameters:-


BSS - PMSL in Regions

N. Hem. Europe N. Am. S. Hem.

PMSL results poor over S. Hemisphere.

T+72

T+24


BSS - 2m Temperature in Regions


T2m results poor over S. Hemisphere.

Best over continents but still poorer than EPS.

T+72

T+24


BSS - Wind Speed in Regions


Benefit for more extreme events in all regions:-

T+72

T+24


Rank Histograms PMSL over Northern

Hemisphere– over-spread at 24-

48h

– good spread but slight bias at longer lead-times

– EPS underdispersive at all times to T+120


Rank Histograms Focus on over-

spreading at T+24-48

– Northern hemisphere average hides strong regional bias over Europe

– still some over-spreading

– And an opposite regional bias over N. America


Rank Histograms Focus on over-

spreading at T+24-48

– Southern hemisphere shows stronger over-spreading

– probably due to analysis biases

Difficult to separate ensemble spread from differences in model biases

Some apparent over-spreading may be due to biases in the verifying ECMWF analysis

Need for bias correction


Rank Histograms Weather parameters PMSL

500hPa Height– Strong bias (analysis?)

– Some over-spreading

T850– Over-spreading


Rank Histograms Weather parameters PMSL

2m Temperature– Over-spreading

10m Wind Speed– Over-spreading

– Bias


Reliability Diagrams PMSL<970mb over

Northern Hemisphere– reliability good for

PEPS and for EPS


Reliability Diagrams H500<480dm over

Northern Hemisphere– some general

under-forecasting - possibly bias in ECMWF analysis, as seen in Rank Histograms


Reliability Diagrams 2m Temperature

– <260 deg C

– better reliability than EPS for all thresholds

– <280 deg C

– <300 deg C


Conclusions on PEPS PEPS advantage over EPS was due to the 12h

lag applied to EPS– little scientific advantage of PEPS method at T+24

– slight advantage at T+84 (multi-model?)

PEPS over-spread at short-range– regional biases make interpretation difficult

– some evidence for better reliability for extreme events

Experiments with bias-corrected PEPS should clarify results– set up to run over the coming winter


Plans for LAMEPS The Met Office is devising plans for a short-range

ensemble based on a LAM covering the Atlantic and Europe. Aims:

– Risk assessment for rapid cyclogenesis

– Uncertainty of sub-synoptic systems

– assess probability forecasts of precipitation, low cloud and visibility

– LBCs for future storm-scale ensembles


LAMEPS Perturbation Strategy

To be fully effective LAMEPS will need perturbations to:

Initial conditions Model physics parametrizations Lateral boundaries Surface parameters




Initial conditions Model physics parametrizations Surface parameters Lateral boundaries


Initial Condition Perturbations

Options: Singular vectors (as used at ECMWF) Error breeding (Toth and Kalnay, 1993) (as

used at NCEP) Ensemble data assimilation (CMC,

Houtekamer et al, 1996) Ensemble Kalman Filter (Bishop et al, 2001) Multi-analysis (INM)


Initial Condition Perturbations Singular

vectors Error breeding Ensemble

data assimilation

Ensemble Kalman Filter

Multi-analysis

Maximise ensemble growth over early forecast range (48h at ECMWF)

Possibility of combining SVs optimised at 6h, 12 and 18h (Hollingsworth, personal communication)

Some evidence that SVs only provide reliable probabilities for severe weather events well after the optimisation period


Early Warnings of Severe Weather from EPS

Verification of severe weather warnings based on the EPS

– Discrimination of events is best at 4 days (ROC)

– Better discrimination is independent of calibration

– Reliability is best at day 4 and useless at days 1-2

4 days

1 day 2 days

3 days


Early Warnings -Brier Skill Scores

Brier Skill also tends to increase after day 2.

Heavy Rain Severe Gales




data assimilation


Multi-analysis

Relatively simple to implement Identifies modes growing

rapidly at analysis time– may provide a more random

sampling in the early forecast

But… bred vectors are not

orthogonal– tend to converge

– not worth running more than 5-8 cycles




data assimilation


Multi-analysis

Multiple data assimilation cycles with perturbed observations

– computationally expensive

Accounts for model errors Monte-Carlo method

– random sampling, so should provide reliable probabilities

In practice did not perform very well at CMC

– insufficient spread to scale with forecast errors




data assimilation


Multi-analysis

Various configurations exist Theoretically optimal

– not tested in full NWP models

– difficulties with some obs types

– computationally expensive

Ensemble Transform Kalman Filter (Bishop et al, 2001) may provide the best system in the long-term




data assimilation


Multi-analysis

Relatively cheap and simple– reliability may be a problem

Accounts for model errors No attempt to identify

rapidly growing modes Monte-Carlo method

– random sampling, so should provide reliable probabilities

PEPS results suggest:– over-spreading

– need for bias corrections




data assimilation


Multi-analysis

Initially we will use Error Breeding

Later we hope to develop EnKF






Model Physics Perturbations

Again many options… main priorities: Convection Cloud/microphysics

– impact on radiation

Surface roughness


Model Physics PerturbationsApproaches: Multi-model

– effective – opportunity for effective collaboration

Multi-scheme– eg. Kain-Fritsch or Betts-Miller convection

Perturbed tendency– as used at ECMWF

Stochastic physics schemes– conceptually and theoretically elegant– research required - role for universities






Surface Parameters

Surface Roughness– fixed but uncertain - perturb between members

– variable over sea

– impact through windspeed, heat and moisture fluxes

Soil moisture, SST, snow cover etc– analysed

– could be perturbed randomly






Lateral Boundary Conditions

Large domain designed to allow uncertainties to grow within the domain, but...– By T+72 significant uncertainty may emanate

from beyond the western boundary

– Error breeding will grow modes over the previous 24h, so important even for 48h forecasts


Lateral Boundary ConditionsOptions: ECMWF Ensemble Random perturbations Global model breeding at low resolution


Lateral Boundary Conditions ECMWF Random Global

breeding

Readily available– especially if use member-state

time on ECMWF computers

But… Possible balance problems

using LBCs from different model

Each new EPS run has new perturbations - no continuity with the LAM bred modes

– likely generate noise



breeding

Simple to apply Usual problem of random

perturbations - not focussing on the growing modes



breeding

Avoids problems of others:– identifies growing modes

– continuity from run to run

But… Expensive, unless run at low

resolution– grid-length for LBCs should

not be more than 4-5 times longer


Lateral Boundary ConditionsOptions: ECMWF Ensemble Random perturbations Global model breeding at low resolution

No decision has been taken


Outline of LAMEPS Plans Ensemble based on European Mesoscale

– 20km grid-length initially– Minimum 10 members– Run to T+48, possibly to T+72 later

Error breeding - possibly EnKF later Multi-schemes for convection

– research into stochastic physics Perturbed Surface Roughness Perturbed LBCs

– ECMWF EPS or low-resolution Global breeding


Collaboration Opportunity

Dispersed multi-model ensemble Relatively simple approach to model

errors Share computing demands Share system maintenance demands Option to run multiple components at

ECMWF on member-states’ time


Planned Time-Scales Start work April 2003 Year 1 (incl. Relocation):

– Error breeding system– Convection perturbations– First test run

Year 2 (2004-2005): – Version 1 of full perturbation system– System set up for real-time running

Year 3 (2005-2006): – Verification report on real-time performance

New Met Office HQ, Exeter


Questions?

Documents

Poor Man's EPS experiments and LAMEPS plans at the Met Office