Developing an Ensemble Prediction System based on COSMO-DE

10th September 2009 Theis, Gebhardt, Buchhold, Paulat, Ohl, Ben Bouallègue, Peralta Project COSMO-DE-EPS

Developing an Ensemble Prediction System

based on COSMO-DE

Project COSMO-DE-EPS

Deutscher Wetterdienst

Susanne Theis, Christoph Gebhardt, Michael Buchhold,Marcus Paulat, Roland Ohl, Zied Ben Bouallègue, Carlos Peralta



Overview

Motivation for Project COSMO-DE-EPS

Tasks within Project

Current Status

where to get information


Motivation for the Project




Numerical Weather Prediction Model COSMO-DE

grid box size: 2,8 km

without parametrization

of deep convection

assimilation of radar data

lead time: 0-21 hours

operational since April 2007

COSMO-EU

GME

COSMO-DE


mm/h

Benefits of COSMO-DE

COSMO-EU (7 km)


COSMO-DE (2,8 km)

captures extremes


mm/h

COSMO-EU (7 km) COSMO-DE (2,8 km)

Benefits of COSMO-DE


looks more realistic


What about Deterministic Predictability?


lead time

Atmosphere is

a chaotic system

(Lorenz,1963)


lead time

typi

cal t

ime

scal

e

typical spacial scale

Scale Diagram

Zyklonen

Cumulo-nimbus

10 km 1000 km

1 Stunde

1 Woche

Predictability

100 m




COSMO-DE (2,8 km)

mm/h


paradigm shift from deterministic to probabilistic prediction

Need for probabilistic approach

so that user really gets benefit


COSMO-DE (2,8 km)

mm/h


Project COSMO-DE-EPS:

Development of a convection-allowing

Ensemble Prediction System

Aims and Tasks



Setting up an Ensemble System

several predictions = „members“ of ensemble

very large investment of computing power

„variations“ inprediction system

ensemble products:

- probability density fct- exceedance probs- quantiles- „spread“- mean- ...




How many ensemble members?

20 members, pre-operational

40 members, operational

When?

2010 pre-operational

2011 operational




implementation of perturbations

post-processing

verification & diagnostics

visualization

early user feedback





post-processing


visualization

early user feedback



Implementation of Perturbations



Implementation of Perturbations

X X

initial conditions boundary conditions model physics


Perturbation of the Model



Perturbation of the Model

Alter parameters in physics parametrizations

entr_sc=0.0003 entrain_sc=0.002

rlam_heat=1. rlam_heat=10.

rlam_heat=1. rlam_heat=0.1

…q_crit…

…tur_len…

11 2233 44

55

requires careful tuning affect forecast, but not long-term quality

11

22

33

55

44




Perturbation of Boundary Conditions




COSMO-EU

GME

COSMO-DE

Reminder: operational chain




COSMO-DE-EPS(DWD, Germany) COSMO-DE-EPS(DWD, Germany)

Part of COSMO-SREPS(ARPA-SIMC, Bologna)Part of COSMO-SREPS(ARPA-SIMC, Bologna)

Part of AEMet Ensemble(AEMet, Spain)Part of AEMet Ensemble(AEMet, Spain)

(Garcia-Moya et al., 2007)(Marsigli et al., 2008)(Gebhardt et al. 2009)







GME IFS UM NCEPCOSMO-DE (2,8km) COSMOCOSMO COSMOCOSMO




long-term Plan: take boundaries from ICON Ensemble




GME IFS UM NCEPCOSMO-DE (2,8km) COSMOCOSMO COSMOCOSMO



Perturbation of Initial Conditions

first experiments:

perturb “nudging” at forecast start

current work:

use differences between control and COSMO-SREPS

plans:

Ensemble Transform Kalman Filter (COSMO project KENDA)



Current Status of Perturbation

X X




Back to the List of Tasks





post-processing


visualization

early user feedback



More Information

• poster at this COSMO General Meeting

• past meetings: download presentationshttp://www.metoffice.gov.uk/conference/srnwp_workshop/presentations/session_3/18_Theis_EPS09.pdfhttp://www.pks.mpg.de/~dswc09/ contribution files

• future meetings: ECAM/EMS, SRNWP/EWGLAM, COSMO User Seminar

• publications:Gebhardt et al (2008): Experimental ensemble forecasts of precipitation based on a convection-resolving model. Atmospheric Science Letters 9, 67-72.

Gebhardt et al (2009): Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Submitted to Atmospheric Research.

• contact: [email protected], [email protected]



Extra Slides



Current Experiments



Current Status

X X




Current Experiments

no perturbationof initial conditions

11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP




11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP

Current Experiments



mm/24h

24h accumulations of precipitation [mm]

2 July 2007, 00 UTC + 24h




11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP

Current Experiments



mm/24h

24h accumulations of precipitation [mm]

2 July 2007, 00 UTC + 24h



Ensemble Diagnostics

- how do perturbations affect the forecast?



Effect of perturbations on precipitation

boundaries:

dominate after a few hours

(not necessarily, also case-dependent)

physics:

dominate during first few hours

(Gebhardt et al., 2009)



Ensemble Dispersion (precipitation)

NormalizedVariance Difference

= variance (BC only) - variance (PHY only)

normalized bysum of variances

(Gebhardt et al., 2009)



Ensemble Dispersion (precipitation)

NormalizedVariance Difference

= variance (BC only) - variance (PHY only)

normalized bysum of variances

for individual days(Gebhardt et al., 2009)



Ensemble Verification

- how good is the Ensemble (at this stage)?



Verification Data Sets

RADAR



11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP

Verification Data Sets (Ensemble Forecasts)

Period:1 July – 16 Sep 2008



11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP





11 22 33 44 55

GMEGME

IFSIFS

UMUM

NCEPNCEP


when all 15 membersavailable(60% of days)




Probabilistic Verification Measures

Traditional:

- Brier Score, Brier Skill Score

- ROC curve

- Reliability Diagram

- Talagrand Diagram



Talagrand Diagram

24h precipitation

Rank

1 3 5 7 9 11 13 15



Talagrand Diagram

24h precipitation

Rank

1 3 5 7 9 11 13 15

observation



Talagrand Diagram

24h precipitation

Rank

1 3 5 7 9 11 13 15

- underdispersive !

- add IC perturbations and statistical PP !

- also look into more spread measures



Future Plan:

Statistical Postprocessing



Statistical Postprocessing (just starting)

Aim:

improve quality of ensemble products

(probabilities, quantiles, mean, entire pdf, extremes,…)

Focus:

Precipitation

1. exceedance probabilities (“30% probability of heavy rain”)

2. entire pdf, extremes?



Statistical Postprocessing (just starting)

Envisaged Methods:

1. Logistic Regression

+ spatio-temporal neighbourhood

+ lagged average ensemble

2. Bayesian Approach

cooperation with P.Friederichs

calibration of probabilities

entire pdf

enhance sample



Reaching the User





Reaching the user (= “serve mankind”)

• make ensemble forecast accessible

• choose a good format and reduce information

• work towards acceptance

• work towards correct (useful) interpretation

• work towards integration into decision making

• increase trust in forecast provider



Visualization in NinJo

- make forecasts accessible

- choose formats



Visualization in NinJo

NinJo = visualization tool for forecasters

Challenge:

- amount of data

- good communication of complex matters

- must fit into an existing system (NinJo)

1 computer scientist, 4 years (Roland Ohl)



ExceedanceProbabilitiesin %

Probability ofRR > 1 mm/h

EPS Product Example:


Quantilesrange of80% probability(= uncertainty)

1h-sums of precipitation [mm]

2m-temperature [°C]

EPS Product Example:

10% probability (= risk):interesting for some users




Further Plans: Ensemble Navigation Window (draft)

also possible tolook at individual members



Early User Feedback

- work towards acceptance

- work towards useful interpretation



Early User Feedback

- Forecasters (DWD)


„Event somewhere in 2,8km-Box“„Event somewhere in 2,8km-Box“


EPS Product Example: Probability Maps

%





%Forecasters:Probabilities are too low!





%Forecasters:Probabilities are too low!

Reason:

Forecasters are used tolarger reference regions

Experiment:

present probabilitieson coarser grid





%

„Event somewhere in 28km-Box“„Event somewhere in 28km-Box“

Forecasters:Probabilities are too low!



Early User Feedback

- Hydrologists



- atmosphere ischaotic system

- limited predictability

- specify uncertainties

- better than giving out wrong forecasts

This is me, talking to some hydrologists







this will increase trust,because I am doingthe right thing

This is me, talking to some hydrologists







Why don‘t you put all effortsinto improving your model?

Please point out the best member in your ensemble!











I do not believe thatthe atmosphere is achaotic system.











I do not trust thisforecast provideranymore











I do not trust thisforecast provideranymore

Educate the user?Yes, and also understand the user.



Research about the Users



Center of Attention:The Model



Center of Attention:The Human Being

(WAS*IS Workshop, 2005)



Literature about perception of ensemble forecasts:

Demeritt, D., Cloke, H., Pappenberger, F., Thielen, J., Bartholmes, J. and M.-H. Ramos (2007): Ensemble predictions and perceptions of risk, uncertainty, and error in flood forecasting. Environmental Hazards 7, 115-127.

Gigerenzer, G., Hertwig, R., van den Broek, E., Fasolo, B. and K.V.Katsikopoulos (2005): „A 30% Chance of Rain Tomorrow“: How does the public understand probabilistic weather forecasts? Risk Analysis 25(3), 623-629.

Joslyn, S., Pak, K., Jones, D., Pyles, J. and E. Hunt (2007): The effect of probabilistic information on threshold forecasts. Weather and Forecasting 22, 804-812.

Morss, R.E., Demuth, J.L. and J.K. Lazo (2008): Communicating Uncertainty in Weather Forecasts: A Survey of the U.S. Public. Weather and Forecasting 23, 974-991.

Morss, R.E., Wilhelmi, V.W., Downton, M.W. and E. Gruntfest (2005): Flood risk, uncertainty, and scientific information for decision making: Lessons from an interdisciplinary project. Bulletin of the American Meteorological Society 86(11), 1593-1601.

National Research Council (2006): Completing the Forecast: Characterizing and Communicating Uncertaintiy for Better Decisions Using Weather and Climate Forecasts. National Academies Press, 124 pp.

Roulston, M.S., Bolton, G.E., Kleit, A.N. and A.L. Sears-Collins (2006): A laboratory study of the benefits of including uncertainty information in weather forecasts. Weather and Forecasting 21, 116-122.

extremely interestin

g for m

e

extremely interestin

g for m

e



Back to the Project COSMO-DE-EPS:

Summary and Challenges



Summary of Project COSMO-DE-EPS

• setting up an ensemble with a complex NWP model

• one of the first convection-permitting ensembles (with operational perspective)

• simple approach first, improve later („just do it“)

• covering all parts of forecast chain (perturbations, products, postprocessing, verification, users)



Challenges (Forecast Provider’s Perspective)

• “describe nature”:

try to be scientifically appropriate probabilistic approach

• “provide forecasts”:

technical requirements: fast, stable, maintainable

• “serve mankind”:

convey benefits to the real world

Documents

Developing an Ensemble Prediction System based on COSMO-DE