IX General COSMO meeting Athens,18-21 September 2007

A.Montani; The COSMO-LEPS system: 5-year milestoneCOSMO meeting, Athens, 18-21 September 2007

The COSMO-LEPS system:

getting close to the 5-year milestone

Andrea Montani, Chiara Marsigli and Tiziana Paccagnella

ARPA-SIM

Hydrometeorological service of Emilia-Romagna, Italy

IX General COSMO meetingAthens,18-21 September 2007


Outline

• Motivations• Methodology of COSMO-LEPS• Verification results:

– SYNOP on the GTS

• Future plans


COSMO-LEPS (developed at ARPA-SIM)

• What is it?It is a Limited-area Ensemble Prediction System

(LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).

• Why?Because the horizontal resolution of global-model

ensemble systems is limited by computer time constraints and does not allow a detailed description of mesoscale and orographic-related processes.

The forecast of heavy precipitation events can still be inaccurate (in terms of both locations and intensity) after the short range.


COSMO-LEPS project

combine the advantages of global-model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of intense and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)

generation of COSMO-LEPS in order to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of the so-

called “severe weather events”.


Dim 2

Initial conditions Dim 1 Dim 2

Possible evolution scenarios

Dim 1 Initial conditions

ensemble size reduction

Cluster members chosen as representative members (RMs)

LAM integrations driven byRMs

LAM scenario

LAM scenario

LAM scenario

COSMO-LEPS methodologyCOSMO-LEPS methodology


The COSMO-LEPS suite @ ECMWFNovember 2002 – May 2004

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

oldest EPSoldest EPS

1212

middle EPSmiddle EPS

oldest EPSoldest EPS

youngest EPSyoungest EPS

clustering clustering periodperiod

0000

1212

Cluster Analysis and RM identificationCluster Analysis and RM identification

4 variables4 variables

Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa

2 2 time time stepssteps


European European areaarea

Complete Complete LinkageLinkage

5 Representative 5 Representative MembersMembers

Driving the 5Driving the 5COSMO-model COSMO-model integrationsintegrations

COSMO-LEPS

Integration Domain

COSMO-LEPS

clustering area


The COSMO-LEPS suite @ ECMWFJune 2004 – January 2006

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

middle EPSmiddle EPS

youngest EPSyoungest EPS


0000

1212



Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa





COSMO-LEPS

Integration Domain

10 10 Representative Representative

MembersMembersdriving the 10driving the 10COSMO-model COSMO-model integrationsintegrations

employing either employing either Tiedtke or Kain-Tiedtke or Kain-Fristch scheme Fristch scheme

randomly randomly choosenchoosen

COSMO-LEPS

clustering area

• Suite running in real time at ECMWF managed by ARPA-SIM;

• Δx ~ 10 km• Fc length: 120h


The new COSMO-LEPS suite @ ECMWFsince February 2006

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

older EPSolder EPS

younger EPSyounger EPS


0000

1212



Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa





COSMO-LEPS

Integration Domain

1616 Representative Representative Members driving the Members driving the

1616 COSMO-model COSMO-model integrations integrations

(weighted according (weighted according to the cluster to the cluster populations)populations)

employing either employing either Tiedtke or Kain-Tiedtke or Kain-

Fristch convection Fristch convection scheme (randomly scheme (randomly

choosen)choosen)

COSMO-LEPS

clustering area

• suite running as a ”time-critical application” managed by ARPA-SIM;

• Δx ~ 10 km; 40 ML;• COSM0-LM 3.20 since Nov06;• fc length: 132h;• Computer time (4.3 million

BU for 2007) provided by the COSMO partners which are ECMWF member states.


Present methodology


Operational set-up

Core products:16 perturbed LM runs (ICs and 3-hourly BCs

from 16 EPS members) to generate probabilistic output (start at 12UTC; t = 132h);

Additional products: 1 deterministic run (ICs and 3-hourly BCs from the

high-resolution deterministic ECMWF forecast) to assess the relative merits between deterministic and probabilistic approach (start at 12UTC; t = 132h);

1 “hindcast” run (ICs and 3-hourly BCs from ECMWF analyses) to “downscale” ECMWF information (start at 00UTC; t = 36h).


Dissemination

probabilistic products deterministic products (individual COSMO-LEPS runs) derived probability products (EM, ES) meteograms over station points

products delivered at about 1UTC to the COSMO weather services, to Hungary (case studies) and to the MAP D-PHASE and COPS communities (field campaign).


– SYNOP on the GTS (COSMO-LEPS only);

Objective verification of COSMO-LEPS

Verification package includes the traditional probabilistic scores:

•Brier Skill Score (Wilks, 1995)•ROC area (Mason and Graham, 1999)•Cost-loss Curve (Richardson, 2000)•Percentage of Outliers (Buizza, 1997)•Ranked Probability Skill Scores (Wilks, 1995)


– SYNOP on the GTS (COSMO-LEPS only);


Main features:

variable: 12h cumulated precip (18-06, 06-18

UTC);

period: from Dec 2002 to Aug 2007;

region: 43-50N, 2-18E (MAP D-PHASEPHASE

area);

method: nearest grid point; no-weighted fcst;

obs: synop reports (about 470 stations x

day);

fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h;

thresholds: 1, 5, 10, 15, 25, 50 mm/12h;

system: COSMO-LEPS

both (57) monthly and (19) seasonal scores are computed.

work is in progress for verification over the full domain (1500

stations)


Time series of Brier Skill ScoreBSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems

if BSS > 0. BS measures the mean squared difference between forecast and observation in probability

space. Equivalent to MSE for deterministic forecast.BSS improvement of

performance detectable for all thresholds along the years;

still problems with high thresholds, but good trend in 2007.

fc step: 30-42h

Jun04: 5m 10m

Feb06: 10m16m; 32ML 40 ML


Time series of ROC areaArea under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.

ROC

Jun04: 5m 10m

Feb06: 10m 16m; 32ML 40 ML

fc step: 30-42h

the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges.

poor performance of the system in Spring and Summer 2006 (both particularly dry), despite upgrades.

fc step: 78-90h

Jun04: 5m 10m

Feb06: 10m 16m; 32ML 40 ML


Outliers: time series + seasonal scoresHow many times the analysis is out of the forecast interval spanned by the ensemble

members. … the lower the better … Performance of the system assessed as time series and for 5 different Summers (JJA).

OUTL Evident seasonal

cycle (more outliers in winter), but overall reduction of outliers in the years.

Jun04: 5m 10m Feb06: 10m 16m; 32ML 40 ML

Reduction of outliers from one Summer to the other, related to the increase of ensemble size (more evident for the 5 to 10 increase).

Need to take into account the different statistics for each season (JJA 2003 less rainy than the others).


Semi-diurnal cycle in COSMO-LEPS scores

BSS score … the higher the better … Performance of the system assessed for 5 different Summers (JJA).

BSS Evident 12-hour cycle in

BSS scores (the same holds for RPSS, while less evident for ROC area scores).

Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).

The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range.

The bad performance in Summer 2006 is confirmed.


Score dependence on the domain size (1)

Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM) and the full domain (~ 1500 stations; fulldom):

different statistics of the verification samples; up to now, performance of the system over the 2 domains assessed only for the

last 6 months (March-August 2007). difficult to draw general conclusions


Score dependence on the domain size (2)

RPSS

RPSS score… the higher the better… (and positive).

ROC area… the higher the better… (and above 0.6).Smoother transitions from month to month in “fulldom” scores.

Slightly better performance of COSMO-LEPS over the MAPDOM, but the signal varies from month to month.

Higher predictability with orographic forcing?

Need to check individual regions and/or to stratify for type of stations.

OUTL

ROC

Outliers percentage … the lower the better.


Main results

• COSMO-LEPS system runs on a daily basis since November 2002 (6 “failures” in almost 5 years of activity) and it has become a “member-state time-critical application” at ECMWF ( ECMWF operators involved in the suite monitoring).

• COSMO-LEPS products used in EC Projects (e.g. PREVIEW) , field campaigns (e.g. COPS, MAP D-PHASE) and met-ops rooms across COSMO community.

• Nevertheless, positive trends can be identified:• increase in ROC area scores and reduction in outliers percentages;• positive impact of increasing the population from 5 to 10 members (June

2004);• although some deficiency in the skill of the system were identified after the

system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!), scores are encouraging throughout 2007.

Time series scores cannot easily disentangle improvements related to COSMO-LEPS itself from those due to better boundaries by

ECMWF EPS.

2 more features:

• marked semi-diurnal cycle in COSMO-LEPS scores (better skill for “night-time” forecasts);

• better scores over the Alpine area rather than over the full domain (to be confirmed).


• Verification vs synop: assess performance over the full domain and

consider other scores (Rank histograms, Reliability diagrams,…).

• Implement “weighted” derived probability products:

– weighted ensemble mean;

– weighted ensemble standard deviation.

• Introduce more model perturbations (see COSMO-SREPS approach).

• Verification will be further developed → new variables verified.

Future plans

• Implement EFI index in the operational suite?

• Implement “TIGGE-LAM compliant” COSMO-LEPS which can run

on any domain, on any boundary conditions (relocatable

COSMO-LEPS).

• Develop “hybrid” clustering (considering both ECMWF EPS and

UKMO MOGREPS) ---> work with GRIB2 format!


Thank you !


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

tp > 1mm/24h

tp > 5mm/24h

Average values (boxes 0.5 x 0.5)MAM06

As regards AVERAGE precipitation above these two threshols, the 3 systems have similar performance.


ENSEMBLE SIZE REDUCTIONIMPACT EVALUATED ON CASE STUDIES (1)


2003082512 Friuli

(+72-+96)

20 mm 100 mm

5 RMs

10 RMs

All 51


ENSEMBLE SIZE REDUCTIONIMPACT EVALUATED ON CASE STUDIES (2)

Observed precipitation between 15-11-2002 12UTC and 16-11-2002 12 UTC

Piedmont case


2002111212 Piedmont

20 mm 150 mm

5 RMs

10 RMs

All 51


COSMO-LEPS

Real time products


Operational COSMO-LEPS ~ Operational EPS

“Friuli case”S.E. 1535 RMs

The youngestEPS


>20mm/24h >50mm/24h

Operational COSMO-LEPS ~ Operational EPS : Friuli case

probability maps – fc. range +96

COSMOLEPS

EPS 51

members


old suite new suite

BSS outliers

EVALUATION OF S.E. SIZE (either 51, or 102, or 153) & ENSEMBLE SIZE REDUCTION (either 5 or 10 RMs)

Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly with respect of using just 1 EPS but leading only to a small decrease of the skill.

Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkable.The impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).


TEST OF NEW CLUSTERING VARIABLES AND/OR DIFFERENT CLUSTERING INTERVALS

Consider a fixed configuration in terms of ensemble size (10 RMs selected out of 2 EPS sets, 2eps-10rm) and the properties of the “reduced” (10-member) global ensemble in 4 different cases:

OPE: the 10 members are selected like in the operational set-up (clustering variables: z,u,v,q; clustering levels: 500, 700, 850 hPa; clustering times: fc+96h, fc+120h);

D2: like OPE, but clustering times: fc+24h, fc+48h;D3: like OPE, but clustering times: fc+48h, fc+72h;D4: like OPE, but clustering times: fc+72h, fc+96h.BS

S

Brier Skill Score: OPE has slightly better scores at all verification ranges (less evident for ROC area .. not shown);

Outliers percentage: results heavily depends on the verification range.

outliers


COSMO-LEPS derived probability products

The generation of the Ensemble Mean (EM) and Ensemble Standard deviation (ES) was implemented and tested for COSMO-LEPS runs in terms of:

Z700 - T850 – MSLP - T2M - UV10M - TP

from fc+0h to fc+132h every 3 hours

Dissemination will start at the end of the month.EM and ES are calculated without considering the weights relative to each integration.


Why Limited Area Ensemble Prediction?

• Global Ensemble Prediction Systems

– have become extremely important tools to tackle the problem of predictions beyond day 2

– are usually run at a coarser resolution with respect to deterministic global predictions → skill in forecasting intense and localised events is currently still limited.


Why Limited Area Ensemble Prediction?(2)

As regards high resolution deterministic forecast in the short range, where limited-area models play the major role, a “satisfactory” QPF is still one of the major challenges. The same can be said for other local parameters.

This is due, among other reasons, to the inherently low degree of predictability typical of severe and localised events.

Probabilistic/Ensemble approach is so required also for the short range at higher resolution


From Global EPS to LAM EPS

• In the Limited-area ensemble systems, tailored for the short range, perturbations must be already “active” during the first hours of integration

• The characteristic of the LAM ensemble are strongly dependent by the lateral boundaries forcing.

• Due to the “regional” application of these Limited Area Ensembles, methodologies can be different in different geographycal regions.

A pratical consideration:Global EPS ~ Big Centres

Limited Area EPS ~ (also) Relatively Small Centres


In the last period the verification package is being developed keeping into account two measure of precipitation:

the cumulative volume of water deployed over a specific region

the rainfall peaks which occur within this region

OBJECTIVE VERIFICATION OF COSMO-LEPS

COSMO observations


CLEPS EPS

Verification grid

OBS MASK


Archiving of C0SMO-LEPS

From July 2005, products are archived on MARS at ECMWFDeterministic run (fc+0h to fc+132h every 3h).

Ensemble Prediction System:16 perturbed forecasts (fc+0h to fc+132h every 3h):

PLEV (500, 700, 850 hPa): Z, RH, T.

SURF: albedo, LCC, MCC, TCC, SW radiation flux, CAPE, hzerocl, snowlmt, mslp, T_2m, Td_2m, TMAX_2m, TMIN_2m, U_10m, V_10m, UVMAX_10m, large-scale rain, convective rain, large-scale snow, TP.

Forecast probability (various intervals and thresholds):SURF: CAPE, hzerocl, TMAX_2m, TMIN_2m, UVMAX_10m, TP, snowfall, showalter index.

Clustering information (population, clustering variable used, …).Derived products (fc+0h to fc+132h every 3h):

ens-mean and ens-spread: Z700, T850, MSLP, T2M, UV10M, TP.


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

tp > 1mm/24h

tp > 5mm/24h

Average values boxes 0.5x0.5 deg3 sis


COSMO-LEPS

16-MEMBER EPS

tp > 1mm/24h

NOCC=610

NOCC=1195

tp > 5mm/24h

NOCC=2671

tp > 10mm/24h

ave 0.5


Maximum values boxes 0.5x0.5 deg

tp > 1mm/24h

tp > 5mm/24h

tp > 10mm/24h

COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

3 sis


COSMO-LEPS

16-MEMBER EPS

tp > 20mm/24hNOCC=227

Average values boxes 0.5x0.5 deg


Precipitation: average on 1.5 x 1.5 boxes

tp > 10mm/24h tp > 20mm/24h

COSMO-LEPS

5-MEMBER EPS

51-MEMBER EPSSON 2003

ROC area

As regards AVERAGE precipitation above these two threshols, EPS wins.

Worsening due to the ensemble reductionPositive impact of LM downscaling


maxima on 1.5 x 1.5 boxes

tp > 20mm/24hCOSMO-LEPS

5-MEMBER EPS

51-MEMBER EPS

tp > 50mm/24h

SON 2003

ROC area

COSMO-LEPS is more skilful than EPS in forecasting correctly high precipitation values over a rather large area.

Number of occurrences: 600 (20 mm threshold) and 150 (50 mm).


Time series of ROC area for COSMO-LEPS (2)

Area under the curve in the FAR vs HIT diagram. Valuable forecast systems have ROC area values > 0.6

ROC

the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges.

poor performance of the system in Spring and Summer 2006, despite system upgrades.

Jun04: 5m 10m

Feb06: 10m 16m; 32ML 40 ML

fc step: 78-90h


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

Average values (boxes 0.5 x 0.5 deg)

MAM06


Time series of TEST OF NEW CLUSTERING VARIABLES AND/OR DIFFERENT CLUSTERING INTERVALS

Consider a fixed configuration in terms of ensemble size (10 RMs selected out of 2 EPS sets, 2eps-10rm) and the properties of the “reduced” (10-member) global ensemble in 4 different cases:

OPE: the 10 members are selected like in the operational set-up (clustering variables: z,u,v,q; clustering levels: 500, 700, 850 hPa; clustering times: fc+96h, fc+120h);

D2: like OPE, but clustering times: fc+24h, fc+48h;D3: like OPE, but clustering times: fc+48h, fc+72h;D4: like OPE, but clustering times: fc+72h, fc+96h.BS

S

Brier Skill Score: OPE has slightly better scores at all verification ranges (less evident for ROC area .. not shown);

Outliers percentage: results heavily depends on the verification range.

outliers


Main results

• COSMO-LEPS system runs on a daily basis since November 2002 and it has become a “member-state time-critical application” at ECMWF ( partial involvement of ECMWF operators in the suite management).

• Time series scores identify positive trends in the performance of the system:• increase in ROC area scores and reduction in outliers percentages;• positive impact of increasing the population from 5 to 10 members (June

2004);• some deficiency in the skill of the system identified after the system

upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!) need of further investigation.

• High-resolution verification shows better scores of COSMO-LEPS with respect to EPS in forecasting precipitation maxima within boxes.• No clear impact of the weighting procedure as regards precipitation, BUT COSMO-LEPS ensemble mean for 2m temperature has better skill (lower standard deviation to observations) with weighting the ensemble members according to cluster populations compared to the mean with unweighted members.


Downscaling

Global Ensemble

LAMNesting

GCM members

LAM members

Total Downscaling

(Brute-Force Approach)

Global Ensemble

LAMNesting

GCM members

LAM members

EnsembleSize

Reduction

Representative members

COSMO

LEPS

Approach


COSMO-LEPS forecast for 72h precipitation

by AW (Meteoswiss)


Probability precipitation > 250mm/72h

C. Frei, MeteoSwiss

by AW (Meteoswiss)


Standard deviation: 2m temperature 12UTC

Forecast range (h) Forecast range (h)

DJF 2004/2005 JJA 2005

STD2 = RMS2 – BIAS2 uw: unweighted / w: weighted according to cluster population

0C 0C


COSMO-LEPS NW

COSMO-LEPS W

tp > 1mm/24h

COSMO-LEPS weighting procedure

maximum values (boxes 0.5x0.5 deg)


Outliers: time series + seasonal scoresHow many times the analysis is out of the forecast interval spanned by the ensemble

members. … the lower the better … Performance of the system assessed for 4 different Autumns.

OUTL

Continuous reduction of outliers from one year to the other, also related to the increase of ensemble size (more evident for the 5 to 10 increase, rather than from 10 to 16).

Need to take into account the different statistics for each season (SON 2006 less rainy than the others).

Jun04: 5m 10m Feb06: 10m 16m; 32ML 40 ML


COSMO-LEPS NW

COSMO-LEPS W

tp > 1mm/24h

COSMO-LEPS weighting

procedure

maximum values (0.5x0.5 boxes)

average values (0.5x0.5 boxes)

tp > 1mm/24h


Synop-Verification: COSMO-LEPS & aLMo

aLM

o do

mai

n

Common verification domain,about 1000 synop stations

aLMo: - deterministic 7 km model (45 levels) of MeteoSwiss for 72h forecasts - same code (LM) as COSMO-LEPS

by AW (Meteoswiss)


Standard deviation: 2m temperature 00UTC

Forecast range (h) Forecast range (h)

DJF 2004/2005 JJA 2005

w

0C 0C

STD2 = RMS2 – BIAS2 uw: unweighted / w: weighted according to cluster population

by AW (Meteoswiss)


Case study: flood event in Switzerland in August 2005

Photos:Tages-Anzeiger

Courtesy of Andrè Walser (Meteoswiss)


Synoptic overview: 22 August 2005

Temperature 850 hPa and geopotential 500 hPa:

2º

18º

10º

by AW (Meteoswiss)


Total precipitation over 3 days (20/8 – 23/8)

About 400 stations, precipitation sum locally over 300 mm!

(06 - 06 UTC)

C. Frei, MeteoSwiss

by AW (Meteoswiss)


COSMO-LEPS forecast for 72h precipitation

by AW (Meteoswiss)


COSMO-LEPS vs ECMWF EPS probability of 72h cumulated precipitation

by AW (Meteoswiss)



C. Frei, MeteoSwiss

by AW (Meteoswiss)



C. Frei, MeteoSwiss

by AW (Meteoswiss)


– high-resolution network (COSMO-LEPS vs ECMWF EPS)


Main features:

variable: 24h cumulated precip (06-06

UTC);

period: MAM 2006;

region: Germany, Switzerland, Italy;

method: BOXES (0.5 x 0.5);

obs: COSMO network (2500 stations

x day);

fcst ranges: 18-42h, 42-66h, 66-90h, 90-114h;

thresholds: 1, 5, 10, 20, 50 mm/24h;

systems:

- COSMO-LEPS (16m, 10 km, 40

ML)

- reduced EPS (16m, 50 km, 60 ML)

- full EPS (51m, 50 km, 60 ML)


Verification of the distributions

Station observation

Grid point forecast

The verification has been made in terms of:

• Average value

• Maximum value

• 50th percentile (Median)

• 90th percentile

in a box

two measures of precipitation: the cumulative volume of

water deployed over a specific region;

the rainfall peaks occurring within the same region.


tp > 1mm/24h

tp > 5mm/24h

Maximum values (boxes 0.5 x 0.5)

COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

tp > 10mm/24h

NOCC=3681

NOCC=1592

NOCC=2492

MAM06


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

Maximum values (boxes 0.5 x 0.5)

MAM06

COSMO-LEPS is more skilful than EPS in forecasting correctly high precipitation values over 0.5 x 0.5 boxes.

Documents

IX General COSMO meeting Athens,18-21 September 2007