Eidgenssisches Departement des Innern EDI Bundesamt fr Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation

Eidgenössisches Departement des Innern EDIBundesamt für Meteorologie und Klimatologie MeteoSchweiz

Weather type dependant fuzzy verification of precipitation

COSMO General Meeting, Offenbach, 07.-11.09.2009

Tanja Weusthoff

2 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff

Fuzzy Verification

• „multi-scale, multi-intensity approach“

• „Fuzzy verification toolbox“ of B. Ebert

• Two methods• Upscaling (UP)• Fraction Skill Score (FSS)

• present output scale dependent

• standard setting: 3h accumulationsCOSMO-2 (2.2km): leadtimes 03-06COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15

fcst obs

incr

easi

ng b

ox s

ize

increasing threshold


… Upscaling (UP)

random

random

hitsalarmsfalsemisseshitshitshitsETS

1. Principle:

Define box around region of interest and calculate the average of observation and forecast data within this box.

3. Equitable Threat Score (ETS)

Rave

totalalarmsfalsehitsmisseshitshits random

) )((

Event if Rave ≥ thresholdNo-Event if Rave < threshold

yes no

yes Hit False Alarm

no Miss Correct negative

observation

fore

cast

2. Contingency Table

Q: Which fraction of observed yes - events was correctly forecast?

(Atger, 2001)


… Fraction Skill Score (FSS) (Roberts and Lean, 2005)

X XX X

X Xx XXX

x

1. Principle:

Define box around region of interest and determine the fraction pj and oj of grid points with rain rates above a given threshold.

3. Skill Score for Probabilities

2. Probabilities

Q: On which spatial scales does the forecast resemble the observation?

N

j

N

jjj op

FBS

1 1

22

N1

1FSS

N

jjj op

1

2)(N1 FBS

FBS worst

no colocation of non-zero fractions

0 < pj < 1 = fraction of fcst grid points > threshold0 < oj < 1 = fraction of obs grid points > threshold


… Fraction Skill Score (FSS) (Roberts and Lean, 2005)

4. Useful Scales

useful scales are marked in bold in the graphics


goodbadCOSMO-7 better COSMO-2 better

COSMO-2 COSMO-7 Difference

- =

- =

Upscaling and Fractions Skill ScoreJun – Nov 2007

Frac

tions

ski

ll sc

ore

Ups

calin

g


COSMO-2 vs. COSMO-7

DIFFERENCES

COSMO-7 better

COSMO-2 better

abs(median) / 0.5(q95-q05)[ 10 , Inf ]

[ 2 , 5 [

[ 5 , 10 [

[ 1 , 2 [

[ 0 , 1 [

Values = Score of COSMO-2

Size of numbers = abs(Median) / 0.5(q95-q05)measure for significance of differences

¦ COSMO-2 – COSMO-7 ¦

q(50%)

q(95%)

q(5%)

5% 95%


SensitivitiesOnly 00 and 12 UTC model runsCOSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15

absolute values of COSMO-2 slightly lower, but still the same pattern of differences

DIFFERENCES

COSMO-7 better

COSMO-2 better


Weather type verification: COSMO-2

4 7 5 5 7 45 31 23 15 34 7# cases

Threshold [mm/3h]

Upscaling

Fractions Skill Score

Window size: 3gp, 6.6 km

Window size: 27gp, 60 km

Window size: 3gp, 6.6 km

Window size: 27gp, 60 km

dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE 45 45 45 / / / / /

N 45 45 / / / / / /

NW 3 3 9 9 27 27 / /

SE 1 1 1 1 9 / / /

S 1 1 1 1 1 9 / /

SW 1 1 3 9 9 45 / /

E 9 15 27 / / / / 45

W 1 1 3 9 15 / / /

F 15 15 27 27 45 / / /

H 27 27 27 45 / / / /

L 3 9 15 27 45 / / /

dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE 15 / / / / / / /

N / / / / / / / /

NW 3 3 5 5 9 15 / /

SE 1 1 1 3 9 / / /

S 1 1 1 3 3 9 / /

SW 3 3 5 9 9 / / /

E 15 / / / / / / /

W 3 3 3 5 9 / / /

F 9 9 15 15 / / / /

H / 15 / / / / / /

L 3 5 9 9 15 / / /

COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km

Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition.

+

-+

-

7

4

23

5

7

45

5

31

15

34

7

# cases

% obs gridpts >= thresh (whole period)

16 14 10 7 5 2 0.5 <0.1 16 14 10 7 5 2 0.4 <0.1


% 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE 7 6 4 2 1 0.3 <0.1 <0.1

N 2 1 0.8 0.4 0.1 <0.1 <0.1 0

NW 13 11 8 5 2 0.7 0.1 <0.1

SE 17 15 12 9 6 2 0.4 <0.1

S 21 19 15 11 8 3 0.7 0.1

SW 25 22 18 13 9 3 1 0.2

E 3 2 1 0.7 0.2 <0.1 <0.1 <0.1

W 18 15 11 8 4 1 0.2 <0.1

F 17 15 11 8 5 2 0.8 0.2

H 2 2 0.8 0.5 0.2 0.1 <0.1 <0.1

L 42 36 27 19 11 3 0.8 0.1

ALL 16 14 10 7 5 2 0.5 <0.1

Fraction of observation gridpoints >= threshold climatology


Frequency of Weather Classes, June – November 2007

47

5 57 7

45

31

23

15

34


Northerly Winds (NE,N) 11 days

COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)

COSMO-2 in northerly wind situations clearly worse than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 better

DIFFERENCES

COSMO-2 (all) better

COSMO-2 (wc) better


Southerly Winds (SE,S) 12 days


COSMO-2 in southerly wind situations clearly better than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 better

DIFFERENCES


COSMO-2 (wc) better


Northwesterly winds (NW) 23 days


COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 betterD

IFFERENCES


COSMO-2 (wc) better


Flat (F) 15 days


DIFFERENCES

COSMO-7 better

COSMO-2 betterD

IFFERENCES


COSMO-2 (wc) better

COSMO-2 in flat pressure situations clearly worse than over whole period.


Summary / Conlusions

• Fuzzy verification of the D-PHASE operations period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales

• part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences

• the results for different weather types show large variations • best results were found for southerly winds and winds

from Northwest and West, • Northeasterly winds as well as Flat pressure situations

lead to worse perfomance of both models


UP with other scoresPOFD = FA/(CN+FA)Probability of false detection (Perfect score: 0)

FAR = FA/(H+FA)False Alarm Ratio (Perfect score: 0)

POD = H/(H+M)Probability of Detection (Perfect score: 1)

BIAS = (H+FA) / (H+M)Frequency Bias (perfect score: 1)

HK =POD – FARTrue Skill Statistik (perfect score: 1)

OR = (H*CN) / (M*FA)Odds Ratio (perfect score: infinity)


dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE

N

NW

SE

S

SW

E

W

F

H

L

dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE

N

NW

SE

S

SW

E

W

F

H

L

COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km

UP „useful scales“?

How to define usefuls scales for ETS?


Fuzzy Verifikation Intensity Scale (IS)

(Casati et al. 2004)• Transformation of Fcst and Obs into binary images on a rain/no-rain basis for the rainfall

rate thresholds. • Difference between forecast and observation = binary error image. • Decomposition into the sum of components at different spatial scales by performing a

two dimensional discrete wavelet decomposition.

Binary forecast Binary observation

Score: Mean squared error (MSE) and MSE skill score (SS) for each spatial scale component of the binary error image.

What is the relative improvement of the forecast over some reference forecast?


Intensity ScaleCOSMO-2 COSMO-7


Intensity Scale

COSMO-2


Outlook

• operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score• season, year• weather-type dependant

• Intensity scale results will further be investigated (new developments of B. Casati)


All Weatherclasses

• Overview over all weatherclasses – differences COSMO-2 minus COSMO-7 and absolute values of COSMO-2, here without bootstrapping


Northerly Winds (NE,N,NW)N NWNE

23 days4 days7 days

DIFFERENCES

COSMO-7 better

COSMO-2 better


Southerly Winds (SE,S,SW)SSE SW

7 days 45 days5 days

DIFFERENCES

COSMO-7 better

COSMO-2 better


East and West (E,W)E W

31 days5 days

DIFFERENCES

COSMO-7 better

COSMO-2 better


Flat, High, Low (F,H,L)F H L

34 days 7 days15 days

DIFFERENCES

COSMO-7 better

COSMO-2 better


Summary

JJA SON DOP

Useful scales of COSMO-2 are the same for all three time periods considered: <= 0.2 mm/3h scales larger or equal 2.2 km (= 1* gridscale ) 0.5 mm/3h scales larger or equal 6.6 km (= 3* gridscale ) 1.0 mm/3h scales larger or equal 19.8 km (= 9* gridscale ) 2.0 mm/3h scales larger or equal 33.0 km (= 15* gridscale ) 5.0 mm/3h scales larger or equal 99.0 km (= 45* gridscale )

Most prominent advantage for COSMO-2! Differences are significant.

COSMO-2 better but on many scales only slightly. Advantages on small scales/all thresholds and large thresholds/large scales. Differences are significant.

COSMO-2 better than COSMO-7 on all scales. Differences are significant.

Most frequent weatherclasses: SW (32 days) and W (21 days).

Most frequent weatherclasses: H (24 days) and NW (17 days).

Most frequent weatherclasses: SW (45 days) and H (34 days).


JJA SON DOP

NE -Relatively low values for UP-FSS: COSMO-2 better for thresholds <= 2 mm/3h, useful scales only for largest window size (99 km)

- COSMO-2 better on nearly all scales, especially for thresholds of 2-5 mm/3h. Large threshold gave no result in UP only small - scale events (averaging damps values out). Apparently, COSMO-2 could simulate those events quite well.

-Low absolute values (FSS < 0.6, UP < 0.3), no useful scales-COSMO-2 clearly better for thresholds <= 2 mm/3h (FSS)- UP: the averaged precipitation was equally well represented in both models. Only 2 mm/3h threshold better captured in COSMO-2.

N Only few days – no clear superiority of one model. Absolute values relatively low. No useful scales.

NW - COSMO-2 performed better than COSMO-7, especiall for large thresholds (> 1 mm/3h, FSS and UP).

-Many days with that weather type-Over most scales rarely different behaviour-Thresholds 5-10 mm/3h better in COSMO-2 as well as spatial scale of 6.6 km

- COSMO-2 performed better than COSMO-7, especiall for large thresholds (>= 5 mm/3h, FSS and UP).

Summary Weatherclasses I


JJA SON DOP

SE -Only few days; similar behaviour in all time periods: COSMO-2 better, especially for thresholds around 1 mm/3h and small spatial scales (FSS) and large spatial scales (UP)-Large absolute values (e.g. FSS up to 0.9, UP up to 0.66 for DOP)

S - Only few days; COSMO-2 better on most scales, most clearly for low thresholds concerning averaging (UP) and large thresholds and small spatial scales regarding probabilities (FSS), in JJA COSMO-2 clearly better also for smallest eindow size (6.6 km) for all thresholds

SW - Clearly better performance of COSMO-2, especially on smaller spatial scales

- Similar performance of both models, but COSMO-2 better for moderate thresholds (1-2 mm/3h)

- Clearly better performance of COSMO-2, especially on smaller spatial scales

Summary Weatherclasses II


JJA SON DOP

E -Only few cases (5 for DOP), COSMO-2 clearly better for low thresholds (<= 1 mm), while for a threshold of 2 mm/3h COSMO-7 better on largest spatial scales-Rather low absolute values

W - COSMO-2 clearly better on all scales, especially for thresholds <= 2 mm and smaller spatial scales

- COSMO-2 slightly better only on small spatial scales

- COSMO-2 better for thresholds <= 2 mm and especially smaller spatial scales

F -Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds.

-Comparable low absolute values, rarely difference in the performance of the two models. -COSMO-2 slightly better for small thresholds

- Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds.

H -very low absolute values (especially in SON), but COSMO-2 clearly better than COSMO-7 for precipitation thresholds <= 2 mm/3h

L - Only few cases (7 for DOP), COSMO-2 better for small spatial scales and thresholds <= 2 mm/3h

Summary Weatherclasses III

Documents

Eidgenssisches Departement des Innern EDI Bundesamt fr Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation