Upload
nicholas-mcnamara
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
16.6.2005ECMWF User Meeting / [email protected]
Pertti NurmiJuha Kilpinen
Annakaisa Sarkanen( Finnish Meteorological Institute )
Probabilistic Forecasts of Near-Gale Force Winds in the Baltic Applying ECMWF, EPS and Other Methods
ECMWF Forecast Products User Meeting15 – 17 June 2005
16.6.2005ECMWF User Meeting / [email protected]
A study with 2 frameworks:
i. Develop warning criteria / Guidance methods to forecast probability of near-gale force winds in the Baltic Joint Scandinavian research undertaking e.g. Finland and Sweden issue near-gale & storm force wind
warnings for same areas using different criteria => homogenise !
ii. Evaluation of ECMWF products• Deterministic and probabilistic forecasts• Two (maybe three) calibration methods• Here, only ECMWF data applied Later, HIRLAM, too• Here, 6 Finnish coastal stations Later, c. 15-20 stations
from Sweden, Denmark, Norway• Goal: Common Scandinavian operational practice (?)
Introduction:
16.6.2005ECMWF User Meeting / [email protected]
• ECMWF MARS
• u & v components at 10 m => wind speed at 10m
• Forecast lead times: +12 hr to +144 hr
• Data retrieval: 0.5 * 0.5 degree resolution
• Operational, Control, EPS data (interpolated to 0.5o * 0.5o)
• Nearest grid point used
• Forecasts / observations valid: 00, 06, 12, 18 utc
• Observations: 10 minute mean wind speed
• Data coverage: 1/10/04 – 30/4/05 212 days
Data:
16.6.2005ECMWF User Meeting / [email protected]
We may have problems:
• with height of instrumentation ?
• with observing site surroundings and obstacles ?
– with the coast ?
– with nearby islands ?
– with barriers ?
– with installations ?
• with low-level stability ?
NE
16.6.2005ECMWF User Meeting / [email protected]
02_873 - Hailuoto
02_910 - Valassaaret
02_980 - Nyhamn
02_979 - Bogskär
02_981 - Utö
02_987 - Kalbådagrund
Observing stations( 6 out of 39 )
16.6.2005ECMWF User Meeting / [email protected]
(m)
55
50
45
40
35
30
25
20
15
10
5
Heights of the instrumentation ( in red, the 6 out of 39 )
16.6.2005ECMWF User Meeting / [email protected]
32 m
15,5 m/s
10 m
15
Wind speed dependence:Logarithmic wind profile
14 m/s
979 - Bogskär Unstable
Neutral
Stable
threshold
16.6.2005ECMWF User Meeting / [email protected]
-20 -10 0 10 20ERROR
0
30
60
90
Coun
t
0.0
0.1
0.2
0.3
0.4
Proportion per Bar
FITTED DISTRIBUTION
Methods for producing probabilistic forecasts:
Deterministic forecasts:
• Error distribution of original sample (212 cases)
• Approximation of the error distribution with a Gaussian fit (, ):”sample error” method
1. EPS (51 members): Probability of wind speed > 14 m/s
2. Kalman filtering– Various approaches No details given here
3. Deterministic forecasts, adjusted by “a posteriori” estimate of the observed error distribution of the dependent sample Probability distribution of near-gale
– Gives an estimate of the upper limit of the probabilistic predictability.
16.6.2005ECMWF User Meeting / [email protected]
Methods for producing probabilistic forecasts:4. Deterministic forecasts, adjusted with a Gaussian
distribution fitted to model forecasted stability (temperature forecasts at 2 adjacent model levels)
Probability distribution of near-gale, “stability” method- Scheme used at SMHI (H. Hultberg)
5. “Neighbourhood” method- Both spatial (right)
and temporal “neighbours”- c. 25-75 “members”- Applicable primarily for
hi-res models ?
16.6.2005ECMWF User Meeting / [email protected]
• Traditionally, calibration of the EPS is done by re-labeling the probabilities using the information of the reliability diagram (large sample of past forecasts and observations is needed)
• Here, Kalman filtering is used to calibrate the EPS mean (as well as operational and control forecasts). Then each EPS member is transformed with the same relationship (state vector).– This will calibrate the “mean” of the distribution,
hopefully also the “spread”.
• Kalman filtering is also used in the traditional way to correct the deterministic forecasts and then to estimate the probabilities using the observed error distribution.
Calibration of EPS forecasts:
16.6.2005ECMWF User Meeting / [email protected]
Sample climatologic characteristics
Sample mean and variance of wind speed (n=212)
0
10
20
30
40
50
60
70
80
90
100
873_Hailuoto 910_Valassaaret 979_Bogskär 980_Nyhamn 981_Utö 987_Kalbåda
Obs_variance (Unit: (m/s)**2) Obs_mean (Unit: m/s)
46 m 22 m 32 m 25 m 31 m 32 m
16.6.2005ECMWF User Meeting / [email protected]
Sample climatologic characteristics, ref. ECMWF
Sample mean at 12 utc (n=212)
0
2
4
6
8
10
873_Hailuoto 910_Valassaaret 979_Bogskär 980_Nyhamn 981_Utö 987_Kalbåda
Obs_mean (Unit: m/s) T511_mean (+24h) EPS_mean (+24h)
46 m 22 m 32 m 25 m 31 m 32 m
16.6.2005ECMWF User Meeting / [email protected]
Sample variance at 12 utc (n=212)
0
20
40
60
80
100
873_Hailuoto 910_Valassaaret 979_Bogskär 980_Nyhamn 981_Utö 987_Kalbåda
Obs_variance (Unit: (m/s)**2) T511_variance (+24h) EPS_variance (+24h)
46 m 22 m 32 m 25 m 31 m 32 m
Sample climatologic characteristics, ref. ECMWF
16.6.2005ECMWF User Meeting / [email protected]
Obesrvations at 12 utc; T511 & Control forecasts, +48h) Station: 873_Hailuoto12 UTC
0
5
10
15
20
1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191 201 211
Observed
T511
Control
Sample climatologic characteristics, ref. ECMWF
16.6.2005ECMWF User Meeting / [email protected]
Deterministic FCs: Bias - RMSE - 981_Utö
-1
-0,8
-0,6
-0,4
-0,2
0
1 2 3 4 5 6
ME_T511
ME_Control
ME_EPS Mean
ME_Kalman0
1
2
3
4
5
1 2 3 4 5 6
RMSE_T511
RMSE_Control
RMSE_EPS Mean
RMSE_Kalman
ME (Bias) RMSE
“Ensemble spread”
w.r.t to FC lead time
16.6.2005ECMWF User Meeting / [email protected]
Deterministic FCs: Bias - RMSE - 987_Kalbåda
-2
-1,5
-1
-0,5
0
0,5
1 2 3 4 5 6
ME_T511
ME_Control
ME_EPS Mean
ME_Kalman
0
1
2
3
4
5
1 2 3 4 5 6
RMSE_T511
RMSE_Control
RMSE_EPS Mean
RMSE_Kalman
RMSEME (Bias)
w.r.t to FC lead time
16.6.2005ECMWF User Meeting / [email protected]
Probabilistic FCs: Brier Skill w.r.t to FC lead time
-0,1
0
0,1
0,2
0,3
0,4
0,5
1 2 3 4 5 6
BSS_Kalman (EPS)
BBS_Kalman (T511)
987_Kalbåda Brier Score:
BS = ( 1/n ) Σ ( p i – o i ) 2
– Common accuracy measure of prob fcs
– o i is binary (0 or 1)
– Analogous to MSE in probability space
– A quadratic scoring rule Sensitive to large forecast errors ! Careful with limited datasets !
– Influenced by climatologic frequency
of the sample Different samples not to be compared
Brier Skill Score:
BSS = [ 1 – BS / BS ref ] *100
Range: 0 to 1Perfect score = 0
Range: - to 100Perfect score = 100
16.6.2005ECMWF User Meeting / [email protected]
Relative Operating Characteristic
• To determine the ability of a forecasting system to discriminate between situations when a signal is present (here, occurrence of gale) from no-signal cases (“noise”)
• To test model performance relative to a specific threshold
• Applicable for probability forecasts and also for categorical deterministic forecasts Allows for their comparison
• Gained popularity in forecast verification in recent years
Probabilistic FCs: ROC
16.6.2005ECMWF User Meeting / [email protected]
• Graphical representation in a square box of the Hit rate (H) (y-axis) against the False Alarm Rate (F) (x-axis) for different potential decision thresholds
• Curve is plotted from a “binned” set of probability forecasts by stepping (or sliding) a decision threshold (e.g. 10% probability intervals) through the forecasts, each probability decision threshold generating a separate 2*2 contingency table
The probability forecast is transformed into a set of categorical “yes/no” forecasts
A set of value pairs of H and F is obtained, forming the curve
• It is desirable that H be high and F be low, i.e. the closer the point is to the upper left-hand corner, the better the forecast
• A perfect forecast system, with only correct forecasts & no false alarms, (regardless of the threshold chosen) has a “curve” that rises from (0,0) (H=F=0) along the y-axis to (0,1) (upper left-hand corner; H=1, F=0) and then straight to (1,1) (H=F=1)
EventEvent observed
forecastYes No Marginal total
Yes a b a + b
No c d c + d
Marginal total a + c b + d a + b + c + d =n
H = a / ( a + c )
F = b / ( b + d )
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
H
F
10%
20%
30%
40%
50%
60%
90%
80%
70%
Probabilistic FCs: ROC Curve
16.6.2005ECMWF User Meeting / [email protected]
EventEvent observed
forecastYes No Marginal total
Yes a b a + b
No c d c + d
Marginal total a + c b + d a + b + c + d =n
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
H
F
10%
20%
30%
40%
50%
60%
90%
80%
70% To learn more about ROC and Signal Detection Theory, check:
http://wise.cgu.edu/
H = a / ( a + c )
F = b / ( b + d )
a+c =1920 b+d =5351
Example
Probability # of Cumulative # of Non- Cumulative Non- H FThreshold Occurences Occurences Occurences Occurencies (%) (%)
a b
0 - 9 43 1920 613 5351 100 10010 - 19 172 1877 1389 4738 98 8920 - 29 283 1705 1183 3349 89 6330 - 39 350 1422 936 2166 74 4040 - 49 323 1072 602 1230 56 2350 - 59 287 749 327 628 39 1260 - 69 169 462 151 301 24 670 - 79 163 293 88 150 15 380 - 89 89 130 40 62 7 190 - 99 41 41 22 22 2 0
( a ) ( b )
Probabilistic FCs: ROC Curve generation
16.6.2005ECMWF User Meeting / [email protected]
• Area under the ROC curve• Decreases from 1 when curve moves downward from the ideal top-left corner
• A useless forecast system is along the diagonal, when H=F and the area is = 0.5;
Such system cannot discriminate between occurrences and non-occurrences of the event
ROCA based skill score:
ROC_SS = 2 * ROCA - 1
• Negative below the diagonal
• At it’s minimum: ROC_SS = - 1, when ROCA = 0
• ROC is applicable for deterministic categorical forecasts– ROC_SS translates into KSS TSS (= H – F )
– Only one single decision threshold - only a single ROC point results
Typically, this is “inside“ the ROC area, i.e. indicating worse quality
• ROC, ROCA and ROC_SS are directly related to a decision-theoretic approach
– Can be related to the economic value of probability forecasts to end users
– Allowing for the assessment of the costs of false alarms
Range: -1 to 1Perfect score = 1
Range: 0 to 1Perfect system = 1
Probabilistic FCs: ROCA Area
16.6.2005ECMWF User Meeting / [email protected]
Probabilistic FCs: ROC curve/area; T + 48 hr
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
F
H
ROC_EPS
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
F
H
ROCA = 0.73 ROCA = 0.85
ROC_Kalman (EPS)
987_Kalbåda
16.6.2005ECMWF User Meeting / [email protected]
Probabilistic FCs: ROC curve/area; T + 24 hr981_Utö
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
F
H
ROCA = 0.96
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
ROCA = 0.88
ROC_”stability” ROC_”neighbour”
16.6.2005ECMWF User Meeting / [email protected]
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
ROC_"sample error"
ROC_EPS
ROC_Kalman (EPS)
ROC_Kalman (T511)
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
ROC_"sample error"
ROC_EPS
ROC_Kalman (EPS)
873_Hailuoto 910_Valassaaret
EPS
Probabilistic FCs: ROC Area w.r.t to FC lead time
16.6.2005ECMWF User Meeting / [email protected]
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
ROC_"sample error"
ROC_EPS
ROC_Kalman (EPS)
981_Utö 987_Kalbåda
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
ROC_"sample error"
ROC_EPS
ROC_Kalman (EPS)
ROC_Kalman (T511)
Probabilistic FCs: ROC Area w.r.t to FC lead time
16.6.2005ECMWF User Meeting / [email protected]
• So far we’ve just scratched the (sea) surface Need much more experimentation with various methods Different methods for different time/space scales
e.g. very-short vs. medium-range ?
• Biases and other scores depend on station (e.g. observation height)
(Statistical) adjustment of original observations required ? Finland has an operational scheme for this !
• EPS forecasts are slightly under dispersive
• Kalman filtering reduces the biases and produces better prob. forecasts for most stations in terms of the ROC curve/area
• Apply to data from other counterparts
Reach the goal… !!!
Conclusions Future:
16.6.2005ECMWF User Meeting / [email protected]
Forecast Quality Project 2005The Royal Meteorological Society at the behest of the UK
weather forecasting industry and their customers, has
undertaken a project to establish methodologies and metrics
by which the quality of weather forecast services can be
assessed from a user perspective on a basis that is clear,
scientifically well founded, relevant to the users’ needs
and easily applied and understood.
UK forecast user and provider input is NOW needed!
www.rmets.org/survey
Almost finnish(ed), but one advertisement…