Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
15/11/2005Juha Kilpinen 1
Comparison of ECMWF and HIRLAM wind forecasts in the Baltic Sea
Juha Kilpinen, AnnaKaisa Sarkanen, Pertti Nurmi and Sigbritt NäsmanFinnish Meteorological Institute (FMI)
• FMI news– New building (one floor for Finnish Marine Research Institute)
– New supercomputer• Meteorological visualisation applications
(EGOWS 2005)
• Use and interpretation of medium and extended range forecast guidance (the title above)
15/11/2005Juha Kilpinen 2
15/11/2005Juha Kilpinen 3
15/11/2005Juha Kilpinen 4
15/11/2005Juha Kilpinen 5
New supercomputer
• SGI Altix 3700 BX2– 304 processors– 304 Gb of memory– 30 times faster than the
previous IBM – HIRLAM and other
applications (ECHAM5)
15/11/2005Juha Kilpinen 6
Meteorological visualisation applications
• Last EGOWS meeting in Exeter– http://www.metoffice.gov.uk/egows2005/programme.html
15/11/2005Juha Kilpinen 7
Comparison of ECMWF and HIRLAM wind forecasts in the Baltic Sea
Juha Kilpinen, AnnaKaisa Sarkanen, Pertti Nurmi and Sigbritt NäsmanFinnish Meteorological Institute (FMI)
• This study is a part of two frameworks– evaluation of ECMWF EPS forecasts in FINLAND– evaluation of warning criteria for forecasting probability of near gale
force wind in the Baltic Sea (Finland and Sweden issue near galewarnings and storm warnings for same areas, but with different criteria)
• Both ECMWF and HIRLAM data is used• Period of data is one winter (~ September 2004 – April 2005) • Observations from 21 coastal stations• Both deterministic and probabilistic forecasts are verified• Different methods for producing probability forecasts from
deterministic data has been tested • Different calibration methods has been tested and verified
15/11/2005Juha Kilpinen 8
Data (ECMWF model data)• ECMWF MARS data
– u10m, v10m->speed10m• Period 1.10.2004-30.4.2005
(Finnish stations)• Period 1.10.2004-31.3.2005
(Swedish stations)• Period 1.11.2004-31.3.2005
(Danish and Norwegian stations)
• forecasts valid at 00, 06, 12 and 18 UTC
• forecast lead time from +12h-+144h
• Operational data and Control data (interpolated to 0.5x 0.5 degree resolution)
• EPS data (interpolated to 0.5x 0.5 degree resolution)
15/11/2005Juha Kilpinen 9
Data (HIRLAM Model data)• HIRLAM RCR
– 10m wind speed & additional parameters
• Grid length about 22 km• forecast lead time from +6h-
+48h• HIRLAM MBE
– 10m wind speed & additional parameters
• Grid length 9 km• forecast lead time from +1h-
+24h
Different grids of HIRLAM (RCR (above) and BME)
0.5 deg
1.0 deg
02981 (Utö)
02987 (Kalbådagrund)
EC grid 0.5x0.5 or 1.0x1.0
RCR
MBE
15/11/2005Juha Kilpinen 10
Different grids of HIRLAM (RCR (above) and BME)
0.5 deg
1.0 deg
02981 (Utö)
02987 (Kalbådagrund)
EC grid 0.5x0.5 or 1.0x1.0
RCR
MBE
15/11/2005Juha Kilpinen 11
Observation stations:01427014480146701482
060810605206041060790617906193
0251702616026440268002584
028730291002980029810297902987
15/11/2005Juha Kilpinen 12
Stations
Hailuoto 02873
Valassaaret 20910
Nyhamn 02980
Utö 02981
Bogskär 02979
Kalbådagrund 02987 31 m
15/11/2005Juha Kilpinen 13
Stations
Hailuoto 02873
Valassaaret 20910
Nyhamn 02980
Utö 02981
Bogskär 02979
Kalbådagrund 02987
32 m
15/11/2005Juha Kilpinen 14
The methods for producing probabilistic forecasts for near gale force winds
• Deterministic data• Method 1a: observed error distribution (dependent sample) is
used to produce probability forecasts from operative deterministic forecasts (“error dressing”)
• Method 1b: as above, but observed error distribution is approximated with Gaussian distribution (µ , σ)
• Method 2: as above but σ is derived from model “stability”(temperature difference of two model levels) with an empirical formula (by Håkan Hultberg, SMHI)
• Method 3: operational forecast is Kalman filtered and σ (a bi-product of Kalman filter) is used to estimate the probability (also Gaussian approximation)
15/11/2005Juha Kilpinen 15
The methods for producing probabilistic forecasts for near gale force winds
• Deterministic data (cont.)Method 4: Uncertainty method (also called Neighbourhood
method by e.g. Susanne Theis)– Spatial (Fig.) and/or temporal – neighbouring grid points– “Unc”
• EPS data (51 members): the probability of near gale wind (speed > 14 m/s or >13.9 m/s)
15/11/2005Juha Kilpinen 16
The methods for producing probabilistic forecasts for near gale force winds
• Calibrated EPS data: the calibration is made by Kalman filtering the EPS mean and then all EPS member are transformed with the same relationship to provide a new 51 member ensemble.
• Deterministic operative data: Kalman filtering of operative wind forecasts and application of Kalman filter residuals to provide the error distribution for probability of near gale. Also other alternatives exits but they have not yet been tested.
15/11/2005Juha Kilpinen 17
Some examples of theoretical pdf’s with different parametersLimit (upper) Probability
of the phenomenon
15/11/2005Juha Kilpinen 18
Examples of individual ensembles (station 02680 +24h (51 members) )
Error distribution of Controlforecasts (sample of 180 cases)and fitted Gaussian distribution
day 49 day 83
15/11/2005Juha Kilpinen 19
Examples of forecast/observation distributions (station 02680 +24h)
Error distribution of EPS meanforecasts (sample of 180 cases)and fitted Gaussian distributionBLUE line corresponds to meanspread of EPS mean
Distribution of EPS mean Distribution of observations
Distribution of error
15/11/2005Juha Kilpinen 20
Calibration of EPS forecasts• Traditionally calibration of ensemble forecasts means that the
probabilities are relabelled using the information in reliability diagram (a large sample of past forecasts and observations are needed)
• In this study Kalman filtering is used to calibrate ensemble mean forecasts (and also operative as well as control). Then every individual ensemble member is transformed with the state vector. This calibrates at least the “mean” of distribution and hopefully also the “spread”.
• Kalman filtering is also used in traditional way to correct the deterministic forecasts and then to estimate the probabilities using observed error distribution
15/11/2005Juha Kilpinen 21
Verification measures• For deterministic forecasts (operative, control and EPS mean,
Kalman filtered operative)
– Mean Error ME– Root Mean Squared Error, RMSE– Standard Deviation of Error, STD
• For probabilistic forecasts (”Error Dressing/classical”, EPS, calibrated EPS (with Kalman filter), Kalman filtering, “Stability” method, “Uncertainty” method)
– (Brier Score), Brier Skill Score, ROC curve, ROC area
• Some comparisons with HIRLAM data (mostly probabilistic scores)
• SPREAD in this content is mean of standard deviation of ensemble mean (51 members)
15/11/2005Juha Kilpinen 22
Results: Deterministic forecastsPooled results from 21 stations (00 UTC, 1 = +12h, …)
lead time (days)
ME
-0,2
0
0,2
0,4
0,6
0,8
1
1 2 3 4 5 6
ME_OPERME_CONTRME_EPSME_KAL
15/11/2005Juha Kilpinen 23
Results: Deterministic forecastsPooled results from 21 stations (00 UTC, 1 = +12h, …)
lead time (days)
RM
SE
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
1 2 3 4 5 6
RMSE_OPERRMSE_CONTRRMSE_EPSRMSE_KALSPREAD
15/11/2005Juha Kilpinen 24
Results: Deterministic forecastsPooled results from 21 stations (00 UTC, 1 = +12h, …)
lead time (days)
STD
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
1 2 3 4 5 6
STD_OPERSTD_CONTRSTD_EPSSTD_KALSPREAD
15/11/2005Juha Kilpinen 25
Results: Deterministic forecastsPooled results from 21 stations (12 UTC)
lead time (days)
ME
-0,1
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1 2 3 4 5 6
ME_OPERME_CONTRME_EPSME_KAL
15/11/2005Juha Kilpinen 26
Results: Deterministic forecastsPooled results from 21 stations (12 UTC)
lead time (days)
RM
SE
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
1 2 3 4 5 6
RMSE_OPERRMSE_CONTRRMSE_EPSRMSE_KALSPREAD
15/11/2005Juha Kilpinen 27
Results: Deterministic forecastsPooled results from 21 stations (12 UTC)
lead time (days)
STD
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
1 2 3 4 5 6
STD_OPERSTD_CONTRSTD_EPSSTD_KALSPREAD
15/11/2005Juha Kilpinen 28
Relative Operating Characteristic
• To determine the ability of a forecasting system to discriminatebetween situations when a signal is present (here, occurrence of near-gale) from no-signal cases (“noise”)
• To test model performance relative to a specific threshold
• Applicable for probability forecasts and also for categorical deterministic forecasts
Allows for their comparison
Probabilistic FCs: ROC
15/11/2005Juha Kilpinen 29
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Alarm Rate
Hit
Rat
e
ROC AREA 0.934
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Alarm Rate
Hit
Rat
e
ROC AREA 0.895ROCA (x) = 0.93ROCA (y) = 0.89
F=16%
H=78%
H=87%
F
H
F=26%
Forecast method “x”
Forecast method “y”
Example of Decision-Makingusing ROC Verification
15/11/2005Juha Kilpinen 30
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
classsp20041103_24_12_06079_out_1
False Alarm Rate
Hit
Rat
e
0.1
0.2
0.30.40.50.60.70.8
0.9
1
ROC AREA 0.902 ( 0.902 )
”Dressing” method: station 06079 +24h
A ”typical” ROC curve
15/11/2005Juha Kilpinen 31
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
classsp20041003_24_12_02584_out_1
False Alarm Rate
Hit
Rat
e
0.10.20.30.40.5
0.6
0.7
0.8
0.9
1
ROC AREA 0.985 ( 0.977 )
”Dressing” method: station 02584 +24h
An example of a ”perfect” ROC curve
15/11/2005Juha Kilpinen 32
Comparison of methods; station_02981_ +24 hrs
HIRLAM ECMWF
MBE MBE MBE RCRDr Stb Unc Unc Dr EPS Kal
ROC A .91 .84 .90 .85 .98 .88 .95BSS .47 .12 .43 .34 .57 .44 .54
No. of events: ~ 25 /130 No. of events: ~ 30 /210
Dr - "Dressing" of dependent sampleStb - "Stability" method EPS - EPS 51 membersUnc - "Uncertainty area" method Kal - Kalman filter EPS
Brier Score: BS = ( 1/n ) Σ ( p i – o i ) 2
Brier Skill Score: BSS = [ 1 – BS / BS ref ]
Range: - οο to 1Perfect score = 1
– MSE in probability space– Sensitive to large forecast errors !– Careful with limited datasets !– Influenced by sample climatology
Different samples not to be compared
15/11/2005Juha Kilpinen 33
ROC Area & BSS w.r.t. to FC lead time; station_02981
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
"Dressing" EPS Kalman
0
0,2
0,4
0,6
0,8
1 2 3 4 5 6
"Dressing" EPS Kalman
ROC A BSS
15/11/2005Juha Kilpinen 34
ROC Area & BSS w.r.t. to FC lead time; station_02987
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6
"Dressing" EPS Kalman
ROC A
-0,2
0
0,2
0,4
0,6
0,8
1 2 3 4 5 6
"Dressing" EPS Kalman
BSS
15/11/2005Juha Kilpinen 35
Results: Probability forecasts
lead time (days)
RO
C a
rea
02987 12 UTC
0,5
0,55
0,6
0,65
0,7
0,75
0,8
0,85
0,9
0,95
1
1 2 3 4 5 6
ROC_DressROC_Dress_GROC_epsROC_eps_kalmROC_kalm_oper
15/11/2005Juha Kilpinen 36
Conclusions
• Inhomogeneous observations a problem for verification• EPS forecasts are slightly under dispersive• The biases depend on station (height of anemometer etc.)• Kalman filtering is able reduce biases and produce better
probability forecasts for most stations in terms of ROC curve ROC area and Brier Skill Score
• “Dressing” of dependent sample: quality level hard to reach
THANK YOU