Upload
paley
View
53
Download
0
Tags:
Embed Size (px)
DESCRIPTION
www.ec.gc.ca [email protected]. Verification of multi-model ensemble forecasts using the TIGGE dataset . Laurence J. Wilson Environment Canada Anna Ghelli ECMWF. With thanks to Marcel Vallée. Outline. Introduction – TIGGE goals and verification - PowerPoint PPT Presentation
Citation preview
Verification of multi-model ensemble
forecasts using the TIGGE dataset
Laurence J. WilsonEnvironment Canada Anna Ghelli
ECMWF
With thanks to Marcel Vallée
April 22, 2023
Outline• Introduction – TIGGE goals and
verification• Status of verification of TIGGE
ensembles– Standard methods– Spatial methods
• Precipitation verification project: plan and early results
• Summary• SEE the extended abstract also
April 22, 2023
Verification and the goals of TIGGE
• Goals:– Enhance collaborative research– Enable evolution towards GIFS– Develop ensemble combination methods; bias
removal• Essential question: If we are going to move
towards a GIFS, then we must demonstrate that the benefits of combined ensembles are worth the effort with respect to single-center ensembles. OR: Do we get a “better” pdf by merging ensembles?
• Verification – Relevant, user-oriented
April 22, 2023
Status of Verification of TIGGE ensembles
• Mostly model-oriented verification so far– Upper air data– Against analyses– Standard scoring and case studies
• Studies on the TIGGE website– Park et al, 2008
• First study involving several months of data• Found modest improvement with combined ensembles, greatest
benefits in tropics and lower atmosphere• “Advantage” of using one’s own analysis as truth
– Pappenberger et al.• Case study of flooding event in Romania• User-oriented, Q-Q plots, RPS and RMSE main scores used• Multimodel ensemble has best average properties, ECMWF next.
April 22, 2023
Studies using TIGGE data (cont’d)
• Johnson and Swinbank, 2008– Study of calibration/combination methods– Used only 3 ensembles– Mslp and 2m temperature, but from analyses– Multimodel ensemble improves on individual ensembles, but
not by much in general. More at 2m than 500mb• Matsueda 2008
– Comparison of 5 combined ensembles vs ECMWF alone– RMSE skill and RPSS with ECMWF as standard forecast– Multimodel eps outperforms ECMWF at medium and longer
ranges.
April 22, 2023
Status of TIGGE – related verification
• Current efforts – Verification of surface variables?– This conference ---?– Studies using spatial methods
• Ebert – application of CRA technique to ensemble forecasts. So far, only ECMWF.
• Application of Wilks minimum spanning tree or T. Gneiting’s multi-dimensional rank histogram for TC centers. (idea stage)
– Precipitation verification project:
April 22, 2023
Precipitation verification project
• Goal: to verify global 24h precipitation forecasts from all the ensembles in the TIGGE archive and combinations
• One region at a time, using highest density observations
• Canada and Europe so far• Methodology
– Cherubini et al upscaling, verify only where data available– Single station, nearest gridpoint where data is sparser– Kernel density fitting following Peel and Wilson to look at
extremes of distributions.
April 22, 2023
Precipitation verification project : methodology - Europe
• Upscaling:– 1x1 gridboxes, limit of model resolution– Average obs over grid boxes, at least 9 stns
per grid box (Europe data)– Verify only where enough data– Matches obs and model resolution locally– Answers questions about the quality of the
forecasts within the capabilities of the model– Most likely users are modelers.
April 22, 2023
European Verification
-Upscaled observations according to Cherubini et al (2002)
-OBS from gauges in Spain, Portugal, France, Italy, Switzerland, Netherlands, Romania, Czech Republic, Croatia, Austria, Denmark, UK, Ireland, Finland and Slovenia
-At least 9 stns needed per grid box to estimate average
-24h precip totals, thresholds 1,3,5,10,15,20,25,30 mm
-one year (oct 07 to oct 08
April 22, 2023
Reliability – Summer 08 – Europe – 42h
April 22, 2023
Reliability – Summer 08- Europe 114 h
April 22, 2023
Reliability – Winter 07-08 – Europe – 114h
April 22, 2023
ROC – Summer 08 – Europe – 42h
April 22, 2023
ROC – Summer 08 – Europe – 114 h
April 22, 2023
Precipitation verification project: methodology - Canada
• Single station verification– Canadian verification over
20 widely-spaced stations, only one station per gridbox; comparison of nearest gridpoint fcst to obs
– Pointwise verification, does not (we cannot) upscale properly because don’t have the data density necessary.
– Valid nevertheless as absolute verification of model predictions
April 22, 2023
Results – Canada – ROC curves – 24h
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
ROC Curve
False Alarm Rate
Hit
Rat
e
00.10.2
0.30.40.5
0.60.7
0.8
0.9
1
msc 0.825 ( 0.838 )
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
ROC Curve
False Alarm Rate
Hit
Rat
e
00.10.20.30.40.50.60.70.80.9
1
ecmwf 0.785 ( 0.842 )
April 22, 2023
Results – Canada – ROC Curves – 144h
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
ROC Curve
False Alarm Rate
Hit
Rat
e
00.10.20.3
0.40.5
0.6
0.7
0.8
0.9
1
msc 0.664 ( 0.66 )
0.0 0.2 0.4 0.6 0.8 1.00.
00.
20.
40.
60.
81.
0
ROC Curve
False Alarm Rate
Hit
Rat
e
00.10.20.3
0.40.5
0.6
0.7
0.8
0.9
1
ecmwf 0.703 ( 0.701 )
April 22, 2023
RMSE of pcpn probability – Canada – Oct 07 to Oct 08 – 20 stns
BOM in blue (darker blue); ECMWF in red; UKMET in greenCMC in gray; NCEP in cyan (lighter blue)
2.0 mm 10 mm
April 22, 2023
Verification of TIGGE forecasts with respect to surface observations – next steps
• Combined ensemble verification• Other regions – Southern Africa should be
next. – Non-GTS data is available• Evaluation of extreme events – kernel density
fitting to ensembles.• Other high-density observation datasets such
as SHEF in the US• Other variables: TC tracks and related surface
weather• Use of spatial verification methods• THEN maybe we will know the answer to the
TIGGE question.
April 22, 2023
www.ec.gc.ca
April 22, 2023
Issues for TIGGE verification• Use of analyses as truth – advantage of one’s own
model. Alternatives:– Each own analysis– Analyses as ensemble (weighted or not)– Random selection from all analyses – Use “best” analysis; eliminate the related model from
comparison– Average analysis (may have different statistical characteristics)– Model-independent analysis (restricted to data – rich areas, but
that is where verification might be most important for most users
• Problem goes away for verification against observations (as long as they are not qc’d with respect to any model)
April 22, 2023
Park et al study – impact of analysis used as truth in verification
April 22, 2023
Issues for TIGGE Verification (cont’d)
• Bias adjustment/calibration– Reason: to eliminate “artifical” spread in
combined ensemble arising from systematic differences in component models
– First (mean) and second (spread) moments– Several studies have/are being undertaken– Results on benefits not conclusive so far
• Due to too small sample for bias estimation?– Alternative: Rather than correcting bias,
eliminate inter-ensemble component of bias and spread variation.
April 22, 2023
ROC – Winter 07-08 – Europe – 42h
April 22, 2023
ROC – Winter 07-08 – Europe – 114h
April 22, 2023
Reliability – Winter 07-08 – Europe – 42h
April 22, 2023
Results – Canada – Brier Skill, Resolution and Reliability
Brier Skill and components - POP~6400 cases, 20 stns
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
1 2 3 4 5 6 7 8 9 10
Forecast day
BSS
, REL
and
RES
BSS - MSC
BSS - ECMWF
REL - MSC
REL - ECMWF
RES - MSC
RES - ECMWF