View
212
Download
0
Category
Tags:
Preview:
Citation preview
Caio A. S. CoelhoDepartment of Meteorology
University of Readingc.a.d.s.coelho@reading.ac.uk
Met Office, Exeter (U.K.), 20 February 2006
PLAN OF TALKCalibration and combination issuesConceptual framework for forecastingForecast Assimilation:
•Example 1: Nino-3.4 index forecasts•Example 2: Equatorial Pacific SST forecasts•Example 3: S. American rainfall forecasts•Example 4: Regional rainfall downscaling
EUROBRISA project
•
Forecast calibration and combination: Bayesian assimilation of
seasonal climate predictions
Thanks to: David B. Stephenson, Magdalena Balmaseda, Francisco J. Doblas-Reyes and Sergio Pezzulli
This talk is based on the following work: Coelho C.A.S. 2005: “Forecast Calibration and Combination: Bayesian Assimilation of Seasonal ClimatePredictions”. PhD Thesis. University of Reading. 178 pp.
Coelho C.A.S., D. B. Stephenson, M. Balmaseda, F. J. Doblas-Reyes and G. J. van Oldenborgh, 2005: Towards an integrated seasonal forecasting system for South America. ECMWF Technical Memorandum No. 461, 26pp. Also in press in the J. Climate.
Coelho C.A.S., D. B. Stephenson, F. J. Doblas-Reyes, M. Balmaseda, A. Guetter and G. J. vanOldenborgh, 2006: A Bayesian approach for multi-model downscaling: Seasonal forecasting of regionalrainfall and river flows in South America. Meteorological Applications, 13, 1-10.
Stephenson, D. B., Coelho, C. A. S., Doblas-Reyes, F.J. and Balmaseda, M., 2005: “Forecast Assimilation: A Unified Framework for the Combination of Multi-Model Weather and Climate Predictions.” Tellus A, Vol. 57, 253-264.
Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2004: “Forecast Calibration and Combination: A Simple Bayesian Approach for ENSO”. Journal of Climate. Vol. 17, No. 7, 1504-1516.
Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2003: “Skill of Coupled Model Seasonal Forecasts: A Bayesian Assessment of ECMWF ENSO Forecasts”. ECMWF Technical Memorandum No. 426, 16pp. Available from: http://www.met.rdg.ac.uk/~swr01cac
Calibration and combination issues
• Why do forecasts need it?• Which are the best ways to calibrate?• How to get good probability estimates?• Who should do it?
Calibration
Combination• Why combine forecasts?• Should model predictions be weighted or selected?• How best to combine?• Who should do it?
Conceptual framework
)y(p
)x(p)x|y(p)y|x(p
i
iiiii
Data Assimilation “Forecast Assimilation”
)x(p
)y(p)y|x(p)x|y(p
f
fffff
Multi-model ensemble approach
DEMETER DEMETER Development of a European Multi-Model Ensemble
System forSeasonal to Interannual Prediction
Solution: Multi-model Ensemble
Errors: Model formulationInitial conditions
http://www.ecmwf.int/research/demeter
DEMETER Multi-model ensemble system
7 coupled global circulation models
Hindcast period: 1980-2001 (1959-2001)
9 member ensembles
ERA-40 initial conditions
SST and wind perturbations
4 start dates per year
(Feb, May, Aug and Nov)
6 month hindcasts
Model Country
ECMWF International
LODYC France
CNRM France
CERFACS France
INGV Italy
MPI Germany
UKMO U.K.
.
.
..
Examples of application
•
• 0-d: Niño-3.4 index • 1-d: Equatorial Pacific SST• 2-d: South American rainfall
Example 1: Empirical Niño-3.4 forecasts
Well-calibrated: Most observations in the 95% prediction interval (P.I.)
95% P.I.valueJulyY
valueDecemberY
),Y(N~Y|Y
5t
t
2t05t1o5tt
ECMWF coupled model ensemble forecasts
Observations not within the 95% prediction interval! Coupled model forecasts need calibration
m=9DEMETER: 5-month
lead
2X
2ttt
2ttt sˆ;Xˆ);,(N~X
Prior:
Univariate X and Y
),(N~Y 2t0t0t
)V,Y(N~Y|X tttt
'
2X
t m
m
m
sV
),(N~X|Y 2tttt
t
t
2
2t0
t02
t
t
t
2
2t0
2t
X
V
V
11
)X(p
)Y(p)Y|X(p)X|Y(p
t
ttttt
Posterior:
Likelihood:
Bayes’ theorem:
Comparison of the forecasts
Empirical Coupled
Combined SUMMARY
Combined forecasts:• are better calibrated than coupled• have less spread than empirical• match obs better than either
Blue dots = observationsRed dots = mean forecastGrey shade = 95% prediction interval
Mean Absolute Error (MAE) defined as:
The Brier score (BS) is a simple quadratic score for probability forecasts of binary events(e.g. whether SST anomaly < 0). It is defined as:
Some verification statisticsn
k kk 1
1MAE | x y |
n
n2
k kk 1
1BS (p o )
n
Forecast MAE
(C)
Brier
score
Spread
(C)
Climatology 1.16 0.25 1.19
Empirical 0.53 0.05 0.61
Coupled 0.57 0.18 0.33
Combined 0.31 0.04 0.32
Combined forecasts have smallest MAE, BS, and spread
)C,Y(N~Y b
1TT
111T
obba
)SGCG(CGL
C)LGI()CGSG(D
)]YY(GX[LYY
)S],YY[G(N~Y|X o
Prior:
Likelihood:
Posterior:
1YYXYSSG
YGXGYo T
YYXX GGSSS
)D,Y(N~X|Y a
Multivariate X and Y: More than one Normal variable
qq:D
qn:Y
pn:X
qq:C q1:Yb
pp:S qn:Ya
Matrices
Example 2: Equatorial Pacific SST
Forecast Brier Score
Climatol p=0.5 0.25
Multi-model 0.19
FA 58-01 0.17
)0YPr(p tt
SST anomalies: Y (°C)Forecast probabilities: p
DEMETER: 7 coupled models; 6-month lead
Y 0Y
Forecast assimilation reduces (i.e. improves) the Brier score in the eastern and western equatorial Pacific
1BS0)op(n
1BS
n
1k
2kk
Brier Score as a function of longitude
Brier Score=0.25 for p=0.5 climatology
Brier Score<0.25 more skilful than climatology
Brier Score decomposition
1BS0)op(n
1BS
n
1k
2kk
)o1(o)oo(Nn
1)op(N
n
1BS
l
1i
2ii
l
1i
2iii
iNk
ki
i1i oN
1)p|o(po
n
1kko
n
1o
reliability resolution uncertainty
Forecast assimilation improves reliability in the western Pacific
Reliability as a function of longitude
Resolution as a function of longitude
Forecast assimilation improves resolution in the eastern Pacific
Why South America?
El Niño (DJF)
La Niña (DJF)
Source: Climate Prediction Center (http://www.cpc.ncep.noaa.gov)
Seasonal climate potentially predictableDEMETER
Multi-model
Correlation of ensemble meanDJF rainfall forecasts withPREC/L observations
Why South American rainfall?
Agriculture
Electricity: More than 90% produced by hydropower stations
e.g. Itaipu (Brazil/Paraguay):• World largest hydropower plant• Installed power: 12600 MW • 18 generation units (700 MW each)• ~25% electricity consumed in Brazil• ~95% electricity consumed in Paraguay
Example 3: S. American rainfall anomaly composites
Obs Multi-modelForecastAssimilation
(mm/day)
DEMETER: 3 coupled models
(ECMWF, CNRM, UKMO)
1-month lead
Start: Nov DJF
ENSO composites: 1959-2001
• 16 El Nino years
• 13 La Nina years
ACC=0.51
ACC=0.28
ACC=0.97
ACC=0.82
ACC=1.00
ACC=1.00
ACC=Anomaly Correlation CoefficientSpatial correlation of map with obs map
DJF rainfall anomalies for 1975/76 and 1982/83Obs Multi-model Forecast
Assimilation
(mm/day)
ACC=-0.09
ACC=0.32
ACC=0.59
ACC=0.56
La Nina1975/76
El Nino1982/83
DJF rainfall anomalies for 1991/92 and 1998/99Obs Multi-model Forecast
Assimilation
(mm/day)
ACC=0.04
ACC=0.08
ACC=0.32
ACC=0.38
Brier Skill Score for S. American rainfall
Forecast assimilation improves the Brier Skill Score (BSS) in the tropics
limcBS
BS1BSS
)0YPr(p tt
Reliability component of the BSS
Forecast assimilation improves reliability over many regions
limc
reliabreliab BS
BSBSS
Resolution component of the BSS
Forecast assimilation improves resolution in the tropics
limc
resolresol BS
BSBSS
oY | Z ~ N(M[Z Z ],T)1
YZ ZZM S S
oMZ Y MZ 1 T
YY YZ ZZ YZT S S S S
Empirical model for South American rainfall
Y : n q
Z : n p
T : q q
M : q p
Matrices
Z: ASO SSTY: DJF rainfall
Empirical Multi-model Integrated
Correlation maps: DJF rainfall anomalies
Comparable level of determinist skillBetter skill in tropical and southeastern South America
Mean Anomaly Correlation Coefficient
Most skill in ENSO years and forecast assimilation can improve skill
Multi-modelIntegrated
Empirical
limcBS
BS1BSS )0YPr(p tt
ENS
Forecast assimilation improved Brier Skill Score (BSS) in the tropics
Brier Skill Score for S. American rainfall Empirical Multi-model Integrated
limc
reliabreliab BS
BSBSS
Forecast assimilation improved reliability in many regions
Reliability component of the BSS Empirical Multi-model Integrated
limc
resolresol BS
BSBSS
Forecast assimilation improved resolution in the tropics
Resolution component of the BSSEmpirical Multi-model Integrated
Example 4: regional rainfall downscaling
Multi-model ensemble
3 DEMETER coupled models
ECMWF, CNRM, UKMO
3-month lead
Start: Aug NDJ
Period: 1959-2001
Forecast Correlation Brier Score
Multi-model 0.57 0.22
FA 0.74 0.17
South box: NDJ rainfall anomaly Multi-model
Forecast assimilation
Forecast assimilation improves skill substantially
- - - Observation Forecast
Forecast
Forecast Correlation Brier Score
Multi-model 0.62 0.21
FA 0.63 0.18
- - - Observation
Forecast assimilation improved skill marginally
North box: NDJ rainfall anomaly Multi-model
Forecast assimilation
• Forecasts can be improved both by calibration and by combination
• Statistical calibration and combination is analogous to data assimilation and is a fundamental and essential part of the forecasting process (forecast assimilation)
• Forecast assimilation is easy to do for normally distributed predictands such as monthly mean temperatures and seasonal rainfall:• Nino-3 probability forecasts improved – less biased and smaller spread• Equatorial SST forecasts improved in eastern and western Pacific• S. American rainfall forecasts improved in Equatorial and Southern regions
• Combination can improve the resolution of the forecasts (the ability to discriminate between different observed situations) whereas calibration can improve the reliability of the forecasts
• First steps towards an integrated seasonal forecasting system for South America including both empirical and coupled model predictions
• EUROBRISA project will implement this system at CPTEC - Brazil
Summary
The EUROBRISA ProjectLead Investigator: Caio A.S. Coelho
Key Idea: To improve seasonal forecasts in S. America:a region where there is seasonal forecast skill and useful value.
Aims• Strengthen collaboration and promote exchange of expertise and information between European and S. American seasonal forecasters
• Produce improved well-calibrated real-time probabilistic seasonal forecasts for South America
• Develop real-time forecast products for non-profitable governmental use (e.g. reservoir management, hydropower production, and agriculture)
EUROBRISA was approved by ECMWF council in June 2005
http://www.met.rdg.ac.uk/~swr01cac/EUROBRISA
Institutions Country Partners
CPTEC Brazil Coelho, Cavalcanti, Silva Dias, Pezzi
ECMWF EU Anderson, Balmaseda, Doblas-Reyes, Stockdale
INMET Brazil Moura, Silveira
Met Office UK Graham, Davey, Colman
Météo France France Déqué
SIMEPAR Brazil Guetter
Uni. of Reading UK Stephenson
Uni. of Sao Paulo Brazil Ambrizzi, Silva Dias
CIIFEN Ecuador Camacho, Santos
Direct and inverse regression
y
Regression of obs on forecasts Regression of forecasts on obs
More natural to model uncertainty in forecasts for a given observation(ensemble spread of dots) than to model uncertainty in observationsfor a given ensemble forecast. so we model the likelihood on right ratherthan the more common forecast calibration (MOS) approach on the left.
),X28.168.6(N~X|Y 2X|Y ),Y73.065.6(N~Y|X 2
Y|X
Recommended