View
27
Download
2
Category
Tags:
Preview:
DESCRIPTION
An evaluation of the probabilistic information in multi-model ensembles. David Unger Huug van den Dool , Malaquias Pena, Peitao Peng, and Suranjana Saha 30 th Annual Climate Prediction and Diagnostics Workshop October 25, 2005. Objective. - PowerPoint PPT Presentation
Citation preview
An evaluation of the probabilistic information in multi-model
ensembles
David Unger
Huug van den Dool , Malaquias Pena, Peitao Peng, and Suranjana Saha
30th Annual Climate Prediction and Diagnostics Workshop
October 25, 2005
Objective
• Produce a Probability Distribution Function (PDF) from the ensembles.
Challenge• Calibration• Account for skill• Retain information from ensembles (Or not if no skill)
Schematic illustration
Temperature
σ
Schematic example
Temperature (F)
σz
Kernel vs. Mean
Regression
ZF F
F
( )_ _ _ _
Z RZ^
Step 1. Standardization
Step 3. Make the forecast
Step 2. Skill Adjustment
F Z CC
^ ^
^ ^
^
σe
σe= σc√ 1-Rm2
______
Analysis of Ensemble Variance
V = Variance
Total = Explained + Unexplained
Variance Variance Variance
σc2 = Ve + Vu
σc2 = σFm
2 + σe
2
σc2 = σFm
2 + (E2 + σz
2)
^
^
Analysis of Ensemble Variance (Continued)
With help of some relationships commonly used in linear regression:
<E2 > = (Rm2 - Ri
2) σc2
σz2 = (1-2Rm
2 + Ri2) σc
2
Rz2 = 2Rm
2 - Ri2 Rz≤ 1.
^
Ensemble Calibration
ZF F
ii
F
( )
_ _ _ _
Step 1. Standardization
Step 3. Skill Adjustment
Zi = Rz Zi ’
Step 2. Ensemble Spread Adjustment
Zi’ = K(Zi - Zm) + Zm
F Z Ci C i
^ ^
^ ^
^
Step 4. Make the Forecast
Schematic example
Temperature (F)
σz
Rz=.97, Rfm=.94, Ri=.90
Rz=.93, Rfm=.87, Rf=.30
Rz=.85, Rfm=.67, Ri=.41
Rz=.62, Rfm=.46, Rf=.20
Weighting
Wgts: 50% Ens. 1, 17%Ens 2, 3, 4
Real time system
• Time series estimates of Statistics.– Exponential filter
FT+1 = (1-α)FT + α fT+1
- Initial guess provided from 1956-1981 CA statistics
Continuous Ranked Probability Score
Some Results
• Nino 3.4 SSTs
Operational system 15 CFS ,12 CA, 1 CCA , 1 MKV
• Demeter Data9 CFS, 12 CA, 1 CCA, 1 MKV
9 UKM, 9 MFR, 9 MPI,
9 ECM, 9 ING, 9 LOD, 9 CER,
Nino 3.4 SSTs
Nino 3.4 SST 5-month lead by initial time 1982-2001
-0.1
0
0.1
0.2
0.3
0.4
0.5
CR
PS
S
Feb May Aug Nov All
Con-Op
Con-86
Ens-86b
Ens86
0
0.1
0.2
0.3
0.4
0.5
0.6
CR
PS
S
1 2 3 4 5
Con-Op
Con-86
Ens-86b
CRPSS – Nino 3.4 SSTs
All Initial times 1982-2001
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
CR
PS
S
1 2 3 4 5
Con-86
Ens-86b
CRPSS – Nino 3.4 SSTs
All Initial times 1990-2001 (Independent)
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
CR
PS
S
c09 c21 c30 c39 c48 c57 c66 c75 c84 c86 t27 b27
Con-86
Ens-86b
CRPSS – Nino 3.4 SSTs 5-month Lead, All Initial times 1990-2001 (Independent data)
Reliability Nino 3.4 SST (1990-2001)
U.S. Temperature and Precipitation Consolidation
• 15 CFS
• 1 CCA
• 1 SMLR
Trends are removed from models
Statistics and distribution are computed
Trend added to end result.
Trend Problem
• Should a skill mask be applied? How much?
- This technique requires a quantitative estimate of the trend.
• Component models sometimes “learn” trends, making bias correction difficult. – Doubles trends.
• Errors in estimating high frequency model forecasts
U.S. T and P consolidation
Skill Mask on Trends No Skill Mask on Trends
.060
.091
.053 .063
.101 .160
.074 .105
.228 .372
.173 .259
.010 .067
.000 .000
Ensm
CRPS RPS
Cons
CRPS and RPS-3 (BNA) Skill Scores: Temperature
.020 .040
.010 -.025
.026 .047
.051 .045
.057 .070
.039 .045
.016 -.007
.052 .029
190 .253
.138 .190
.033 .048
.030 .037
-.001 -.013
.039 .019
.084 .121
.043 .074
.086 .110
.044 .050
High
Moderate
Low
None
Skill
.15
.07
.02
1-Month Lead, All initial times
Reliability U. S. Temperatures(1995-2003)
.018
.018
.017 .018
.005 .002
.005 .002
.078 .059
.076 .059
.055 .064
.055 .064
Ensm
CRPS RPS
Cons
CRPS and RPS-3 (BNA) Skill Scores: Precipitation
.001 .002
.002 .002
.005 .005
.005 .007
-.004 -.009
-.004 -.009
-.006 -.004
-.006 .004
.006 .011
.006 .009
.030 .037
.030 .037
.013 .007
.012 .007
.016 .031
.016 .029
.014 .020
.013 .020
High
Moderate
Low
None
Skill
.10
.05
.01
1-Month Lead, All initial times
Reliability U.S. Precipitation(1995-2003)
Conclusions
• Calibrated ensemble and ensemble means score very closely (by CRPS)
Calibrated ensembles seem to have a slight edge.
• No penalty for including many ensembles (but not much benefit either)
• Considerable penalty for including less skillful ensembles – Weighting is critical.
• Probabilistic predictions are reliable (when looked at in terms of a continuous PDF)
Conclusions (Continued)
• Calibrated ensembles tend to be slightly overconfident
• Trends are a major problem – and are outside the realm of consolidation (but they are critically important for seasonal temperature forecasting).
6-10 day Forecasts (based on Analogs)
Recommended