Upload
jordan-berry
View
214
Download
0
Embed Size (px)
DESCRIPTION
Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Problems: a) Sample size!, b) Wait a long time (and funding agents are impatient)
Citation preview
Huug van den Dool and Suranjana Saha
Prediction Skill and Predictability
in CFS
Definitions Prediction Skill and Predictability
Opinion: Literature fuzzies up ‘predictability’ vs ‘prediction skill’
Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Problems: a) Sample size! , b) Wait a long time(and funding agents are impatient)
Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Definition 2: Evaluation of skill of hindcasts; hard, not impossible.Problems: a) Sample size, b) ‘honesty’ of hindcasts
Definition 1: Evaluation of skill of real time prediction; the old-fashioned way.
Definition 2: Evaluation of skill of hindcasts; hard, not impossible.
Definition 3: Predictability of the 1st kind (~ sensitivity due to uncertainty in initial conditions)
Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Sample size!
Definition 2: Evaluation of skill of hindcasts; hard, not impossible
Definition 3: Predictability of the 1st kind (~ sensitivity due to uncertainty in initial conditions)
Definition 4: Predictability of the 2nd kind due to variations in external boundary conditions (AMIP; Potential Predictability; Reproducibility; Madden’s approach)
Predictability (theoretical/intrinsic) is a ceiling for actual prediction skill.
Any other ‘kinds’ of predictability?
CFS forecast:X (space, lead, member ,year)
• Space is 2.5oX2.5o (Z500) or 1oX2o (SST/mask), or 1.875 by Gaussian (Soilw, T2m, Precip)
• Basic data used is monthly mean• Lead = 0, 8 in units of months; member = 1, 15• Year = 1981 – 2003 (increases annually)• Example: ‘Initial’ Month is August (= lead 0); • Note IC is Jul 11/21/Aug 1 for SST, and Jul 09-13/ 19-23
/ Jul 30-Aug3 for atmosphere and soil. • ‘Member’ 16 is ensemble average• ‘Member’ 17 is matching observed field• X = ( Z500, SST, Soilw, T2m, Precip)
ASPECTS
• Prediction skill (member i vs member 17)• Predictability (member i vs member j)• Monthly mean• Seasonal mean• Ensemble average• Predictability of 1st kind only.
Two types of climatology plus complications
• Xclim_mdl (space, lead) is average over years and (14 or 15) members, depending.
• Xclim_verif (space, lead) is ave over (same) years for either member 17, or member i, i=15.
• Anomaly = X minus Xclim, whichever is relevant• Systematic error (SE) is automatically corrected
by the above• CV of the SE correction (exclude from Xclim the
member and the year to be verified). Not trivial.
Prediction Skill
Monthly
0
0.1
0.2
0.3
0.4
0.5
a s o n d j f m a
Anom.Corr vs LeadZ500 NH
0
0.1
0.2
0.3
0.4
0.5
a s o n d j f m a
monthly
monthly ens ave
Anom.Corr vs LeadZ500 NH
0
0.1
0.2
0.3
0.4
0.5
a s o n d j f m a
monthly
monthly ens ave
seasonal ens ave
Anom.Corr vs LeadZ500 NH
0
0.1
0.2
0.3
0.4
0.5
a s o n d j f m a
monthly
monthly ens ave
seasonal ens ave
prdctblty seasonal ens ave
Anom.Corr vs LeadZ500 NH
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a s o n d j f m a
skill monthly
skill monthly ens ave
skill seasonal ens ave
prdctblty seasonal ens ave
Anom.Corr vs LeadSST TR (20S-20N;0-360)
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a s o n d j f m a
skill monthly
skill monthly ens ave
skill seasonal ens ave
prdctblty seasonal ens ave
Anom.Corr vs LeadSST Nino34 (5S-5N;170W-120W)
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a s o n d j f m a
skill monthly
skill monthly ens ave
skill seasonal ens ave
prdctblty seasonal ens ave
Anom.Corr vs LeadSST NorthAtl (30-60N;80W-0W)
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
a s o n d j f m a
skill monthly
skill monthly ens ave
skill seasonal ens ave
prdctblty seasonal ens ave
Anom.Corr vs LeadSST Tropical Atl (20S-20N)
Seasonal Prediction Skill Z500 NHCFS 1981-2003
0
0.1
0.2
0.3
0.4
0.5
djf
jfm fma
mam am
j
mjj jja jas
aso
son
ond
ndj
djf
jfm fma
mam am
j
mjj
corr
elat
ion
Jan Mar May Jul Sep Nov
Seasonal Predictability Z500 NHCFS 1981-2003
0
0.1
0.2
0.3
0.4
0.5
djf
jfm fma
mam am
j
mjj jja jas
aso
son
ond
ndj
djf
jfm fma
mam am
j
mjj
corr
elat
ion
Jan Mar May Jul Sep Nov
0
0.1
0.2
0.3
0.4
0.5
0.6
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Prediction Skill T2m (NH-landCFS 1981-2003
0
0.1
0.2
0.3
0.4
0.5
0.6
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Predictability T2m (NH-land)CFS 1981-2003
0
0.1
0.2
0.3
0.4
0.5
0.6
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Prediction Skill Prc (NH-landCFS 1981-2003
0
0.1
0.2
0.3
0.4
0.5
0.6
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Predictability Prc (NH-landCFS 1981-2003
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Predictability (NH-landSoilw CFS 1981-2003
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
djf fma amj jja aso ond djf fma amj
Jan Mar May Jul Sep Nov
Seasonal Prediction Skill (NH-landSoilw CFS 1981-2003
-60-50-40-30-20-10
010
corr
elat
ion(
%)
0 1 2 3 4 5 6 7 8 9 10111213target month (jan=1)
NCEP model observed ens average
W, T correlation at lag 1 mo
Conclusions (monthly data)
• CFS data is a goldmine.• CFS has enough (?) data for forecast evaluation
(and diagnostics)• Member i vs member j unifies predictability of 1st
and 2nd kind in CFS output • CFS has some prediction skill. In order of skill:
SST, {tropical variables}, soilw,T2m, Precip• CFS has some more predictability (as defined), but
ceiling is ‘low’ in mid-latitudes.• Seasonality (no surprise)
To do:
• Identify interdecadal skill source (if any)• Identify soil moisture skill source (are models still
too strong on local effects? How about non-local effects)
• Daily data for the finer temporal scales in skill/predictability.
• Why do models like CFS have predictability in so few d.o.f. (and is that really all there is)
• Further ideas about ‘new’ predictability notions
A case for the importance of knowing the effective number of degrees of freedom (edof) in which we have forecast skill.
Considerations:-) physical models have one clear strength: they can execute the non-linear terms-) a model needs at least 3 degrees of freedom to be non-linear (Lorenz, 1960)-) a non-linear model with nominally a zillion degrees of freedom, but skill in only <= 3 dof is functionally linear in terms of the skill of its forecasts - and, to its detriment, the non-linear terms add random numbers to the tendencies of the modes with predictability.
==> Therefore: Physical models need to have skill in, effectively, > 3 dof before they can be expected to take advantage of non-linearity. (In a forecast setting). ( Note: not any 3 degrees of freedom will do.)
0
20
40
60
80
100co
rrel
atio
n
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30forecast lead (days)
AC(Z) TC(spave(Z))TC(area Z-Below)
SH Z500
95
97.5
100
corr
elat
ion
0 1 2 3forecast lead (days)
AC(Z) TC(spave(Z))TC(area Z-Below)
SH Z500
‘Lingering memory’Cai+Van den Dool(2005); Schemm et al calibration data set,(CFS daily data set will be used also).