Complicated ENSO models do not significantly outperform very simple ENSO models

INTERNATIONAL JOURNAL OF CLIMATOLOGYInt. J. Climatol. 28: 219–233 (2008)Published online 1 June 2007 in Wiley InterScience(www.interscience.wiley.com) DOI: 10.1002/joc.1519

Complicated ENSO models do not significantly outperformvery simple ENSO models

Halmar Halidea† and Peter Riddb*a Australian Institute of Marine Science, PMB3, MC, Townsville, Australia

b School of Mathematical and Physical Sciences, James Cook University, Townsville, Australia

ABSTRACT: An extremely simple univariate statistical model called ‘IndOzy’ was developed to predict El Nino-SouthernOscillation (ENSO) events. The model uses five delayed-time inputs of the Nino 3.4 sea surface temperature anomaly(SSTA) index to predict up to 12 months in advance. The prediction skill of the model was assessed using both short- andlong-term indices and compared with other operational dynamical and statistical models. Using ENSO-CLIPER(climatologyand persistence) as benchmark, only a few statistical models including IndOzy are considered skillful for short-rangeprediction. All models, however, do not differ significantly from the benchmark model at seasonal Lead-3–6. None ofthe models show any skill, even against a no-skill random forecast, for seasonal Lead-7. When using the Nino 3.4 SSTAindex from 1856 to 2005, the ultra simple IndOzy shows a useful prediction up to 4 months lead, and is slightly lessskillful than the best dynamical model LDEO5. That such a simple model such as IndOzy, which can be run in a fewseconds on a standard office computer, can perform comparably with respect to the far more complicated models raisessome philosophical questions about modelling extremely complicated systems such as ENSO. It seems evident that muchof the complexity of many models does little to improve the accuracy of prediction. If larger and more complex models donot perform significantly better than an almost trivially simple model, then perhaps future models that use even larger datasets, and much greater computer power may not lead to significant improvements in both dynamical and statistical models.Investigating why simple models perform so well may help to point the way to improved models. For example, analysingdynamical models by successively stripping away their complexity can focus in on the most important parameters for agood prediction. Copyright 2007 Royal Meteorological Society

KEY WORDS El Nino; climate forecast; statistical model

Received 30 March 2006; Revised 08 February 2007; Accepted 08 February 2007

1. Introduction

There has been considerable research into modellingand predicting the El Nino-Southern Oscillation (ENSO)phenomenon due to the huge economic costs of extremeENSO events. The ability to accurately predict ENSOevents a few months in advance has been shown tobenefit the fisheries, animal husbandry and agriculturalsectors (Adams et al., 1995; Lehodey et al., 1997; Solowet al., 1998; Jochec et al., 2001; Chen et al., 2002;Letson et al., 2005). For example, using an accurateENSO forecast in managing a salmon fishery resultedin a considerable productivity increase (Costello et al.,1998). An increased production of 7–9% was obtainedwhen an ENSO prediction was incorporated into grazingstrategies in the beef industry in Australia (McKeonet al., 2000). Similar benefits were also observed in othersectors such as natural gas purchase (Changnon et al.,2000), hydropower price (Hamlet et al., 2002), insurance

* Correspondence to: Peter Ridd, School of Mathematical and PhysicalSciences, James Cook University, Townsville, 4811 Australia.E-mail: [email protected]† On a post-doc leave from Physics Dept. Hasanuddin Univ., Makassar- Indonesia.

industry (Chichilnisky and Heal, 1998), and disease risk(Bouma et al., 1997).

Not surprisingly, there are many models for predict-ing ENSO (Barnston et al., 1999) and they have greatlyvarying degrees of complexity. These models can bedivided into two categories: statistical models and phys-ical/dynamical models. In statistical predictions, findingan optimal predictor (input) model that gives a better fitto a predictant (output) is the objective regardless of anycausal relationship between them. Belonging to this cat-egory are models such as the canonical correlation anal-ysis (Barnston and Ropelewski, 1992) and the Markov(Xue et al., 2000) models of NCEP, the linear-inversemodel of NOAA-CDC (Penland and Magorian, 1993),the constructed-analog models (van den Dool, 1994), cli-matology persistence model (Knaff and Landsea, 1997),the UBC nonlinear canonical correlation analysis (Hsieh,2001), and the regression model of the Florida State Uni-versity (Clarke and van Gorder, 2001).

Physical/dynamical models apply the governinglaws of fluids and thermodynamics to describe thephenomenon. Some of the models in this category are:the Lamont–Doherty model (Zebiak and Cane, 1987),the hybrid dynamical model of SIO/MPI (Barnett et al.,

Copyright 2007 Royal Meteorological Society

220 H. HALIDE AND P. RIDD

1993), the European Centre for Medium-range WeatherForecast model (Molteni et al., 1996), the NCEP coupledmodel (Ji et al., 1996), the JMA GCM model (Shibataet al., 1999), the KMA-SNU model (Kang and Kug,2000), the BMRC CGCM (Wang et al., 2002) and theAGCM of NISPP/NASA (Bacmeister and Suarez, 2002).

There also exist hybrid models that combine an oceanmodel with a statistical atmosphere. In this case, a windstress is produced from sea surface temperature anomaly(SSTA) data using various statistical techniques rangingfrom CCA/linear regression and EOF analysis to neuralnetwork methods. The resulting wind stress is then usedto drive the ocean model (e.g, Barnett et al., 1993;Balmaseda et al., 1994; Syu et al., 1995; Tang and Hsieh,2002).

The predictive skills of the above models have beenreported by Kerr (1998, 2000, 2002), Latif et al. (1998),Barnston et al. (1999), Landsea and Knaff (2000), andGoddard et al. (2001). These studies show that the skill ofboth types of models remain problematic irrespective ofwhether the model success is gauged relative to a bench-mark model, or for the forecasting of a single ENSOevent. In addition, it is also important to assess predic-tion skill within a period, which includes both ENSO (ElNino/La Nina) and non-ENSO events (Barnston et al.,1996) in order to assess occasions when prediction mightgive false alarms (Chen et al., 2004).

In this study, we have developed a very simplestatistical model called ‘IndOzy’, which was designedto predict Nino 3.4 SSTA. Although there are some newfacets of this model that have not been used in otherENSO models, we do not make any great claims about themodel. Instead, the model was developed to demonstratehow a very simple model can make surprisingly robustpredictions and to raise the question of why the moreelegant statistical and dynamical models do not performeven better than they presently do.

2. Method

2.1. Data

The IndOzy model was developed to predict the futuremonthly Nino 3.4 index using past monthly Nino 3.4 data.No other data is used as input to the model and thusthe model can, in this regard, be considered extremelysimple. The Nino 3.4 index is obtained by averaging thesea surface temperature (SST) in the region 5°N–5°Slatitude and 120°W–170°W longitude, and subtractingthe average of the SST data from 1971 to 2000, on amonth-by-month basis.

Two Nino 3.4 SST data sets of different lengths wereused in the analysis to address the statistical signifi-cance of the prediction skill. These two sets of dataare standard sets that are used commonly in otherstudies of El Nino prediction and are used in thiswork, so that a comparison can be made betweenthe simple model presented in this paper and othermore complicated models. The first data set (Data Set

NOAA) was the monthly Nino 3.4 data, from January1950 to December 2005. This data set was compiledby the Climate Prediction Centre and is available athttp://www.cpc.ncep.noaa.gov/data/indices/sstoi.indices.The monthly observed data was converted into a3 month seasonal grouping such as April-May-June(AMJ) averaged Nino 3.4 index to comply with ENSOpredictions issued from many institutions and col-lected in the International Research Institute website (athttp://iri.columbia.edu/climate/ENSO/currentinfo/arch-ive/index.html). In order to make comparisons of theIndOzy model with other models, the IRI monthly sum-mary of a 3 monthly averaged SST Nino 3.4 indexpredictions of several lead times was used. It is impor-tant to note about lead times used for this evaluation.For a summary released in March 2002, the AMJ 2002and May-June-July (MJJ) 2002 values are referred asthe Lead-1 and Lead-2 predictions, respectively. In thisstudy, the models comparison starts from AMJ 2002 sea-son and ends in September-October-November (SON)2005 season. Although this is only a short data set, pre-dictions of a large suite of models are available and thus,it is convenient to determine the comparative skill ofthe IndOzy model. The physical models are: the Lam-ont model (Zebiak and Cane, 1987), the SIO model(Barnett et al., 1993), the European Centre for Medium-range Weather Forecast model (Molteni et al., 1996), theNCEP coupled model (Ji et al., 1996), and the AGCMof NISPP/NASA (Bacmeister and Suarez, 2002) The sta-tistical model includes the canonical correlation analy-sis (Barnston and Ropelewski, 1992) and the Markov(Xue et al., 2000) models of NCEP, the linear-inversemodel of NOAA-CDC (Penland and Magorian, 1993),the constructed-analog models (van den Dool, 1994), cli-matology and persistence (CLIPER) model (Knaff andLandsea, 1997), and the UBC nonlinear canonical corre-lation analysis (Hsieh, 2001).

The second Nino 3.4 data set (Data Set KAPLAN) wasobtained from extended monthly SSTA Nino 3.4, anal-ysed from the ICOADS collection (Kaplan et al., 1998).This data starts from January 1856 to December 2005 andis available at http://ingrid.ldeo.columbia.edu/SOURCES/.Indices/.nino/.EXTENDED/.NINO34/. IndOzy modelskill using this data is compared to that of the Zebiak–Cane LDEO5 model (Anderson, 2004; Chen et al., 2004).This data set covers the period that is also covered by dataset NOAA and the data are very similar, but not preciselythe same for the overlapping period, 1950–2005.

2.2. IndOzy model description

As mentioned previously, the aim of the paper was todeliberately develop a very simple model for ENSO pre-diction and to compare its performance with more ele-gant and complicated statistical models. IndOzy appliesa method of state-space reconstruction of a dynamicalsystem using delayed coordinates (Packard et al., 1980;Takens, 1981; Sauer, 1994; Abarnabel and Lall, 1996).In our ENSO case, the dynamical system to be recon-structed refers to a value of Nino 3.4 index at some time

Copyright 2007 Royal Meteorological Society Int. J. Climatol. 28: 219–233 (2008)DOI: 10.1002/joc

SIMPLE ENSO MODELS VERSUS COMPLICATED ENSO MODELS 221

t in the future (predictant), while the delayed coordi-nates are the set of predictors (previous values of theindex). In other words, the delayed coordinates are sim-ply the already measured data points that are used tomake a prediction. The number of delay coordinates, D

is related to an attractor dimension, d by Takens the-orem (i.e. D = 2 d + 1). For example, a chaotic timeseries has d = 2, D = 5, i.e. there will be five spacevariables to reconstruct the series (Gautama et al., 2004).Since ENSO phenomenon also exhibits chaotic behaviour(Tziperman et al., 1997; Samelson and Tziperman, 2001),the model is set to have D = 5. In fact, on their attemptto find the best state-space parameters for predictinggeophysical time series including ENSO, Regonda et al.(2005) found that D ranges from 2 to 5 and the numberof delay time in each embedded space ranges from 11to 21. Even though their model was apparently able tocorrectly predict the time evolution of the phenomenonaround the peaks in the years 1982, 1984, 1997, and 1999,its skill still needs to be compared with the other modelsusing appropriate skill measures. Further details on howto implement this dynamical reconstruction principle intothe IndOzy model are described in Appendix A.

2.3. Skill measures

Three skill measures were used to assess the predictionskill of an ENSO model. The two commonly usedmeasures for evaluating ENSO prediction are the Pearsoncorrelation coefficient and the root-mean-squared error(RMSE). In this study, another skill measure commonlyused in weather prediction, the Peirce score, is applied

to assess the ENSO prediction. These score measures aredescribed in Appendix B.

3. Result

3.1. Predictions using data set NOAA (1950–2005)

3.1.1. Out-of-sample prediction

The IndOzy model was first run and tested using Data SetNOAA. The training data set for the model starts fromthe 3 month averaged SSTA index of January-February-March (JFM) 1950 up to SON 1995. The training dataset was used to calculate weights and biases for eachprediction lead. These weights and biases were then usedto produce an out-of-sample prediction for the period1995–2005. Data from 1995 to 2005 was not used totrain the model.

Seasonal out-of-sample ENSO prediction results alongwith observational data are presented in Figure 1. Here,we only plot predictions from seasonal Lead-1, Lead-3, Lead-5, and Lead-7. The model predicts the warmevents, such as the El Nino event 1997–1998, La Ninaevent of 2000–2001 and the predictions decay at longerlead times, as expected. However, the model producesspurious results at longer lead times mostly during normalconditions and the La Nina event of 1999–2000. Thelack of skill for near-normal condition than that ofEl Nino and La Nina conditions is in agreement withprevious findings (van den Dool and Toth, 1991; Masonand Mimmack, 2002; Kirtman, 2003). However, thespurious prediction result at longer leads for the La

Figure 1. Time series of out-of-sample prediction plotted for 120 data points (n = 120). IndOzy model predicts up to seven seasons ahead. Here,we only plot Lead-1 (average of the 1st, 2nd, and 3rd monthly Nino 3.4 SST anomaly), Lead-3 (average of 3rd, 4th and 5th monthly anomaly),Lead-5 (average of 5th, 6th and 7th monthly anomaly) and Lead-7 (average of 7th, 8th and 9th monthly anomaly). This figure is available in

colour online at www.interscience.wiley.com/ijoc



Nina event of 1999–2000 could be due to the inabilityof the model to capture the fast changing dynamicsof the event, i.e. from strong El Nino to strong LaNina. This phenomenon is also experienced by manymodels,which fail during that particular event (Kirtman,2003).

It is interesting to note that the predictions of the ElNino events become cooler with increasing lead times.For example, for the 97/98 event, for a seasonal Lead-1,the prediction is 0.3 °C too cool; however, for seasonalLeads 3, 5, and 7, the predictions are 0.8, 1.5, and 2 °Ctoo cool, respectively. Generally, a prediction might beexpected some times to over predict and sometimes tounder predict; however, for the peak events, IndOzy doesnot do this.

It is apparent that the model predictions do not divergefrom the present value by more than about 1 °C and thusit is unlikely for the model to predict an extreme eventsuch as the peak of an El Nino event from conditionswhen the temperature is low, i.e. a few months beforea major El Nino event. On the other hand, at a timeof seasonal Lead-1 before a large El Nino event peaks,the present temperature is already elevated and thus thesmall deviation of the prediction value from the presentvalue still allows a result similar to the peak values to beachieved. We do not fully understand why the predictionsof the results are always too cool.

The degradation of the model performance withincreasing lead time is most evident in the scatter plotdiagrams (Figure 2). The correlation coefficient, RMSEand Peirce skill score (and its error estimate for both theIndOzy model and the no-skill random prediction) are

plotted in Figure 3. In Figure 3, the IndOzy predictionhas skill only up to Lead-5 since the Peirce score of sea-sonal Lead-6 and Lead-7 predictions overlap with thoseof the no-skill random forecast. We suggest that a predic-tion is considered useful if the Peirce skill score is higherthan that of the no-skill forecast.

We also investigate whether or not the occurrenceand strength of ENSO events affect prediction skill.In order to do that, the out-of-sample forecast isdivided into two parts, each containing 60 points(n = 60). The first part is from October-November-December (OND) 1995 to SON 2000, which includesthe strongest 1997–1998 ENSO event (Anderson, 2004;McPhaden, 2004) while the second consists mostlyof the normal years from OND 2000 to SON 2005.The result using the Peirce skill score is presented inTable I.

Table I shows that the strength of ENSO events affectsthe prediction skill in two respects. First, sections, whenthe events are dominated by strong ENSO, have a higherscore than that of weak and normal periods for eachlead time. Second, the presence of strong ENSO eventsalso extends the skill for a slightly longer lead whilethe absence of such events limits this skill. For instance,the IndOzy model has useful skill up to seasonal Lead-5during the OND 1995–SON 2000 period, whereas duringthe normal years, i.e. during the OND 2000–SON 2005period, useful skill is only up to seasonal Lead-3. It isevident that care must be taken when evaluating modelperformance, as the particular data set used for modeltesting can have a significant effect on the apparent modelperformance.

Figure 2. Scatter plot of IndOzy out-of-sample prediction against observation for seasonal Lead-1, Lead-3, Lead-5, and Lead-7 (with n = 120).This figure is available in colour online at www.interscience.wiley.com/ijoc



Figure 3. IndOzy out-of-sample prediction skill against observation for the seasonal 7 leads (n = 120). Using the Peirce score (right-handdiagram), seasonal Lead-6 and Lead-7 has no skill since each of them has an overlap with the no-skill prediction. This figure is available in


Table I. Peirce skill score measures for different sections of the out-of-sample prediction.

Predictionhorizon

Peirce skill score

OND 1995–SON 2000 (n = 60) OND 2000–SON 2005 (n = 60)

IndOzy No-skill IndOzy No-skill

Seasonal Lead-1 0.84 ± 0.1 0.01 ± 0.15 0.75 ± 0.09 0.00 ± 0.13Seasonal Lead-2 0.67 ± 0.12 0.01 ± 0.15 0.50 ± 0.12 0.00 ± 0.13Seasonal Lead-3 0.54 ± 0.13 0.01 ± 0.15 0.32 ± 0.12 0.00 ± 0.13Seasonal Lead-4 0.53 ± 0.13 0.01 ± 0.14 0.18 ± 0.13 0.00 ± 0.13Seasonal Lead-5 0.29 ± 0.14 0.01 ± 0.14 0.03 ± 0.13 0.00 ± 0.13Seasonal Lead-6 0.18 ± 0.14 0.01 ± 0.14 −0.08 ± 0.14 0.00 ± 0.14Seasonal Lead-7 −0.03 ± 0.14 0.01 ± 0.14 −0.08 ± 0.13 0.00 ± 0.13

3.1.2. Comparisons with other models

In this section, the prediction skill of the IndOzy model iscompared with the results of several dynamical and statis-tical models. Predictions of these models are available forthe period from 2002 to 2005 and are obtained from var-ious institutes mentioned in the introduction and all arepublicly available through the IRI website. The IndOzymodel was run using the training period from 1950 to1995. The results of all the models for seasonal Lead-1to Lead-4 are presented in Figure 4(a)–(d) and the skillsfor all seasonal leads are presented in Tables II–IV.

For short lead times (Figure 4(a) and (b)), most mod-els, except the IndOzy model, predict ENSO peaks muchearlier and weaker than they should. On the other hand,even though IndOzy predicts the peak after its occur-rence, this time delay of up to two seasonal lead timesis present. In addition, the predicted amplitude is quite

close to the observed value. This results in the IndOzymodel having higher skill for one and two seasonal leadtimes in comparison with other models, including theno-skill random and ENSO-CLIPER predictions, in allskill measures (Tables II–IV). Other models that per-form well in this short-range prediction are the NationalCenters for Environmental Predictions (NCEP)-Markov,the Constructed-analog (CA) of van den Dool, and theUBC nonlinear CCA statistical models and the hybridSIO dynamical model. On the other hand, when usingcorrelation coefficients as a measure of skill (Table IV),the CA, CCA and SIO models perform poorly with coef-ficients below 0.5.

At longer leads, i.e. seasonal Lead-3 to Lead-6, IndOzymodel, lost its skill even in comparison with the no-skill random forecast. The CA of van den dool alsosuffered a similar fate as IndOzy. On the other hand, some



Table II. Skill measure for all ENSO models using RMSE. A symbol ‘ – ’ means that the model does not have a prediction atthat particular lead time.

Model Root-mean-squared error (RMSE) °C

Lead-1 Lead-2 Lead-3 Lead-4 Lead-5 Lead-6 Lead-7

NASA 1.01 0.95 0.89 0.86 0.84 0.84 0.81NCEP CM 0.59 0.52 0.47 0.44 0.49 – –SIO 0.5 0.45 0.4 0.43 0.46 0.47 0.46LAMONT 0.61 0.56 0.47 0.43 0.49 – –ECMWF 0.62 – – – – – –NCEP/MKV 0.47 0.38 0.29 0.3 0.34 0.36 0.41NOAA/CDC 0.65 0.71 0.77 0.79 0.83 0.85 0.85DOOL CA 0.55 0.55 0.62 0.69 0.73 0.78 0.81NCEP/CCA 0.41 0.43 0.48 0.51 0.47 0.46 0.44CLIPER 0.6 0.58 0.58 0.56 0.57 0.62 0.63UBC 0.54 0.49 0.46 0.5 0.5 0.49 0.45IndOzy 0.24 0.39 0.49 0.58 0.71 0.75 0.72

Table III. Skill measure using Peirce skill score for all ENSO models. Symbol ‘∗’ is used when the skill is different from thatof the no-skill prediction. Symbol ‘∗∗’ is used when the skill is different from those of the no-skill prediction and the CLIPER

prediction. The score with any skill is also shaded.

Model Peirce skill score


NASA 0.14 ± 0.15 0.31 ± 0.15 0.04 ± 0.16 −0.08 ± 0.16 0.37 ± 0.15∗ 0.24 ± 0.16 0.21 ± 0.16NCEP CM 0.12 ± 0.15 0.25 ± 0.15 0.2 ± 0.16 0.34 ± 0.15∗ 0.32 ± 0.15∗ – –SIO 0.18 ± 0.15 0.31 ± 0.15∗ 0.27 ± 0.15 0.39 ± 0.15∗ 0.37 ± 0.15∗ 0.4 ± 0.15∗ 0.21 ± 0.16LAMONT 0.05 ± 0.15 0.13 ± 0.15 0.26 ± 0.15 0.39 ± 0.15∗ 0.37 ± 0.15∗ – –ECMWF 0.07 ± 0.15 – – – – – –NCEP/MKV 0.42 ± 0.15∗∗ 0.37 ± 0.14∗∗ 0.42 ± 0.15∗ 0.29 ± 0.15 0.26 ± 0.15 0.17 ± 0.16 0.18 ± 0.16NOAA/CDC 0.32 ± 0.15∗ 0.27 ± 0.15 0.02 ± 0.16 0.1 ± 0.16 −0.1 ± 0.16 −0.11 ± 0.16 −0.01 ± 0.16DOOL CA 0.26 ± 0.15 0.4 ± 0.14∗∗ 0.39 ± 0.15∗ 0.43 ± 0.14∗ 0.16 ± 0.16 0.16 ± 0.16 0.1 ± 0.17NCEP/CCA 0.2 ± 0.15 0.12 ± 0.15 0.09 ± 0.16 0.1 ± 0.16 0.1 ± 0.16 −0.05 ± 0.16 −0.1 ± 0.17CLIPER 0.05 ± 0.15 0.06 ± 0.16 0.29 ± 0.15 0.37 ± 0.15∗ 0.32 ± 0.15∗ 0.21 ± 0.16 0.2 ± 0.16UBC 0.31 ± 0.15∗ 0.4 ± 0.14∗∗ 0.15 ± 0.16 −0.07 ± 0.15 −0.21 ± 0.16 −0.14 ± 0.16 0 ± 0.17IndOzy 0.76 ± 0.1∗∗ 0.46 ± 0.14∗∗ 0.29 ± 0.15 0.27 ± 0.15 0.11 ± 0.16 0.01 ± 0.16 −0.07 ± 0.17CONSENSUS 0.46 ± 0.14∗∗ 0.36 ± 0.14∗ 0.55 ± 0.13∗ 0.58 ± 0.13∗ 0.37 ± 0.15∗ 0.37 ± 0.15∗ 0.41 ± 0.15∗

Table IV. Skill measures for all ENSO models using correlation. The useful prediction with correlation greater than 0.5 is shaded.

Model Pearson correlation


NASA −0.08 0.15 0.29 0.41 0.45 0.42 0.36NCEP CM 0.18 0.33 0.41 0.45 0.29 – –SIO 0.33 0.46 0.56 0.52 0.43 0.42 0.4LAMONT 0.26 0.38 0.55 0.62 0.51 – –ECMWF −0.04 – – – – – –NCEP/MKV 0.48 0.62 0.78 0.78 0.72 0.71 0.6NOAA/CDC 0.25 0.16 0.05 0.06 −0.06 −0.05 0.06DOOL CA 0.42 0.49 0.52 0.52 0.53 0.52 0.52NCEP/CCA 0.57 0.54 0.42 0.35 0.44 0.49 0.51CLIPER 0.14 0.27 0.37 0.43 0.51 0.53 0.5UBC 0.34 0.4 0.44 0.36 0.31 0.31 0.45IndOzy 0.88 0.7 0.55 0.38 0.11 0 0.17



Figure 4. (a) IndOzy model and other model predictions against observation: season Lead-1; (b) IndOzy model and other model predictionsagainst observation: season Lead-2; (c) IndOzy model and other model predictions against observation: season Lead-3; (d) IndOzy model and

other model predictions against observation: season Lead-4.

dynamical models and another statistical model start toobtain much higher skill with longer lead time. Theyare the Lamont, NCEP coupled, and NASA dynamicalmodels and CLIPER-ENSO statistical models. The poorskill of the van den Dool CA model and the highskill of the NASA model using the Peirce skill score(Table III) measure are in contradiction to the correlationmeasures (Table IV). Some earlier investigators haverejected correlation as a criterion for this purpose (refer

for instance Woodcock (1976); Harvey et al., (1992) andreferences therein). At seasonal Lead-7, all skill measuresshow that none of the models has any skill even againstthe no-skill random prediction.

In some instances, predictions can be improved bygenerating a consensus forcast, i.e. averaging the outputsof the different models at each prediction lead (Table III).This consensus/ensemble technique is widely used inclimate forecasting (Thompson, 1977; Hagedorn et al.,



Figure 4. (Continued).

2005). It can be observed in Table III that the consensusforecast provides a skill, significantly improved from ano-skill random prediction.

It is interesting to note that many model results at shortlead times are surprisingly poor, i.e. only a few modelsare considered skillful against both the CLIPER and theno-skill random forecast. This is similar to the situationwhen the skill of both dynamical and statistical modelswas found to be lower than that of persistence as reportedby Latif et al., (1998) and Goddard et al. (2001).

One reason for the relatively poor performance of themodels may be that the prediction set used does notcontain a major ENSO event as found in other studies

(e.g. van den Dool and Toth, 1991; Mason and Mim-mack, 2002). Using the COLA anomaly coupled model,Kirtman (2003) showed that near-normal conditions havelower prediction skill than those of El Nino and La Ninaconditions at lead times up to 12 months.

3.2. Predictions using data set KAPLAN (1856–2005)

In this section, the IndOzy model is used to predictNino 3.4 indices using the extended data set that hasalso been used in the Zebiak–Cane model (Chen et al.,2004). Predictions were made for successive 20 yearperiods using the remaining data as training data. Forexample, the model was run to predict the Nino 3.4 index




from 1916 to 1935 using training data from the period1856–1915 and also the period from 1936 to 2005. Thus,a slightly different training set was used for each of the20 year prediction periods. It should be noted that thisis a slightly different approach from that of Chen et al.(2004) who used the period from 1976 to 1995 to tunetheir model and the rest of the data for comparison withmodel predictions.

Figure 5 illustrates the correlation coefficient andRMSE for the predictions over the various 20 year peri-ods. The results for each period are broadly the same withrapidly decreasing correlation coefficients and increasingRMSE, as the lag period is increased. These results are

also similar to those of Chen et al. (2004). The similarityis shown when we plot the average and standard devia-tion of correlation and RMSE at each lead time for theseven 20 year period predictions (Figure 6). Both modelsshow similar correlation up to 5 month lead. Beyond thislead, IndOzy prediction has lower correlation than that ofthe LDEO5. In terms of RMSE, however, both modelsdo not significantly differ at all lead times.

The prediction skill of the IndOzy model is alsoevaluated using the Peirce skill score in Figure 7. Itshows that the IndOzy model only has skill up to 4 monthlead, i.e. the skill is only better than the no-skill randomprediction up to and including 4 month lead.




Finally, a time series of the IndOzy prediction forthe period from 1981 to 2005 is shown in Figure 8.Visual comparison of the data and prediction seems toindicate a high degree of model skill even for Lead-6for which it has been shown above; the model in facthas no skill. Close inspection of the predictions revealthat the predictions have a very similar waveform tothe data, but are slightly delayed in time and smaller inamplitude. Because the data is plotted on a time scale thatis 20 years long, the slight delay in the predicted peaksappears very small and the predicted waveform appearsto follow the data very closely. This raises an importantpoint about assessing the performance of a model based

on representations such as those in Figure 8, suggestingthe real possibility of gaining a false impression of themodel performance.

4. Discussion and conclusion

IndOzy is an extremely simple model that makes useof only the previous Nino 3.4 SSTA data to makepredictions. Despite its simplicity, IndOzy has beenshown to perform favourably when compared with othermore elaborate and complex models. Using a short periodof test data, for which other model predictions wereavailable, the performance of IndOzy for short seasonal



Figure 5. IndOzy out-of-sample prediction skill against observation up to 12 month leads measured using the Pearson correlation (griddedleft-hand diagram) and RMSE (right-hand diagram). Each solid line in both diagrams represents skill for the seven 20 year periods and the

1981–2005 period. This figure is available in colour online at www.interscience.wiley.com/ijoc

Figure 6. IndOzy and LDEO5 (Chen et al., 2004) models prediction skill obtained by averaging the Pearson correlation (left diagram) and RMSE(right diagram) up to 12 month leads for the seven 20 year periods excluding the 1981–2005 period. The standard deviation is plotted as error

bar. This figure is available in colour online at www.interscience.wiley.com/ijoc

lead times (Lead-1 and Lead-2) was found to be superiorto other models. However, with Lead-3 the skill of themodel had dropped to be no better than the no-skillrandom prediction.

When tested against longer data sets, IndOzy wasfound to have skill up to 4 or 5 month leads and

performed only slightly worse than the more complicatedmodel of Chen et al. (2004); although, to do this, a longertraining data set was used compared to that used by Chenet al. (2004).

The fact that such a simple model as IndOzy can per-form comparably with respect to the far more complicated



Figure 7. IndOzy out-of-sample prediction skill against observation upto 12 month leads measured using the Peirce skill score. Each line in thediagram represents score along with an error bar for the seven 20 yearperiods and the 1981–2005 period. The no-skill prediction depicted bycrosses ‘×’ is also plotted with an error bar. This figure is available in


Figure 8. Time series of the IndOzy model prediction at Lead-1,Lead-3, and Lead-6 months during the 1981–2005 period. This figure

is available in colour online at www.interscience.wiley.com/ijoc

models raises some philosophical questions about mod-elling extremely complicated systems such as ENSO.Why is it that a model that uses the minimum data andthat can run on an office desktop computer can performon par with much more complicated and computing inten-sive codes? Dynamical models that use the basic lawsof physics are the most scientifically satisfying approachused for prediction of any phenomenon. In the case of theENSO predictions, these models use extremely detaileddescriptions of the atmosphere and the ocean. Many ofthe statistical models use a large number of inputs inorder to make predictions. In the long run, these largedynamical and statistical models offer the best hope ofincreasing the range of ENSO prediction, but the lack ofmajor improvement in prediction ability of these presentmodels over a simple model such as IndOzy leads one tospeculate that much of the complexity of many of thesemodels is of little value at present.

This raises the question of what is the most sensitiveinput data and physical processes that are necessaryfor the best prediction. The success of IndOzy, whichuses only the historic data from Nino 3.4 indicates thatsome of the data used in more complicated models maycontribute insignificantly to model accuracy. In addition,for the dynamical models, it seems likely that some of thephysical processes that are included also contribute little.

We suggest that in order for the large models toprogress so that they become significantly more usefulthan simple models such as IndOzy, much more atten-tion needs to be focused on the most sensitive inputdata, or combinations of input data (Zhang et al., 2005;AchutaRao and Sperber, 2006; Fei et al., 2006). Addi-tionally, for dynamic models, the way forward may notbe to simply increase the computer power and reducegrid sizes but to be focused more on determining whatphysical processes, i.e. model parameterizations and cou-pling strategies (Wu and Kirtman, 2005; Power and Col-man, 2006). It is evident from this paper that even thelargest models running on amongst the largest computersin the world are presently not performing much betterthan models that can be run on a desktop computer.There seems to be no reason why using even fastercomputers will do any better than the present day super-computers unless a fundamental breakthrough is made inunderstanding the processes that trigger the ENSO events(Eisenman et al., 2005; Kondrashov et al., 2005; Perezet al., 2005; Saynisch et al., 2006; Vecchi et al., 2006).Such an understanding would allow more targeted datacollection to drive the models and would focus modi-fications to dynamical model codes to those processesthat matter most. Also, in order to solve the problem ofsensitivity to initial conditions, higher quality input datais likely required before significant improvement in thedynamical model performance can be expected.

Acknowledgements

We thank the IRI-Columbia Univ. and the CPC-NCEP-NOAA for making the seasonal ENSO prediction andthe Nino 3.4 index available to public. We express ourgratitude to Dr A. Kaplan of LDEO Columbia Universityfor helping us finding the extended Nino 3.4 SSTA dataset. The critical comments by two anonymous reviewersconsiderably helped us on focusing and improving thepaper. HH would also like to express his gratitude to theAustralian Government for the AUSAID PhD scholarshipand the ACIAR postdoctoral fellowship and also to DrD. McKinnon for his continuous encouragement.

A1. Appendix A

A1.1. IndOzy model

Let the data be represented by the vector X with the inputdata elements represented by X(k), where k = 1, . . . , n;and n is the number of data points in the time series.This data could be either the monthly Nino 3.4 index



or the 3 monthly averaged Nino 3.4. To predict thevalue of X at time P steps ahead of the present time,t , i.e. X(t + P), we need as predictors five elements ofthe previous X, data, i.e. [X(t); X(t − P); X(t − 2P);X(t − 3P); and X(t − 4P)]. This form of time delayingfollows Lapedes and Farber (1987) and Jang (1993)and references therein. For example, predicting 4 timesteps ahead (X(t + 4)), the predictor elements are: [X(t);X(t − 4); X(t − 8); X(t − 12); and X(t − 16)]. It isclear that the model uses simpler predictors than thoseof the CLIPER model (Knaff and Landsea, 1997). Thelatter uses historical ENSO events from 1950 to 1994,persistence, and trends in recent observed SSTA data(Barnston et al., 1999; Landsea and Knaff, 2000). Havingdetermined our predictors/predictant pairs, we then usea simple linear neural network method to configure therelation between pairs.

The linear neural network (LNN) applied in this studyis different from the more complex neural networks usedfor other ENSO prediction (Elsner and Tsonis, 1992;Hsieh and Tang, 1998), in two respects. First, it consistsonly an input and output layer without any hidden layer.Second, it does not make use of the back propagationtechnique of finding the optimum weights for reducingthe error. Instead, it employs a least mean square error(LMS) or Widrow-Hoff algorithm (Hagan et al., 1996).The algorithm adjusts the weights and biases of thenetwork to minimize the error, i.e. difference between thedata and prediction (Demuth et al., 2005). It can be notedthat for an LNN, the weights can be viewed as similarto the regression coefficient in a regression analysis andthe bias is the interception point between the regressionline and the dependent axis.

The model is implemented as follows. The Nino 3.4index data set was divided into two parts of unequallengths called the training and testing data sets. Thetraining data were used to determine the weights andbiases to relate the five lagged inputs/predictors datato the predictant output data. These lagged inputsduring model training, Itrain = {X(t), X(t − P), X(t −2P), X(t − 3P), X(t − 4P)} and an output, Otrain ={X(t + P), become inputs for the MATLAB algorithmcalled NEWLIN. This subroutine is used for calculat-ing the strength of these input-output pair relations, i.e.their weight, and the bias using the LMS algorithm.Let us call the resulting weights and bias for each pairW = {Wt, Wt−P , . . . Wt−4P }, and b, respectively. Theseweights and biases are then used to give an out-of-sampleprediction, Otest prediction, given a set of predictors Itestof the testing data set using SIM, another MATLAB

subroutine used for linearly transforming the inputs andthe bias, i.e. Otest = W I test + b.

B1. Appendix

B1.1. Skill measures

There are two commonly used measures for evaluatingENSO prediction: deterministic approach, i.e. the Pearson

Table BI. Contingency table for the Yes/No ENSO forecast.

ENSO eventforecast

ENSO event observed

Yes No

Yes a (hit) b (false alarm)No c (miss) d (correct rejection)

correlation coefficient and RMSE. Another approach hasalso been applied in ENSO model verification such as:RPSS (rank probability skill score) and ROC (receiveroperating characteristic) (Mason and Mimmack, 2002;Kirtman, 2003). In this study, another type of probabilis-tic skill measure commonly used in weather prediction,the Peirce score, is applied to assess the ENSO prediction.This skill measure can readily be used against no-skillrandom forecast. The formulae for both types of skillmeasures are presented.

B1.2. Pearson correlation and RMSE

Prediction skill for most of the ENSO models can beexpressed as the RMSE and correlation coefficient. Thesemeasures of skill are defined as follows (Wilks, 1995).The root-mean-squared error (RMSE) are defined as

RMSE =[

1/n(

n∑m=1

(pm − om)2

]1/2

(B1)

where pm and om are the mth prediction and observedvalue, respectively (m = 1, 2, . . . , n), and the correlationcoefficient is

r =n∑

m=1

(pm − p)(om − o)/

[n∑

m=1

(pm − p)2

]1/2

[n∑

m=1

(om − o)2

]1/2

(B2)

where p and o are the mean values of the prediction andobservation, respectively.

B1.3. Peirce score and its error estimates

The Peirce skill score (Woodcock, 1976, 1981; Stephen-son, 2000) ranges from −1 to 1, and has been demon-strated by Stephenson (2000) to have good properties asa skill measure because of its fairness against forecasthedging. This is where a forecaster tends to favour aparticular event. It also has a complementary symmetry;interchanging event to non-event and vice versa does notaffect the skill. One useful aspect of the Pierce Score isthat both an error estimate and a random forecast scoreare available. It is thus possible to test whether a particu-lar model has a skill that is superior to a no-skill randomforecast. The score is based on a categorical type of fore-cast such as the Yes/No forecast. A contingency table forthe Yes/No categorical forecast is shown in Table BI.



In Table BI a, b, c, and d refer respectively to thenumber of times the event (either El Nino or La Nina)is forecast and also observed, the event is forecast butdid not occur, the event is not forecast but did occur, andthe event is neither forecast nor observed. El Nino or LaNina events are respectively defined as those occasionswhen the 3 month average of the Nino 3.4 SSTA is equalto or above 0.5 °C, and is equal to or below −0.5 °C(McPhaden, 2004). It can be noted that when El Ninowas predicted, but La Nina was observed or vice versa,a category b was assigned.

Having determined the parameter values of the Yes/Noforecast i.e. a, b, c, and d based on the contingencytable of Table BI, prediction skills along with their errorestimates were calculated for the model. The formulaefor the skill score and the error estimates are fromStephenson (2000):

Peirce skill score PSS = (ad − bc)/(a + c)(b + d)

(B3)

Standard error ePSS = [(n2 − 4(a + c)(b + d)

× PSS2)/4n(a + c)(b + d)]1/2 (B4)

where the total number of predictions and observationsn = a + b + c + d .

The prediction skills obtained from all ENSO models,including the IndOzy model, are compared against arandom no-skill forecast (Stephenson, 2000). The no-skill forecast parameters are obtained by performing thefollowing transformation on a, b, c, d values resultingfrom model’s prediction (Woodcock, 1976; Stephenson,2000).

ar = (a + c)(a + b)/n (B5)

br = (b + d)(a + b)/n (B6)

cr = (a + c)(c + d)/n (B7)

dr = (b + d)(c + d)/n (B8).

The skill scores of this random forecast are calculatedby replacing a, b, c, and d by ar , br , cr , and dr in theabove skill scores formulae.

References

Abarnabel HDI, Lall U. 1996. Nonlinear dynamics of Great SaltLake: system identification and prediction. Climate of Dynamic 12:287–297.

AchutaRao K, Sperber KR. 2006. ENSO simulation in coupled ocean-atmosphere models: are the current models better? Climate ofDynamic 27: 1–15.

Adams RM, Bryant KJ, McCarl BA, Legler DM, O’Brien J, Solow A,Weiher R. 1995. Value of improved long-range weather information.Contemporary Economic Policy 13: 10–19.

Anderson D. 2004. Testing time for El Nino. Nature 428: 709–711.Bacmeister JT, Suarez MJ. 2002. Wind stress simulation and the

equatorial momentum budget in an GCM. Journal of the AtmosphericSciences 59: 3051–3073.

Balmaseda MA, Anderson DLT, Davey MK. 1994. ENSO predictionusing a dynamical ocean model coupled to a statistical atmospheres.Tellus 46A: 497–511.

Barnett TP, Latif M, Graham N, Flugel M, Pazan S, White W. 1993.ENSO and ENSO-related predictability. Part 1: Prediction ofequatorial Pacific sea surface temperature with a hybrid coupledocean-atmosphere model. Journal of Climate 6: 1545–1566.

Barnston AG, Ropelewski CF. 1992. Prediction of ENSO episodesusing canonical correlation analysis. Journal of Climate 7:1316–1345.

Barnston AG, Glantz MH, He Y. 1999. Predictive skill of statisticaland dynamical climate models in SST forecasts during the 1997-98El Nino episode and the 1998 La Nina onset. Bulletin of the AmericanMeteorological Society 80: 217–243.

Barnston AG, van den Dool HM, Zebiak SE, Barnett TP, Ming J,Rodenhuis DR, Cane MA, Leetmaa A, Graham NE, RopelewskiCR, Kousky VE, O’Lenic EA, Livezey RE. 1996. Long-leadseasonal forecasts: Where do we stand? Bulletin of the AmericanMeteorological Society 75: 2097–2114.

Bouma MJ, Poveda G, Rojas W, Chavasse D, Quinones M, Cox J,Patz J. 1997. Predicting high-risk years for Malaria in Colombiausing parameters of El Nino Southern Oscillation. Tropical Medicineand International Health 2: 1122–1127.

Changnon D, Ritsche M, Elyea K, Shelton S, Schramm K. 2000.Integrating climate forecasts and natural gas supply information intoa natural gas purchasing decision. Meteorology Applied 7: 211–216.

Chen C-C, McCarl B, Hill H. 2002. Agricultural value of ENSOinformation under alternative phase definition. Climatic Change 54:305–325.

Chen D, Cane MA, Kaplan A, Zebiak S, Huang D. 2004. Predictabil-ity over the past 148 years. Nature 428: 733–736.

Chichilnisky G, Heal G. 1998. Managing unknown risks–the future ofglobal reinsurance. Journal of Portfolio Management 24: 85–91.

Clarke AJ, van Gorder S. 2001. ENSO prediction using an ENSOtrigger and proxy for western equatorial Pacific warm poolmovement. Geophysical Research Letters 28: 579–582.

Costello CJ, Adams RM, Polasky S. 1998. The value of El Ninoforecasts in the management of Salmon: a stochastic dynamicassessment. American Journal of Agricultural Economics 80:765–777.

Demuth H, Beale M, Hagan M. 2005. Neural Network Toolbox–Foruse with MATLAB. The Maths Works Inc: Natik, MA, USA.

Eisenman I, Yu L, Tziperman E. 2005. Westerly Wind Bursts: ENSO’sTail Rather than the Dog? Journal of Climate 18: 5224–5238.

Elsner JB, Tsonis AA. 1992. Nonlinear prediction, chaos and noise.Bulletin of the American Meteorological Society 73: 49–60.

Fei Z, Jiang Z, Zhang R-H, Guangqing Z. 2006. Improved ENSOforecasts by assimilating sea surface temperature observations intoan intermediate coupled model. Advances in Atmospheric Sciences23: 615–624.

Gautama T, Mandic DP, van Heulle MM. 2004. The delay vectorvariance method for detecting determinism and nonlinearity in timeseries. Physica D 190: 167–176.

Goddard L, Mason SJ, Zebiak SE, Ropelewski CF, Basher R,Cane MA. 2001. Current approaches to seasonal-to-interannualclimate predictions. International Journal of Climatology 21:1111–1152.

Hagan MT, Demuth HB, Beale M. 1996. Neural Network Design. PWSPublishing Company: Boston.

Hagedorn AF, Doblas-Reyes FJ, Palmer TN. 2005. The rationalebehind the success of multi-model ensembles in seasonal forecasting.I. Basic concept. Tellus 57A: 219–233.

Hamlet AF, Huppert D, Lettenmaier DP. 2002. Economic value oflong-lead streamflow forecasts for Columbia River hydropower.Journal of Water Resources Planning-ASCE 128: 91–101.

Harvey LO, Hammond KR, Lusk CM, Ross EFM. 1992. Theapplication of signal detection theory to weather forecastingbehavior. Monthly Weather Review 120: 863–883.

Hsieh WW. 2001. Nonlinear canonical correlation analysis of thetropical Pacific climate variability using a neural network approach.Journal of Climate 14: 2528–2539.

Hsieh WW, Tang B. 1998. Applying neural network models toprediction and data analysis in Meteorology and Oceanography.Bulletin of the American Meteorological Society 79: 1855–1870.

Jang J-SR. 1993. ANFIS: Adaptive-Network-based Fuzzy InferenceSystem. IEEE Trans in System, Man, and Cybernetics 23: 665–685.

Ji M, Leetmaa A, Kousky VE. 1996. Coupled model forecasts ofENSO during the 1980s and 1990s at the National MeteorologicalCentre. Journal of Climate 9: 3105–3120.



Jochec KG, Mjelde JE, Lee AC, Conner JR. 2001. Use of seaonalclimate forecasts in rangeland-based livestock operations in WestTexas. Journal of Applied Meteorology 40: 1629–1639.

Kang I-S, Kug J-S. 2000. An El Nino prediction system with anintermediate ocean and statistical atmosphere. Geophysical ResearchLetters 27: 1167–1170.

Kaplan A, Cane MA, Kushnir Y, Clement AC, Blumenthal MB,Rajagopalan B. 1998. Analysis of of global sea surface temperature1896–1971. Journal of Geophysical Research 103: 18567–18589.

Kerr RA. 1998. Models win big in forecasting El Nino. Science 280:522–523.

Kerr RA. 2000. Second thoughts on skill of El Nino predictions.Science 290: 257–258.

Kerr RA. 2002. Signs of success in forecasting El Nino. Science 297:497–499.

Kirtman BP. 2003. The COLA anomaly coupled model: EnsembleENSO prediction. Monthly Weather Review 131: 2324–2341.

Knaff JA, Landsea CW. 1997. An El Nino-Southern OscillationCLImatology and PERsistence (CLIPER) forecasting scheme.Weather Forecast 12: 633–652.

Kondrashov D, Kravtsov S, Robertson AW, Ghil M. 2005. A hierarchyof data-based ENSO models. Journal of Climate 18: 4425–4444.

Landsea CW, Knaff JA. 2000. How much skill was there in forecastingthe very strong 1997-98 El Nino. Bulletin of the AmericanMeteorological Society 81: 2107–2119.

Lapedes A, Farber R. 1987. Nonlinear Signal Processing using NeuralNetworks: Prediction and System Modeling . Technical Report LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, NewMexico.

Latif M, Anderson D, Barnett T, Cane M, Kleeman R, Leetmaa A,O’Brien J, Rosati A, Schneider E. 1998. A review of thepredictability and prediction of ENSO. Journal of GeophysicalResearch 103: 14375–14394.

Lehodey P, Bertignac M, Hampton J, Lewis A, Picaut J. 1997. El NinoSouthern oscillation and tuna in the western Pacific. Nature 389:715–718.

Letson D, Podesta GP, Messina CD, Ferreyra RA. 2005. The uncertainvalue of perfect ENSO phase forecasts: Stochastic agricultural pricesand intra-phase climatic variations. Climatic Change 69: 163–196.

Mason SJ, Mimmack GM. 2002. Comparison of some statisticalmethods of probabilistic forecasting of ENSO. Journal of Climate15: 8–29.

McKeon G, Ash A, Hall W, Smith MS. 2000. Simulation of grazingstrategies for beef production in north-east Queesnland. InApplications of Seasonal Climate Forecasting in Agriculturaland Natural Ecosytems-The Australian Experience, Hammer GL,Nicholls N, Mitchell C (eds). Kluwer Acadamic Publishers:Dordrecht; 227–252.

McPhaden MJ. 2004. Evolution of the 2002/03 El Nino. Bulletin of theAmerican Meteorological Society 85: 677–695.

Molteni F, Buizza R, Palmar TN, Petroliagis T. 1996. The ECMWFensemble prediction system: methodology and validation. QuarterlyJournal of the Royal Meteorological Society 122: 73–119.

Packard NH, Crutchfield JP, Farmer JD, Shaw RD. 1980. Geometryfrom a time series. Physical Review Letters 45: 712–716.

Penland C, Magorian T. 1993. Prediction of Nino 3 sea surfacetemperatures using Linear Inverse Modeling. Journal of Climate 6:1067–1076.

Perez CL, Moore AM, Zavala-Garay J, Kleeman R. 2005. A compari-son of the influence of additive and multiplicative stochastic forcingon a coupled model of ENSO. Journal of Climate 18: 5066–5085.

Power S, Colman R. 2006. Multi-year predictability in a coupledgeneral circulation model. Climate Dynamic 2 6: 247–272.

Regonda SK, Rajagopalan B, Lall U, Clark M, Moon Y-I. 2005. Localpolynomial method for ensemble forecast of time series. NonlinearProcesses in Geophysics 12: 397–406.

Samelson RM, Tziperman E. 2001. Instability of the chaotic ENSO:The growth-phase predictability barrier. Journal of the AtmosphericSciences 58: 3613–3625.

Sauer T. 1994. Time series prediction by using delay coordinateembedding. In Time Series Prediction: Forecasting the Futureand Understanding the Past, Weigend AS, Gershenfeld NA (eds).Addison-Wesley Publishers Company: Reading; 175–193.

Saynisch J, Kutths J, Maraun D. 2006. A conceptual ENSO modelunder realistic noise forcing. Nonlinear Processes in Geophysics 13:275–285.

Shibata K, Yoshimura H, Ohizumi M, Hosaka M, Sugi M. 1999. Asimulation of troposphere, stratosphere and mesosphere with anMRI/JMA98 GCM. Papers in Meteorology and Geophysics 50:15–53.

Solow A, Adams RF, Bryant KJ, O’Brien JJ, McCarl BA, Nayda W,Weiher R. 1998. The value of improved ENSO prediction to U.S.agriculture. Climatic Change 39: 47–60.

Stephenson DB. 2000. Use of the “Odds ratio” for diagnosing forecastskill. Weather Forecast 15: 221–232.

Syu HH, Neelin JD, Gutzler D. 1995. Seasonal and interannualvariability in a hybrid coupled GCM. Journal of Climate 8:2121–2143.

Takens F. 1981. Detecting strange attractors in turbulence. InDynamical Systems and Turbulence–Lecture Notes in Mathematics898, Rand DA, Young LS (eds). Springer-Verlag: Berlin; 366–381.

Tang Y, Hsieh WW. 2002. Hybrid coupled models of the tropicalPacific. II. ENSO prediction. Climate Dynamics 19: 343–353.

Thompson PD. 1977. How to improve accuracy by combiningidependent forecasts. Monthly Weather Review 105: 228–229.

Tziperman E, Scher H, Zebiak SE, Cane MA. 1997. Controllingspatiotemporal chaos in a realistic El Nino prediction model.Physical Review Letters 79: 1034–1037.

van den Dool HM. 1994. Searching for analogues, how long must wewait? Tellus 46A: 314–324.

van den Dool HM, Toth Z. 1991. Why do forecasts for “near normal”often fail? Weather Forecast 6: 76–85.

Vecchi GA, Wittenberg AT, Rosati A. 2006. Reassessing the role ofstochastic forcing in the 1997–1998 El Nino. Geophysical ResearchLetters 33: L01706, DOI: 10.1029/2005GL024738.

Wang G, Kleeman R, Smith N, Tseitkin F. 2002. The BMRC coupledgeneral circulation model ENSO forecast system. Monthly WeatherReview 130: 975–991.

Wilks DS. 1995. Statistical Methods in the Atmospheric Sciences.Academic Press: San Diego.

Woodcock F. 1976. The evaluation of yes/no forecasts for scientific andadministrative purposes. Monthly Weather Review 104: 1209–1214.

Woodcock F. 1981. Hanssen and Kuipers’ discriminant related to theutility of yes/no forecasts. Monthly Weather Review 109: 172–173.

Wu R, Kirtman BP. 2005. Roles of Indian and Pacific Ocean air–seacoupling in tropical atmospheric variability. Climate Dynamics 2 5:155–170.

Xue Y, Leetmaa A, Ji M. 2000. ENSO predictions with Markovmodels: the impact of sea level. Journal of Climate 13: 849–871.

Zebiak SE, Cane MA. 1987. A model El Nino-Southern Oscillation.Monthly Weather Review 115: 2262–2278.

Zhang S, Harrison MJ, Wittenberg AT, Rosati A, Anderson JL,Balaji V. 2005. Initialization of an ENSO forecast system usinga paralleized ensemble filter. Monthly Weather Review 133:3176–3201.


Documents

Complicated ENSO models do not significantly outperform very simple ENSO models