Are forecasting methods too complex?

Kesten C. Green*, University of South Australia

J. Scott Armstrong*, University of Pennsylvania *Ehrenberg-Bass Institute at University of South Australia

International Symposium on ForecastingRiverside, California

24 June 2015Slides available at ForPrin.com

G&A ISF 2015 – Complex-V13

On the value of complex forecasting methodsAnalysts have long assumed that complex methods are needed to deal with complex problems.

As long ago as 1985 (Armstrong, pp.225-232), a review of the literature found that complexity tends to:•reduce accuracy and understanding, and•increase costs and mistakes

Despite the evidence, forecasters continue to use ever more complex methods.

We reviewed this issue by organizing a JBR Special Issue on Simple versus complex forecasting. We report on our paper in that issue reviewing the evidence.

2

3

See HANDOUT

Review of experimental research

Reviewed published research from all areas of forecasting, including the Special Issue papers… 1.defined simplicity in forecasting2.identified studies with evidence on comparative accuracy3.assessed directional and effect size evidence

4

Simple forecasting defined

Simplicity in forecasting requires that are all of the following are understood by clients.

1. forecasting method,

2. representation of cumulative knowledge,

3. relationships in models,

4. relationships among models, forecasts, and decisions

5

FindingsFound 32 papers with 97 comparisons:

a) None of the papers found that complexity helped accuracy

b) Complexity increased error by 27% on average across papers

“Simple versus complex forecasting: The evidence” was published in JBR in 2015. Due out in print in August.

6

Summary of evidence on accuracy of forecasts from complex vs. simple methods

7

Simplicity: A checklistScore the following on a 0-to-10 scaleUse of prior knowledge in forecasting models1.Do you know what prior knowledge about the situation was used?2.Do you know how prior knowledge about the situation was used?3.How simply is prior knowledge represented?Nature of the relationships among the model elements (Non-linear… Multiplicative… Additive… Single?)Nature of the relationships among models, forecasts, and decisions (Weak… Strong?)Explaining the forecasting processI am confident that I could explain… to the decision maker1.the forecasting methods2.how prior knowledge about the situation is represented in the forecasting models3.the nature of the relationships among the model elements4.how the models, forecasts, and decisions are related to each other

8

Demonstration: Effect of simplicity vs. complexity in climate forecasting

IPCC warming alarmists do not forecast, they create “scenarios” via computer simulations1.Scenarios are:

a. Stories about “what happened in the future”b. Biased, so they do not provide valid forecasts

(Gregory & Duran, 2001).2.The stories are based on expert judgments. According to prior research, expert judgments about what will happen in complex, uncertain situations are useless:

a. Seer-sucker Theoryb. Tetlock’s 20-year experiment

9

IPCC process of creating climate scenarios via computer simulations is enormously complex

1. Judgments are made on what variables to include (e.g. CO2), and exclude (e.g. Sun);

2. Judgments are made on the values of many parameters, and their (nonlinear) relationships

3. Around 50,000 grid squares are modeled4. Grid square models interact5. Models are adjusted to produce expected

outputs6. Budget for computer simulations is enormous

10

Green, Armstrong, & Soon (2009) no-change model is sophisticatedly simple

1. Based on an examination of diverse long temperature histories…

2. No long-term trend3. Reversals on all time scales4. Correlated with solar cycles and variations in

activity 5. Weakly correlated with CO2…

but temperature changes precede CO2 changes, and high CO2 levels have been associated with ice ages.

11

Simple Forecasting Checklist ratings:IPCC projections vs. no-change forecasts

12

Evidence on accuracy of IPCC projections vs. no-change forecasts using Hadley data

Tests of forecasts over the 1851-1975 forecasting period yielded 58 forecasts for horizons of 91 to 100 years. The errors of these IPCC forecasts were 12.6 times larger than those from the easily understood no-change model (Green, Armstrong, & Soon 2009).

13

Chart from Forecasting Global Climate Change (2014)

14

From Forecasting Global Climate Change (2014)

Evidence on accuracy of IPCC projections vs. no-change forecasts using Loehle & McCulloch (2008)

data

Validation over different time-periods; data: Similar results

From Forecasting Global Climate Change (2014)15

The Complexity Penalty

In the late 1970, a research review found that complex forecasting methods harmed accuracy.

In the late 1990s, the Forecasting Principles Project developed 139 principles for forecasting. All were simple to understand.

Our recent systematic review failed to find a single study showing that a complex method was more accurate than a simpler method.

16

Two Questions for you

What percentage of papers at this conference propose complex methods?

Why does that happen?

17

Seduced by complexity

Some evidence suggests that the popularity of complexity may be due to incentives: (1)researchers are rewarded for publishing in highly ranked journals, which favor complexity;(2)forecasters can use complex methods to provide forecasts that support decision-makers’ plans; and (3)forecasters’ clients may be impressed and reassured by incomprehensibility.

18

Conclusions

1. Those who prefer their forecasts to be accurate should accept forecasts only from simple evidence-based procedures.2. Alarming IPCC temperature projections are based on procedures that are too complex to be trusted.3. Rate the simplicity of forecasting procedures used for problems you are interested in using the questionnaire at… simple-forecasting.com.

19