Universität Hamburg Institut für Wirtschaftsinformatik

Prof. Dr. D.B. Preßmar

Final Results of the NN3 Neural Network Forecasting Competition

Sven F. Crone, Konstantinos Nikolopoulos and Michele Hibon

Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”

2005 SAS & IIF Grant2005 SAS & IIF Grant

RATIONAL

OBJECTIVES

RESULTS

DISCUSSION

FURTHER RESEARCH

2005 SAS & IIF Grant2005 SAS & IIF Grant

RATIONAL

Only 1 evaluation of NN within Forecasting Competitions Distinct fields of research and participation

NN: breakthrough or passing fad?NN: breakthrough or passing fad?

Reid1969

Santa Fe1991

BUSINESS FORECASTING COMPETITIONS

NN COMPETITIONS

Suykens1998

Reid1972

Newbold & Granger1974

Makridakis & Hibon1979

M-Competition 1982

M2-Competition 1988

M3-Competition 2000

H-Competition,Hibon 2006

EUNITE2001

ANNEXG2001

BI Cup2003

CATS2005ISF052005

ISF06 ANNEX 2006

WCCI2006

Only 1 NN entryBalkin & Ord

Most NN competitions = classification (EUNITE’02, WCCI06 etc.) Limited evidence on Regression evaluations

Visit http://www.neural-forecasting.com/competition_data.htm

CI Time Series CompetitionsCI Time Series Competitions

Time Series Data Format Length Submis.

SANTA FE 1991Gershenfeld & Weigend

2 univariate4 multivariate

UV: Laser, UV: Artificial, Sleep, Exchange rate, Astrophysics, Music

1000, 34000, 300000, 100000,

27704, 380830+

Black Box 1998 Suykens & Vandewalle

1 univariate Physics2000

(1000)17

EUNITE 2001 1 multivariate Electrical Load35040(31)

56

ANNEXG 2001Dawson et al.

1 multivariate Hydrology1460 pointsHydrology

12

BI Cup 2003Weber

1 multivariate Sugar sales365 days

(14)10+

CATS 2005, IEEELendasse,

1 univariatein 5 parts

Artificial4905pointas

(95 points, 5*19)25

ISF2005Crone

2 univariate Airline, M3-Competition 144, 85 16

ANNEXG / ISF2006Dawson et al., Crone

3 multivariate Hydrology1460 pointsHydrology

12

WCCI 2006 Predictive Uncertainty, Gawley

1 univariate3 multivariate

UV: Synthetic, Precipitation, Temperature, SO2

380, 10000, 10000, 21000

9

Conduct competition on industry data Evaluate different NN methodologies Can NN forecasting be AUTOMATED on many time series?

Reasons? Modelling DecisionsReasons? Modelling Decisions

Gap between forecasting & NN domains NN evaluations on different data types No positive evidence on M-type data

• Short time series• Noisy time series

Discouraging research findings NN cannot forecast seasonal time series No valid & reliable methodology to model NN No automation of NN modelling possible

Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”

2005 SAS & IIF Grant2005 SAS & IIF Grant

OBJECTIVESa) What is the performance (accuracy, robustness

& resources) of NN in comparison to established forecasting methods?

b) What are the current “best practice” methodologies utilised by researchers to model NN for time series forecasting

Multiple Hypothesis Testing similar to M3-competition

Competition DesignCompetition Design

Multiple empirical Time Series Complete set of 111 time series Reduced set of 11 time series Representative structures monthly industry data

• long & short time series• Seasonal and non-seasonal series

Scaled observations for anonymity No domain knowledge 18 steps ahead forecasts

Simulated ex ante (out of sample) evaluation

Multiple error measures & computational time

Testing of conditions under which NN perform well/bad

NN3 COMPETITION

Competition DesignCompetition Design

46 Submissions for the reduced dataset

9 benchmarks

22 submissions for the complete dataset

8 benchmarks

SubmissionsSubmissions

2005 SAS & IIF Grant2005 SAS & IIF Grant

Automated AI/CI approaches can very well do the job! (batch forecasting)

Balkin’s and Ord approach was not very ‘bad’ after all..

Performance was verified across many metrics (including MASE), parametric + non-parametric

Performance was verified with multiple hypothesis: long/short, seasonal/non seasonal, difficult/easy

So… WHAT do we know NOW that we did not knew before NN3?

2005 SAS & IIF Grant2005 SAS & IIF Grant

Time Series Benchmarks are very hard to beat! Forecast Pro, Theta model and Marc Wildi’s Stat benchmark outperform overall all CI/AI approachesFor the ‘harder’ part of the NN3 dataset – 25 short+non-seasonal series – CI approaches managed to outperform all other approaches!! Full automation seems to be possible in large scale forecasting tasks

+ Side results… New Stat benchmarks that perform outstandinglyImprovement of established forecasting engines in the last 10

years

So… WHAT do we know NOW that we did not knew before NN3?

+

+

+

+

2005 SAS & IIF Grant2005 SAS & IIF Grant

Computational times….

Leaders of the field (Academia + Commercial)

Time series features that would necessitate the use of AI/CI approaches

Replication in a competition of the M3 volume (NN5…111, tourism competition…1000+)

Best practices?

Full automation??

and… WHAT we still DO NOT…

?Sven, Kostas & Michele