Forecast Uncertainty

7/29/2019 Forecast Uncertainty

http://slidepdf.com/reader/full/forecast-uncertainty 1/41

,,)8QLYHUVLWDW /HLS]LJ&RQIHUHQFHRQ7KH)XWXUHRI(FRQRPLF)RUHFDVWLQJ

Forecast Uncertainty,its measurement, representation and evaluation

Kenneth F. Wallis

University of Warwick



)RUHFDVW8QFHUWDLQW\

It is now widely recognised that a point forecast isseldom sufficient for well-informed decision-makingin the face of an uncertain future, and that it needs

to be supplemented with an indication of the degreeof uncertainty. This in turn needs to be reliable.

We survey some of the ways in which economic

forecasters measure and report uncertainty andevaluate the accuracy of their statements.



2XWOLQH,QLWLDODSSURDFKHVWRUHSRUWLQJIRUHFDVWXQFHUWDLQW\

UK government economic forecasts, forecast error margins,

variant forecasts, forecast scenarios, worldwide examples

'HQVLW\IRUHFDVWV

Bank of England, NIESR, more worldwide examples

(YDOXDWLQJGHQVLW\IRUHFDVWV



8.JRYHUQPHQWHFRQRPLFIRUHFDVWV

Forecasts have appeared annually in the FinancialStatement and Budget Report (FSBR) since 1968.

The first two included, alongside the main forecast,a “higher alternative” based on a higher growth rate

of exports than the rate assumed in the main

forecast.



8.JRYHUQPHQWHFRQRPLFIRUHFDVWV«

Warnings were always included that forecasts are subject toa wide margin of error and there is considerable uncertainty;in 1974 the forecasts were “more than usually uncertain” in

the aftermath of OPEC I and the three-day week.

These were not quantified until …

The Industry Act 1975 required the publication of

• WZR forecasts per year, and

• the margin of error attaching to any forecast;• analysis of errors that would have remained even if

forecast assumptions had been correct.



&DOFXODWLQJIRUHFDVWHUURUPDUJLQV

Textbooks give algebraic formulae forcalculating forecast standard errors in a widerange of linear forecasting models:

• univariate and multivariate

• static and dynamic

• single-equation and multiple-equation



&DOFXODWLQJIRUHFDVWHUURUPDUJLQV«

([DPSOH

simple AR(1) fitted to UK RPIX inflation 1997q3-2003q4;forecast standard error incorporating uncertainty with respect

to future random shocks and coefficient estimation error:

In non-linear models, equivalent calculations are donenumerically, using stochastic simulation methods.



and

But the choice of model is uncertain, and practicalforecasts incorporate judgemental adjustments of various

kinds.

So the margins of error are typically calculated withreference to errors made in past forecasts.

Given a sample of past forecast errors, the two standardsummary measures of forecast performance are the root

mean squared error and the mean absolute error, namely

∑=

=Q

W

W H

Q 506(

1

21∑=

=Q

W

W H

Q0$(

1

1




In using these measures to indicate the margin of errorof a current forecast it is implicitly assumed that

uncertainty remains at the same level in the future as inthe past.

Otherwise judgmental adjustments may be made tothese quantities too.

A current example: increased uncertainty about futureoil prices is increasing the uncertainty of inflationforecasts (Bank of England ,QIODWLRQ5HSRUW , Nov. 2005).




Chart 4.3

Market beliefs about oil prices six months ahead(a)

Sources: Bank of England, Bloomberg and New York Mercantile Exchange.

(a) Data refer to the price of West Texas Intermediate crude oil.

(b) Each curve is a probability density function, the area under which sums to one. The area under the curve between two points indicates the inferred probability that market participants attach to oil prices being

between two different levels. See footnote 1 on page 23 for further details.



The first “,QGXVWU\$FW ” forecast was published in(FRQRPLF7UHQGV in December 1976.

A table of mean absolute forecast errors in the mainvariables over the previous ten years was also included

(calculated from unpublished quarterly forecasts).





The 1977 and 1978 FSBRs included similar tables.

They also included, as “another way of expressinguncertainty in economic forecasts”, a variant forecastassuming stronger trade performance than assumed inthe main forecast.

The 1979 FSBR began the practice of including marginsof error in the forecast table. It also warned that “on the

present occasion the large changes in the Budget itselfmay tend to increase margins of error”.



8.JRYHUQPHQWHFRQRPLFIRUHFDVWV«)6%5IRUHFDVWWDEOH



8.JRYHUQPHQWHFRQRPLFIRUHFDVWV«)6%5IRUHFDVWWDEOH



,QWHUSUHWLQJIRUHFDVWHUURUPDUJLQV

A forecast error margin is typically interpreted by

reporting an associated probability.

A distributional assumption is needed, and thenormal distribution is commonplace.

Studies of past forecast errors usually cannotreject the hypothesis that they are normallydistributed.



,QWHUSUHWLQJIRUHFDVWHUURUPDUJLQV«

Then some typical interval forecasts and their expectedcoverage probabilities are:

• point forecast MAE 57%

• point forecast RMSE 68%

• point forecast ± 2 RMSE 95%

• point forecast ± 0.675 RMSE 50% (interquartile range)

Given a time series of interval forecasts and outcomes, their

actual coverage can be checked against these nominalcoverage probabilities.

In the limit, a set of all possible intervals gives the complete

distribution, or density forecast, of which more below.



9DULDQWIRUHFDVWVIRUHFDVWVFHQDULRV

Further examples of variant forecasts and forecast

scenarios:

• US Congressional Budget Office presents baselinebudget projections, and variants that assume lower real

growth, higher interest rates, or higher inflation

• Bank of England has occasionally shown variants to the

central inflation projection based on alternativeassumptions preferred by individual MPC members, wrtbehaviour of exchange rate, scale of global economicslowdown, or degree of domestic spare capacity



Reserve Bank of New Zealand presents two stylisedscenarios representing key risks to the central projection:

9DULDQWIRUHFDVWVIRUHFDVWVFHQDULRV«

6FHQDULR

0RUHDEUXSW

HFRQRPLFVORZGRZQ

6FHQDULR

0RUH PRPHQWXP LQ

WKH HFRQRP\ DQG

VWURQJHU LQIODWLRQ

SUHVVXUH

No probability interpretations are offered by authors,nor can be inferred by users.




CPB Netherlands Bureau for Economic Policy Analysispublishes two scenarios for its medium-term outlook,

termed “favourable” and “unfavourable” with respect tothe exogenous (mostly international) assumptions.

There is no central projection




“The idea is that these scenarios show between which

margins economic growth in the Netherlands is likely to

lie, barring extreme conditions. There is no numerical

probability statement; rather the flavour is informal andsubjective, but coming from independent experts”

(Don, 2001)

• not an interval forecast, because “likely” is not translated

into a probability

• given a long run of scenarios and outcomes, an H[SRVW

probability could be estimated






For the fourth cabinet period, 2003-2006:

• “unfavourable/favourable” is replaced by “cautious/optimistic”

• the “safety margin” is reduced, to help improve the budget

policy process

• ?quantitative assessments?




• the standard error of SRWHQWLDO growth is reported, together

with the probability that potential growth will fall betweenthe two scenarios

• CPB says that the chance that DFWXDO growth falls outside

the two scenarios is considerably larger than in the twoprevious medium-term outlooks

• but still no probabilities are given for observed variables,to interpret the bands and for H[SRVW verification




'HQVLW\IRUHFDVWV

A density forecast is an estimate of the complete

probability distribution of the possible future values of thevariable of interest.

It gives a full description of the uncertainty associated witha forecast.

In decision theory, it is needed to evaluate the expected

loss whenever the loss function is anything other than aTXDGUDWLF function of the future state variable.



The earliest examples ineconomics come from theASA-NBER quarterly

survey, which began in1968, and which is nowknown as the Survey ofProfessional Forecasters

(SPF).

'HQVLW\IRUHFDVWV«



When the Panel of Independent Forecasters was establishedin the UK in late 1992, I suggested that the members beasked not only to provide point forecasts but also to answera similar probability question.

Such information eventually appeared, in the Panel’sNovember 1996 Report:

'HQVLW\IRUHFDVWV«



'HQVLW\IRUHFDVWV«

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

PM

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

BR

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

MW

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

KB

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

TC

0

10

20

30

40

50

60

0 1 2 3 4 5 6 7 8

GD



Meanwhile, in February 1996 the Bank of England andNIESR both began to publish density forecasts of inflation.

NIESR assumed a normal distribution, and presented ahistogram in tabular form:

'HQVLW\IRUHFDVWV«



The Bank’s fan chart allowed for some skewness over andabove the normal distribution, to reflect the balance ofrisks on either side of the central projection.

At this time the Bank also introduced probability questionsinto its Survey of External Forecasters.

'HQVLW\IRUHFDVWV«



A precursor to the Bank’s fan chart argued for theselective shading of quantiles to draw attention to the

uncertainty in forecasting:

'HQVLW\IRUHFDVWV«



The Bank’s fan chart consists of a set of forecast intervals

covering 10, 20, 30,…, 90% of the probability distribution

at each forecast horizon:

'HQVLW\IRUHFDVWV«



These are not percentiles, however, but the shortest intervals for thegiven probabilities, which centre on the mode. If the distribution isasymmetric, the probabilities in the upper and lower same-coloursegments are not equal, but their values are not reported. Likewise,the tail probabilities are unequal but not reported.

'HQVLW\IRUHFDVWV«



In an article in 1999 I argued that the Bank should followstandard practice in statistics and use percentiles to reportprobability distributions:

'HQVLW\IRUHFDVWV«



%DQNRI1RUZD\

%DQNRI)LQODQG

86&RQJUHVVLRQDO%XGJHW2IILFH

'HQVLW\IRUHFDVWV«PRUHH[DPSOHV



'HQVLW\IRUHFDVWHYDOXDWLRQ

First question: are the probabilities accurate, that is, do theoutcomes fall equally often in equal-probability bands?

In a 2004 article I showed that the Bank of England andNIESR had overestimated uncertainty in their current-

quarter and year-ahead inflation forecasts.Fewer outcomes fell in the tails of the distributions than theprobabilities had led users to expect.

This analysis has been revisited by Mitchell (1,(5 , July 05)and the Bank (,QIODWLRQ5HSRUW , August 05):



'HQVLW\IRUHFDVWHYDOXDWLRQ«



With 24 observations, we expect 12 outcomes in the interquartilerange and 6 in each of the tails if the fan chart probabilities arecorrect. For the current-quarter and year-ahead forecasts, there are15 outcomes inside and 9 outcomes outside the interquartile range:these intervals were too wide, and uncertainty was overestimated.


00.1

0.20.3

0.40.5

0.60.7

F e b - 9 8

A u g - 9 8

F e b - 9 9

A u g - 9 9

F e b - 0 0

A u g - 0 0

F e b - 0 1

A u g - 0 1

F e b - 0 2

A u g - 0 2

F e b - 0 3

A u g - 0 3

F e b - 0 4

current- uarter ear-ahead

)DQFKDUWVWDQGDUGGHYLDWLRQV




This is an example of the use of the probability integraltransformation of the outcome data.

For a density forecast I ([ ) with distribution function ) ([ ),the p.i.t. X of the observed outcome \is X = ) (\ ).

If the forecast density is correct, then X is distributeduniformly on (0,1).

For a sample of forecasts and outcomes, we can testgoodness-of-fit via Pearson’s chi-squared test or various

tests based on the sample cumulative distribution

function of X



Example: Bank of England Survey of External Forecasters,

survey average two-years-ahead inflation density forecasts,1998Q1-2003Q2 (Q=22)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

K-S statistic = 0.32 (5% critical value 0.28); ; 2(3) = 17.6



Current research topics include

• Joint tests of independence and goodness of fit

• Use of scoring rules and distance measures (KLIC, etc)

• Comparative testing of competing forecasts

• Combining density forecasts

Some 2005 references: Amisano & Giacomini; Bao, Lee &Saltoglu; Mitchell and Hall; Wallis




• Much progress has been made in recent years in

measuring and reporting forecast uncertainty.

• There is still reluctance in many quarters to adopt the

international language of uncertainty, namely

probability.• Appropriate techniques of forecast evaluation have

been developed.

• But many outstanding research problems remain, and

this is an active and fruitful area in which to work.

6XPPDU\UHPDUNV

Documents

Forecast Uncertainty