Prequential Analysis of Stock Market Returns · 2003. 2. 18. · David A. Bessler and Robert Ruffley Abstract The paper considers the Brier score and a covariance partition due to

May 2001

Prequential Analysis of Stock Market Returns

David A. Besslerand

Robert Ruffley

Abstract

The paper considers the Brier score and a covariance partition due to Yates to study theprobabilistic forecasts of a vector autoregression on stock market returns. Probabilistic forecastsfrom a model and data developed by Campbell (1991) are studied with ordinary least squares.Both calibration measures and the Brier score and its partition are used for model assessment.The partitions indicate that the ordinary least squares version of Campbell’s model does notforecast stock market returns particularly well. While the model offers honest probabilisticforecasts (they are well calibrated), the model shows little ability to sort events which occur intodifferent groups from events that do not occur. The Yates-partition demonstrates this shortcoming, while calibration metrics do not.

____________ Bessler is a Professor at Texas A&M University. Ruffley is a former graduate student inEconomics at Texas A&M University. Thanks are extended to John Y. Campbell for sharing hisdata with us. John L. King’s earlier association with Bessler is acknowledged as helpful for thispaper.

1

Prequential Analysis of Stock Market Returns

Given data which arrives in sequence, prequential analysis uses currently available data to

produce probability distributions on future observations (Dawid 1984) . Such a system is judged

on its forecasting ability and not on a priori grounds such as agreement with prior theory or

within sample goodness of fit. Dawid (1985) suggests that probability calibration be used to

judge the adequacy of probability forecasts.1 Kling (1987) applies prequential analysis to study

the distribution of turning points. Kling and Bessler (1989) use prequential analysis to model a

small macroeconomic vector autoregression. Covey and Bessler (1992) apply the technique to

test for Granger's full causality.2 These applications use calibration as the sole metric of

performance. Calibration is a test of whether an issued probability agrees with its relative

frequency, ex post. So, for example, if a prequential model issues a probability of .25 for one

hundred events, we should observe (ex post) that twenty five of these events occurred if this

model is to be labeled “well-calibrated.”

An alternative metric for evaluating probabilistic forecasts is the mean probability score,

otherwise known as the Brier Score (Brier 1950). The Brier score has received considerable use

in evaluating weather forecasts (see Murphy and Winkler 1977), but relatively little use in

economics. As one exception to this last statement we mention Zellner, Hong and Min (1991),

who use the Brier score to rank probability forecasts of turning points from various fixed and

time-varying-parameter models of aggregate output. The Brier score is a quadratic scoring rule

which has a rich history of use in motivation and evaluation of subjective probabilities, see

deFinetti (1937, 1965 and 1974) and Savage (1971) for theoretical developments on the

quadratic scoring rule and Nelson and Bessler (1989) for an empirical test of the optimal

2

property of this rule. We follow Kling and Bessler (1989) and Zellner, et al (1991) in suggesting

the use of optimal scoring rules for the evaluation of forecasts from econometric models.

One advantage of using a Brier score over calibration is that the Brier score can be

decomposed into components which index both calibration and resolution (sorting). In studying

financial data we may be interested in differences in probability forecasts assigned to events

(rates of return) that ultimately occur versus probabilities assigned to events that do not occur.

This “sorting” characteristic is not captured by calibration metrics. The Brier score thus gives

analysts more information (more than calibration measures) on the performance of a set

probability forecasts.3 Sanders (1963) provides one such partition. Murphy (1973) decomposes

Sanders’ resolution into an outcome variance index and an alternative measure of resolution.

Both the Sanders and Murphy decompositions work off of fixed probability vectors and thus

offer little, beyond usual calibration metrics, where forecasts are continuous (Kling and Bessler

(1989, pp.482-83)). Yates (1982) provides a covariance decomposition which applies to both

discrete and continuous probability forecasts. The Yates-partition is applied in this paper. We

are aware of no applications of the Yates-partition for evaluation of forecasts from financial or

econometric models.

This paper applies prequential analysis, using a standard calibration test and the Yates-

partition of the Brier score to two forecasting models of the U.S. stock market. As decisions

involving stock prices are inherently embedded in uncertainty, and as many if not most decision

theories require the entire probability distribution (e.g. expected utility theory), such methods

(not necessarily the one advocated here) are prima facie of interest. The models we consider

build on an earlier paper by Campbell (1991). First we study probability forecasts from an

ordinary least squares estimated version of Campbell's (1991) three variable vector

3

autoregression (we refer to this as the OLS-VAR) of stock market returns, the dividend price

ratio and short term interest rates. The second model entertained is a vector random walk in each

of the three variables from Campbell's model. The latter is of interest since it provides a

baseline set of probability forecasts which can be compared to the more substantive, knowledge-

based, forecasts from Campbell’s model. Our interest is not in assessing Campbell’s model in

particular, as the data are dated (some would say old) and not particularly relevant to up-to-date

or real time decision-making. Rather, our interest is to study probabilistic forecasts using the

Yates’ partition. By applying the results to Campbell’s model and his data, we provide readers

with a clear example of how such forecasts may be made and a clear indication of the type of

results they may expect to find in a well-designed econometric model (Campbell’s model).

An advantage of Campbell's model is that it incorporates the findings of Campbell (1987)

and Fama and Schwert (1977) that the level of short term interest rates helps forecast stock

market returns and the findings of Fama and French (1988) and Campbell and Shiller (1988) that

the dividend price ratio helps forecast stock market returns. We are particularly interested in

whether the additional information from the OLS-VAR results in improved forecasts of stock

market returns.

The outline of the paper is as follows. Section two provides greater detail on Campbell's

model and replicates his results. Section three discusses prequential data analysis, testing for

calibration and probability partitions (Brier Score and Yates-partition). Section four presents the

results. Section five concludes the paper.

Campbell's Model and Replication

Campbell's (1991) U.S. stock market model is a three variable, one lag, vector

autoregression consisting of real stock returns, the dividend price ratio and interest rates. The

4

data are measured monthly from 1926 thru 1988, with the first year being used for startup lags

for estimation. Following Campbell, the sample is broken into two subgroups, to take into

consideration the Federal Reserve Board-Treasury Accord. Prior to 1951, the Federal Reserve

Board held interest rates fairly constant. After 1951 the Board allowed rates to move more

freely. Thus, the model is run over three periods: (1) entire sample: January 1927 through

December 1988; (2) Pre-Treasury Accord period: January 1927 through December 1951; and

(3) Post-Treasury Accord period: January 1952 though December 1988.

The stock return series (h t) is the log of the real stock return over a month where the real

stock index is measured as the value weighted New York Stock Exchange Index for the CRSP

tapes deflated by the consumer price index. The interest rate series ( r t ) is the one-month

Treasury bill rate minus a one-year backward moving average. The dividend price ratio (d/p t ) is

the ratio of total dividend paid over the previous year divided by the current stock price.

Justification for including these variables in a stock market return model is given in

Campbell(1991). He also estimates a monthly VAR with six lags and a quarterly VAR. This

paper focuses on the one lag VAR on monthly data.

To ensure that the analysis in this paper is consistent with Campbell's analysis, we first

replicate his results, which are shown in table one of his paper (Campbell 1991, page 166).

Instead of using Generalized Method of Moments (GMM) this paper takes the route of using

ordinary least squares (OLS). The drawback of using OLS versus GMM is that the latter

produces a heteroskedastic consistent variance-covariance matrix, while OLS does not. The

reason we used OLS is that ordinary least squares provides an easy form for recursive

forecasting and coefficient updating using the Kalman filter. All of the estimation is carried-out

using RATS (Doan) software.

5

Table 1 contains the results produced by OLS. For the entire sample period and pre-

Treasury accord period, the coefficients are almost identical with Campbell's table 1. For the

post-Treasury accord period, the estimated coefficients differ slightly, but differ by no more than

.01 in any one case. However, we see considerable disagreement in the estimated standard

errors, with no consistent pattern of over or under estimation. Based on these results, the

findings of our paper do not necessarily reflect on Campbell's model; rather they reflect on our

OLS version of Campbell’s model.

Probability Forecasting

Let xt N = (x1t ,...., xmt), t=1,..., n be observed values of the mx1 vector time series Xt. At time n,

given known values xt , t = 1, . . . , n, a set of probability distributions Pn,k = (Pn+j ; j=1,..., k) for

unknown quantities xn+j , j=1,..., k are issued. A rule P which associates a choice Pn,k with each

value of n and any possible set of outcomes xt , t = n+1,..., n+k is a "prequential forecasting

system" (PFS) (Dawid, 1984). A prequential forecasting system is judged as "good" or "bad"

through the sequence of probabilities it actually issues and subsequent outcomes and not

through a priori considerations, such as agreement with theory or goodness of fit.

To judge the adequacy of prequential probabilities, Dawid (1985) proposed using

probability calibration. A PFS is said to be well-calibrated if the ex post relative frequency of all

events whose probability is P* is in fact P*. For example, a well-calibrated PFS should plot along

a 45 degree line with the relative frequency on the y-axis and issued probability on the x-axis.

If the xi,t+k are continuous random variables with continuous distribution functions Fi,t+k,

the random fractiles Ui,t+k = Fi,t+k (X i,t+k ), t=1..n, are independent uniform (U[0,1]) random

variables (Dawid, 1984). If the Xi,t+k are discrete with cumulative distribution functions Fi,t+k,

6

then the random fractiles Ui,t+k also have distribution functions of the form G(ui,t+k) = ui,t+k, though

the functions are not continuous. In either case the assessment of the PFS reduces to a test of

hypothesis that the observed sequence ui,t+k = F i,t+k (xi,t+k) is from a probability distribution with

the cumulative distribution G(ui,t+k)=ui,t+k. If this hypothesis cannot be rejected, then the PFS is

considered to be well-calibrated.

The estimated cumulative distribution function G(i,t+k) for Ui,t+k, is obtained by taking the

observed sequence ui,t+k = Fi,t+k (xi,t+k), t=1..n, ordering the sequence from low to high

ui,k(1),..,ui,k(n) and calculating

G ui,j(j) ' (j/n); j ' 1,..,n. (1)

The empirical cumulative distribution function is referred to as the "calibration function" (Bunn,

1984). For a well-calibrated PFS the calibration function should "look like" a 45-degree line.

A test of calibration can be made by testing the observed fractiles (ui's) from the sequence

of probability forecasts Pt,k. If there is a sequence of n such forecasts, then under the null

hypothesis (well calibration), any subinterval of length L (where 0# L# 1) will have n*L

observed fractiles. If there are J non-overlapping sub-intervals that exhaust the unit interval,

then a chi-squared goodness of fit statistic can be applied:

χ2 ' jJ

j'1

(aj & Ljn)2

Ljn- χ(J&1) . (2)

where aj is the actual number of observed fractiles in the interval j and Lj is the length interval

j. Under weak conditions, not requiring independence for the distributions underlying the

7

forecasts and under the null hypothesis of calibration, the test statistic will be distributed as chi-

squared with J-1 degrees of freedom (Dawid (1984)).

The Brier Score

Evaluation of a prequential system by way of its calibration property is not the only

possibility open to researchers (see the recent paper by Diebold, Hahn and Tay and the

prequential papers listed in our introduction for papers which focuses on calibration). The

probability score introduced by Brier (1950) is an alternative metric which has received

considerable attention in the literature. The Brier score considers calibration and resolution. As

resolution is a measure of a model's ability to "sort" or partition uncertain events into subgroups

which have probability measures that differ from long-run relative frequencies, it should prove

helpful in econometric applications.

Below we summarize known results on the Brier Score. We do so first for the case of

single event trials -- the event (say A) obtains or it does not obtain. We then present the

generalization to multiple event trials -- at each time t one of k possible events obtains. Our

source on this section is Yates (1988).

Let f represent the probabilistic forecast for an event that the forecaster is trying to

predict. Let d represent the outcome index where:

d = 1, if the event occurs

d = 0, if the event does not occur.

The Brier Score (1950) or probability score is then represented, for the single forecast case, as:

PS(f,d) ' (f & d)2. (3)

8

PS reaches a minimum value of 0 when the forecast is perfect (f=d=1). PS is maximum at 1 when either

the forecaster is absolutely certain that the event will occur when in fact it does not occur (f=1,d=0) or

the forecaster is certain that the event will not occur and, in fact, it does occur (f=0,d=1).

Over N occasions, indexed by i=1, . . . , n, the mean of PS is given by

PS(f,d) ' 1N j

N

i'1(fi & di)

2 . (4)

Sanders (1963) and Murphy (1972 and 1973) have decomposed PS into various components including

measures of calibration and resolution.

Yates (1982) further decomposed PS allowing for additional analysis. His formulation, called

"covariance decomposition" is given as:

PS(f,d) '

Var(d) % MinVar(f) % Scat(f) % Bias 2 & 2(Cov (5)

Var(d) represents the variance of the outcome index and defined as:

Var(d) ' d(1 & d) (6)

where

d '1N j

N

i'1di . (7)

Var(d) represents the factors of forecasting which are out of the forecasters control. That is, it represents

9

the base rate in which the target A occurs. The remaining terms in (5) reflect factors that are under the

forecaster's control. Thus, the forecaster wants to minimize MinVar(f), Scat(f) and Bias2 and maximize

Cov(f,d) to obtain the lowest PS.

The Bias is defined as

Bias ' f & d (8)

where

f ' 1N j

N

i'1fi . (9)

Bias is labeled "calibration in the large" or the mean probability judgement. It reflects the overall mis-

calibration of the forecast, i.e. how much the probability assessments are to high or too low. Thus, Bias2

is the amount of calibration error regardless of the direction of error.

The Cov(f,d) term is defined as

Cov(f,d) ' slope var(d) (10)

where Slope is defined as:

Slope ' f1 & f0 (11)

and

10

f1 '1N1

jN1

j'1f1j (12)

f0 '1N0

jN0

j'1f0j . (13)

f21 represents the conditional mean probability forecast for event A over the N1 occurrences for which

the event actually occurs. f2 0 represents the conditional mean probability forecast for event A over the

N0 occurrences that the event does not occur. The maximum value of Slope is 1 which occurs when the

forecaster always reports f=1 and the event does occur and f=0 and the event does not occur. Cov (f,d)

reflects the model's ability to make distinctions between individual occasions in which the event occurs

or does not occur. So, covariance is at the heart of the forecasting problem (Yates(1988)).

Scat(f) is given by:

Scat(f) ' 1N

N1Var(f1) % N0Var(f0) (14)

where

Var(f1) '1N1

jN

j'1(f1j & f1)

2 (15)

and

11

var(f0) '1N0

jN

j'1(f0j & f0)

2 (16)

Var(f1) is the conditional variance of the probability judgements for event A on those N1 times when A

actually occurs. Var(f0) is the conditional variance of the probability forecasts for event A on those N0

times when the event does not occur. Scat(f) is a weighted average of Var(f0) and Var(f1). It can be

interpreted as an index of overall scatter or noise contained in the forecaster’s probability statements.

MinVar(f) is the minimum forecast variance defined as:

MinVar(f) ' Var(f) & Scat(f) (17)

It represents the overall variance in the forecaster's probabilities if there were no scatter about the

conditional means f1 and f0. Thus, MinVar(f) measures how responsive the forecaster is to information

not related the event's occurrence.

The Brier score can also be formulated for a multiple event case. Let A1 , . . , A k represent a k-

event outcome space partition with K$2. Let dk represent the outcome index for each event k = 1, . . . ,

K. Let fk represent the probability forecast for each event k = 1, . . . ,K. In vector form d = (d1 , . . , dk)

and f = (f1 , . . , fk). The multiple event probability score (Murphy, 1972) is given

as:

PSM(f,d) ' (f & d)) (f & d) where 0 # PSM #2 (18)

Let i = 1, . . . , N index multiple-event forecasts fi and outcome indexes di over N different events, then

the mean of PSM is

12

PSM ' jK

k'1PSk (19)

where PSk represents the probability mean score for the kth event in the partition.

The covariance decomposition for the multiple event forecast is

PSM(f,d) ' jK

k'1Var(dk) % j

K

k'1MinVar(fk)

% jK

k'1Scat(fk) % j

K

k'1Bias 2

k

& 2jK

k'1Cov(fk,dk)

(20)

Each term in the multiple event case has an interpretation similar to that given in the single event case

discussed above.

Bootstrap Methodology

An OLS version of the 3-variable VAR model of Campbell (1990) and the 3-variable random

walk model were used to generate forecasts using the chain-rule of forecasting. Probability forecasts

from these models were generated using a bootstrap like procedure as outlined below.

The general VAR model is given as:

φ(B)tXt ' εt (21)

13

Here φ(B)t refers to the 3x3 autoregressive parameter matrix, whose elements are individually

polynomial functions of the lag operator B. The elements of φ(B)t are allowed to change over time

throughout the forecast interval, thus, they are indexed by t. Xt represents a 3x1 vector of stock prices,

dividend price ratios and interest rates observed in period t. The variable ε(t) represents a 3x1 vector of

residuals which are uncorrelated through time, but may be correlated in contemporaneous time.

Following a suggestion by Fair (1986), two sources of uncertainty are used to model the

probability distributions - uncertainty due to the lack of knowledge of φt and uncertainty in the one-

step-ahead forecasts (call this ut+1, a 3x1 vector).

At each date the elements of φ(B)t are assumed to normally distributed with mean φ(B)t and

covariance Vt=PtP't. Here φ(B)t and Vt are the estimated parameter and covariance matrices, found from

updating equation (21) with the Kalman filter at each date t. Uncertainty in φ(B)t is modeled by

making draws from the probability distribution used to describe φ(B)t. A particular draw φ(B)t is

obtained as

φ(B)(t ' φ(B)t % Pte (22)

where e is a vector of standard normal draws.

Uncertainty due the one-step-ahead forecast errors is modeled by drawing (call this draw u*t+1)

from the normal distribution with mean vector of zero and covariance matrix equal to the empirical

covariance matrix Σt on one-step-ahead forecast errors, ut+1. These latter errors are obtained from the

historical forecast performance on earlier data points. Thus, an initial period is required to obtain

estimates of Σt. To accomplish this, each sample is divided into three intervals. All three sample sets are

dealt with in a similar fashion. The first 24 observations in each sample are used to obtain OLS

estimates of φ(B) t, the next 48 observations are then used to simulate one-step-ahead forecasts. By

14

recursively forecasting Xt+1 and updating φ(B)t over this interval, we obtain a sample of 48 one-step-

ahead forecast errors from which we form our initial Σt. The remaining observations in each period are

used to model and evaluate the one-step-ahead probability forecasts.

The one-step-ahead forecast for Xt+1 is given as follows

X (

t%1 ' φ(B)(t Xt % u (

t%1 (23)

Repeating draws on e and ut+1 100 times at each date yields 100 point forecasts of Xt+1 at each time t in

the latter half of each period. The model is then moved forward one data point. The Kalman filter is

applied to obtain new estimates, φ(B)t+1 and Vt+1. In addition, the actual observed Xt+1 and the mean

forecasted Xt+1 are used to update Σt+1. Equation (23) is then reapplied 100 times in order to obtain the

forecast distribution for Xt+2. This procedure is repeated for each of the remaining data points in each

sample set.

Following each month's forecast, the actual outcome is compared to its forecasted distribution in

order to determine the observed fractile for that month.

Results

Table 2 summarizes the chi-squared statistic for the pre-Treasury Accord, post-Treasury Accord

and the entire sample period. At each date, the observed data point will determine the realized fractile by

where it falls under the cumulative distribution function for that date. By the probability integral

transform these should be uniformity distributed on (0,1). We break the interval (0,1) into 20 non-

overlapping and exhaustive classes and test uniformity of the realized fractiles using a chi-squared

"goodness of fit" test. Hence, there are 19 degrees of freedom and a critical chi-squared statistic value

of 30.114 at the 5 percent significance level.

In almost all cases, the random walk has a lower chi-squared statistic than does the OLS-VAR

15

for a one-step ahead forecast. For the pre-Treasury Accord period (1927-1951) and the entire sample

period (1927-1988), forecasts from both the OLS-VAR and the random walk model are rejected as

being well-calibrated. For the post-Treasury Accord period (1952-1988) the OLS-VAR is well-

calibrated for one, two, and three step ahead forecasts for all series. The random walk model is well-

calibrated for all series in the one-step and two-step forecast case; it is only well-calibrated for the

dividend price ratio in the three-step ahead case in the post-Treasury Accord period. Both models tend

to be less well-calibrated as the forecast horizon increases; although there are certainly exceptions to

this. The chi-squared goodness of fit tests indicate that the OLS-VAR model offers prequentially well-

calibrated forecasts only over the post-Treasury Accord period. The discrepancy between the pre and

post accord results is striking and indicative that the simple VAR is probably not capturing fundamental

pre-accord regularities.

Calibration plots for one-step ahead forecasts from the OLS-VAR and the random walk model

over the three time periods are given in Figures 1, 2 and 3. In each plot the horizontal axis is the issued

fractile and the vertical axis the, after-the-fact, relative frequency. So for a model to be well-calibrated

the fractile-relative frequency plot should be the 45 degree line. Figure 1 is for the post-accord period.

Note here that both the random walk and OLS-VAR plot close to the 45 degree line, with the OLS-VAR

closer to the 45 degree line for stock returns and interest rates over much of the issued fractiles. Figures

2 and 3 show much poorer calibration, considerable deviation form the 45 degree line for both models.

The calibration plots mirror quite well the chi-squared “goodness of fit” tests on one-step-ahead

forecasts presented in table 2; that is, both models are clearly not appropriate on pre-accord data.

Below we consider Brier Scores and their partitions on the same pre and post accord data for

both the OLS-VAR and a random walk forecast. Brier scores and covariance decomposition for the

entire sample, pre-Treasury accord and post-Treasury accord periods are contained in Tables 3 through

16

5, respectively. We have not considered the sampling distributions of these scores or their

decompositions, thus our results should be viewed as indicative and not definitive.

The row labeled "score" in Tables 3, 4 and 5 is the Brier score. Components which make up the

covariance decomposition are given beneath it. The columns labeled 1-step, 2-step and 3-step

represent, the Brier score or a component associated with one month ahead, two month ahead and 3

month ahead forecasts. The heading labeled "OLS-VAR” is the OLS version of Campbell's three

variable VAR model. The heading labeled random walk represents the results from a random walk

model.

Generally, the Brier scores increase as the forecast horizon increases (steps increases). This

holds for both the random walk and the OLS-VAR. The only case when this does not occur is for the

OLS-VAR forecasts of stock market returns using the entire sample period. Admittedly, the increase

may be very small as in the case of the stock market returns forecast in the post-Treasury accord sample

period where the one-step, two-step and three-step Brier scores for Campbell' model are .6314, .6339

and .6350, respectively. The increase may be large as in the case of the dividend price ratio forecast in

the post-Treasury accord sample period where the one-step, two-step and three-step Brier scores for the

OLS-VAR model are .1520, .2139 and .2698, respectively. The increase in the Brier scores indicates a

deterioration in both models’ abilities to forecast as the horizon increases. Not a surprising result. Of

course, since the Brier score is composed of various attributes, one of which is not under the model's

control, a more meaningful assessment may be through the individual score components.

Note that variability not under the forecaster's control (DVAR) decreases for the stock market

return and increases for the dividend price ratio and interest rate series. Note further that DVAR on

interest rates increases in post-accord data (contrast DVAR in tables 4 and 5 on interest rates).

This appears to agree with the prior notion that pegging interest rates in the pre-accord period made

them a much less uncertain series in the pre-accord period as compared with the post-accord period.

17

Stock market returns and the dividend price ratio appear to be easier to forecast over the post-accord

period (DVAR on both series is lower in table 5 than in table 4).

Across all horizons and for all variables forecasted, the conditional minimum forecast

(MinVarf), declines. As the number of forecast steps increases the amount of forecast variance that

must be tolerated for the forecaster to apply his expertise declines. Except for the dividend price ratio in

the post-accord period, scatter declines as the number of forecast steps increases for all periods and all

variables. This indicates that the noise or excess variability in the forecast declines as the number of

forecast periods is increased. Also, it shows that, at least for the one-step to three-step forecast, the

models are better at ignoring extraneous information than including important information in making a

forecast.

Except for the OLS-VAR forecast of stock returns in the post-Treasury accord period, bias

increases as the number of forecast steps increases. It is interesting to note that bias tends to increase

more for the random walk model than for the OLS-VAR model.

In general, covariance declines across all sample periods and forecast variables. The exceptions

are the stock market forecasts for the OLS-VAR model over the entire sample period, random walk over

the entire sample period and the OLS-VAR model over the pre-treasury accord period. A declining

covariance indicates that as the number of forecast steps increases, both models weaken in their ability

to respond to information related to forecasting the variable.

With respect to the entire sample period, the OLS-VAR model gives a higher (poorer) Brier

score for the one-step, two-step and three-step forecasts of the dividend price ratio and the one-step

ahead forecast of the interest rate. For the post-Treasury Accord period, the OLS-VAR model gives a

higher Brier score only for the one-step ahead interest rate forecast. For the pre-Treasury Accord

period, the OLS-VAR model gives a higher Brier score as in the entire sample period. It appears as

though the OLS-VAR model is not as good at forecasting the dividend price ratio as a random walk

18

model. This may be due to the fact that the forecasts are made on a monthly basis, while the time-series

is constructed by using total dividends paid over the previous year divided by the current stock price.

Although the Brier score gives an overall indication of the forecaster's ability, the components

of the covariance decomposition provides a clearer indication of the forecaster's ability to forecast. For

all sample periods and all variables, MinVarf is either almost zero or the estimates given by the OLS-

VAR model are less than the ones given by the random walk models. Recall that MinVarf is the

conditional minimum forecast variance given the covariance of forecasts and outcome index. It

represents the minimum amount of forecast variance that must be tolerated given that the forecaster

applies his fundamental forecasting abilities. The OLS-VAR model has a superior characteristic over

the random walk model in that it requires a lower minimum forecast variance.

In only six cases, does the OLS-VAR give a higher bias. None of these instances occur in the

stock market forecast, whereas five occur in the dividend price ratio forecast. Thus, the OLS-VAR

appears to be able to match mean forecasts to relative frequencies better than the random walk model.

Over the entire period, for dividend price ratio forecasts and interest rate forecasts, the OLS-VAR has

lower covariance than the random walk model. But for the stock market forecasts, the OLS-VAR

results in higher covariance in seven out of nine cases. Thus, for the interest rate series and the dividend

price ratios, the random walk model exhibits more forecasting skill, while the OLS-VAR shows more

skill in forecasting stock prices.

In all sample periods and all forecasts, the OLS-VAR gives a better (lower) measure of scatter.

It is superior to the random walk model by being less responsive to information not related to

forecasting for either of the three time series. Stated alternatively, the forecasts from the OLS-VAR

contain less noise. Note that since the forecast variance is the sum of MinVarf and scatter and that

Campbell's model in general had both a lower scatter and Minvarf, then it is also true that the OLS-

VAR has a lower forecast variance than the random walk model.

19

Overall, the OLS-VAR model outperforms the random walk in, scatter and MinVarf. But it is

outperformed in terms of covariance, except in forecasting the stock market. One issue which arises is

why does the OLS-VAR have better scatter, but offers worse covariance estimates. This is possibly due

to the types of models being compared. The VAR model acts as a filter, trying to remove unimportant

information while letting important information pass through. A random walk model does not

discriminate and allows all information to pass through, whether it is relevant or not. Since scatter

represents how responsive the forecaster is to information unrelated to the event, while covariance

represents how responsive the forecaster is to information related to the event, it is possible that the

OLS-VAR does well in screening out irrelevant information, but at some expense of screening out

relevant information for interest rate and dividend price ratio forecasts.

In the case of forecasting the stock market, the OLS-VAR does comparatively well in screening out

irrelevant information, while still incorporating relevant information. Actually, Campbell’s motivation

for his model appears to be forecasting stock market returns, first and foremost, and not so much

forecasting dividend price ratios and interest rates.

The covariance measure reflects differences between average probabilities assigned to events

(fractiles) that occurred and average probabilities associated with fractiles that did not occur. We want

such differences to be large. In table 6 we list these differences. Notice that these differences are tiny

for stock market returns at every horizon, quite large at every horizon for the dividend price ratio and

moderate for interest rates. To the extent that these differences capture the essence of forecasting, we

have to conclude that our ability to forecast stock market returns with this model is not very good.

One point which we wanted to address in this paper is whether the Brier score and its partition is

able to provide additional information with respect to forecast performance relative to the chi-squared

statistics on calibration. In table 7, we summarize performance measures using zero-one indicators for

each series, time period, and forecast horizon. A zero (0) indicates that the random walk forecast

20

outperformed the OLS-VAR for that measure on that particular series, time interval and horizon. A one

(1) indicates that the OLS-VAR outperformed the random walk model.

From Table 7 notice that there is no clear dominance of the OLS-VAR over the random walk

using the chi-squared statistics. Of the 27 cases studied (three variables, three horizons and three time

periods), OLS dominates in 11 cases and is dominated by the random walk in 16. Notice, however, that

in terms of Brier score components, the OLS-VAR-forecasts do have some dominating characteristics.

In particular, scatter and minimum variance components of the OLS-VAR forecasts of the dividend

price ratio (d/p) and interest rate (r) dominate the random walk forecasts across all three time periods

and almost all forecast horizons. Further, in seven of the nine cases, the OLS-VAR forecasts dominate

the random walk forecasts in terms of covariance. Just the opposite is true for dividend price ratio and

interest rate forecasts in terms of the covariance metric -- the random walk forecasts are superior in

terms of the covariance component of the Brier score. Recall, that covariance is at the heart of the

forecasting effort (the difference in average probability assigned to events which ultimately occur versus

average probabilities of events which ultimately do not occur). The covariance metric appears to tell us

that the OLS-VAR does not offer much in sorting relative to the random walk forecast for two of the

three series. Campbell’s’s model was setup (primarily) to forecast stock market returns (h) and not

interest rates (r) or necessarily dividend price ratios(d/p). So perhaps these results are not unreasonable.

However, as noted in table 6, the OLS-VAR does not offer large differences in probabilities between

events which occur and events which do not occur. Nevertheless, it does better than the random walk

model. This last result and interpretations holds for the overall Brier score results as well; the OLS-VAR

performs well over all three time periods and all three horizons in forecasting stock market returns

(relative to the random walk); while its does not perform particularly well in any of the forecasts of the

dividend price ratio and offers mixed results in forecasting interest rates. The OLS-VAR generally gets

good marks (relative to the random walk) in terms of scatter, minimum variance and bias.

21

Discussion

This paper applies prequential analysis to two models of the US stock market. Probability

calibration, mean probability scores (Brier scores) and their partitions are considered. A three variable

VAR, introduced earlier by Campbell, consisting of real stock returns, short term interest rates and

dividend price ratio, is fit to 1927-1988 monthly data using ordinary least squares. Probability forecasts

from an ordinary least squares version of this model (OLS-VAR) are compared to probability forecasts

from a random walk. This paper finds that the OLS-VAR and the random walk model are not well

calibrated for the pre-Treasury Accord period, (before 1952). For the post-Treasury-Accord sample

period (1952-1988), the random walk model is slightly better calibrated for the one-step ahead forecast,

but the OLS-VAR is better calibrated for the two and three-step ahead forecasts. The OLS-VAR tends

to have lower Brier scores than the random walk model for all three series and forecast horizons.

The Yates-partition of the Brier score indicates that the OLS-VAR produces a lower minimum

forecast variance, bias and scatter. But it gives a smaller covariance, except for forecasting stock market

returns. Thus, OLS-VAR is better at screening out information not relevant to issuing forecasts.

However, this improvement comes at the a cost of failing to incorporate some relevant information. The

exception to this is forecasting stock market returns. Here the OLS-VAR outperforms the random walk.

Generally, we would like a model to offer well-calibrated forecasts and sort events into groups

where the assessed probability of events which obtain approach one and probabilities on events which

do not obtain approach zero. Tests of calibration and the bias component of the Yates-partition measure

well-calibration. The covariance component of the Yates-partition measures a model’s ability at sorting.

Our OLS version of Campbell’s VAR does result in well-calibrated probability forecasts over the post-

Treasury accord period. It does not result in large differences in probabilities of stock market return

events which occur versus stock market return events which do not occur. Thus, we do not offer our

model as particularly helpful in forecasting stock market returns. Our model is “honest”, as it reports

22

well-calibrated forecasts over the post-accord period; yet it is not particularly good, as it is not able to

offer a large difference in probabilities between events which ultimately obtain versus events which do

not obtain (for stock market returns). This last point is a strong reason why one might want to use the

additional insights offered by the Brier score and its Yates-partition, rather than confining his/her study

to just the calibration properties of a particular set of probability forecasts. We suggest that recent

papers (Kling and Bessler (1989) for example) which focus on calibration ought to be re-evaluated

under the more general Brier score and its probability partition.

Further research on probability forecasting is certainly warranted. In particular, this model could

be studied using non-normal draws in calculating probability forecasts. As stock market returns are

generally characterized by “fat tailed” distributions, this suggestion is worth consideration. However,

even under alternative error assumptions, the resulting model should be judged according to its

prequential performance--by the sequence of probabilities it issues and subsequent realizations. Further,

work could be done on the sampling distribution of both the Brier score and its covariance partitions.

Here we made statements about the “goodness” of a set of forecasts based on the Brier score and its

Yates-partition. This is consistent with deFinetti’s use of scoring rules as a metric of performance in

probability forecasting (see deFinetti (1965)). One could go further with this idea and make

probabilistic statements on the hypothesis that the difference between two Brier scores (say one from

model A the other from model B) is equal to zero. 4 By extension one might consider, as well,

distributions on the differences between each of the five components in the Yates partition. We leave

these topics for future research.

23

Footnotes

1. For a model to be well-calibrated, events that are assigned a probability of n percent should occur in

ex post assessment with n percent relative frequency. Calibration acts as a long-run assessment of a

model's ability to issue realistic probability forecasts.

2. Actually there are several applications of prequential analysis in the more general literature. Dawid

(1986) reviews this literature under the heading of probability forecasting. A recent application which is

not labeled “prequential analysis”, but is none the less the same, Diebold, et al. 1999 study calibration

properties of probability forecasts of foreign exchange.

3. Resolution refers to the ability of a model to sort individual outcomes into groups which differ from

the long run relative frequency.

4. A point deFinetti probably would not agree with as he was a strict subjectivist, having no clear

attraction to hypothesis testing in the usual sense of the word; see deFinetti (1974).

24

Table 1. Ordinary Least Squares (OLS) VAR Estimates versus Campbell’s Generalized Method of Moments(GMM)Estimates.

_________________________________________________________________________

right-hand side variablesdep. var. ht-1 d/pt-1 rt-1 R2

OLS GMM OLS GMM OLS GMM OLS GMM

(1927 - 1988)

ht .107 .107 .331 .331 -.423 -.424 .024 .024

(.037) (.063) (.155) (.183) (.201) (.195)

d/pt -.007 -.007 .963 .963 .018 .018 .936 .937

(.002) (.005) (.009) (.028) (.012) (.010)

rt .007 .007 -.039 -.040 .669 .669 .450 .450

(.005) (.005) (.021) (.010) (.028) (.061)

(1927 - 1951)

ht .142 .142 .482 .483 .926 .926 .028 .028

(.059) (.091) (.281) (.466) (.643) (.712)

d/pt -.012 -.012 .934 .935 -.033 -.033 .901 .901

(.004) (.007) (.019) (.045) (.043) (.041)

rt .005 .005 -.019 -.019 .308 .309 .092 .101

(.005) (.006) (.024) (.026) (.056) (.161)

(1952 - 1988)

ht .048 .048 .500 .490 -.723 -.724 .065 .065

(.047) (.060) (.227) (.227) (.163) (.192)

d/pt -.001 -.001 .978 .980 .034 .034 .959 .959

(.020) (.003) (.009) (.011) (.007) (.009)

rt .013 .013 -.105 -.017 .739 .739 .548 .547

(.009) (.012) (.046) (.058) (.033) (.009)______________________Here ht refers to real stock market returns in period t; d/pt refers to the dividend price ratio in period t; rt the real interestrate in period t. The numbers in parentheses are standard errors. These variables are defined and discussed in Campbell(1991).

25

Table 2. Chi-Squared “Goodness of Fit” Tests on OLS-VAR and Random Walk Probability Forecastson Horizons of 1,2, and 3 Steps Ahead.________________________________________________________________________________________________________________________________________________________________________

Chi-Squared Statistics1

ForecastedVariable

OLS-VAR Random Walk

1step 2step 3step 1step 2step 3step

(1927-88)

ht 197.85 232.23 229.61 169.58 425.05 563.87

d/pt 486.79 544.34 490.66 482.78 451.29 437.58

rt 206.40 215.29 213.78 144.24 205.12 371.44

(1927-51)

ht 106.17 106.68 110.74 75.81 184.26 229.20

d/pt 156.46 151.72 141.46 123.75 117.42 94.04

rt 479.55 383.28 495.65 443.39 485.23 551.67

(1952-88)

ht 11.00 15.71 13.38 6.42 42.49 92.16

d/pt 15.59 15.80 21.95 3.72 14.70 26.44

rt 20.16 22.84 20.66 14.72 19.51 43.30

_______________1. Five per cent critical value is 30.11

Score is the Brier score, lower values of which suggest “better” performance. The Yates-decomposition is given by the five numbers below score in each column: Score = DVAR + Minvar + Scatter + Bias - 2 Cov

26

Table 3. Brier Scores and Decompositions of OLS-VAR and Random Walk Forecasts, 1928 - 1988.1________________________________________________________________________________________________________________________________________________________________________

OLS-VAR Random Walk

1 step 2 step 3 step 1 step 2 step 3 step

(stock market returns)

score .7162 .7050 .7145 .7787 .8257 .8447

Dvar .6502 .6491 .6488 .6502 .6491 .6488

Minvar .0000 .0001 .0000 .0002 .0000 .0000

Scatter .0155 .0148 .0154 .0277 .0170 .0134

Bias .0556 .0540 .0546 .1098 .1588 .1604

2Cov .0051 .0130 .0045 .0092 .0009 .0021

(Dividend price ratio)

Score .2663 .3787 .4539 .2509 .3508 .4296

Dvar .7035 .7037 .7039 .7035 .7037 .7039

Minvar .1908 .0962 .0569 .2093 .1198 .0743

Scatter .1022 .0929 .0827 .1025 .1013 .0965

Bias .0003 .0020 .0046 .0004 .0024 .0054

2Cov .7304 .5161 .3942 .7648 .5764 .4505

(Interest rates)

Score .5739 .6145 .6421 .5608 .6307 .6664

Dvar .6569 .6569 .6578 .6569 .6569 .6578

Minvar .0110 .0003 .0009 .0279 .0095 .0047

Scatter .0691 .0365 .0251 .1359 .0876 .0623

Bias .0020 .0029 .0027 .0067 .0255 .0457

2Cov .1651 .0847 .0443 .2665 .1488 .1042

1.Score is the Brier Score, lower values of which suggest “better” performance. The Yates-decomposition is given by the five numbers below score in each column: Score = DVAR + Minvar +Scatter + Bias - 2 Cov.

27


OLS-VAR Random Walk



score .7588 .7602 .7659 .8254 .8555 .8617

Dvar .6881 .6845 .6845 .6881 .6845 .6845

Minvar .0002 .0001 .0001 .0001 .0000 .0001

Scatter .0122 .0114 .0121 .0205 .0132 .0097

Bias .0749 .0757 .0773 .1193 .1566 .1754

2Cov .0166 .0115 .0081 .0027 - .0012 .0081


Score .3459 .4947 .5836 .3370 .4718 .5475

Dvar .7028 .7046 .7063 .7028 .7046 .7063

Minvar .1324 .0463 .0192 .1455 .0633 .0034

Scatter .0984 .0735 .0580 .1078 .0963 .0824

Bias .0151 .0235 .0269 .0139 .0230 .0281

2Cov .6028 .3531 .2267 .6630 .4154 .3027

(Interest rates)

Score .5348 .5491 .5553 .4987 .5712 .6319

Dvar .4988 .4940 .4940 .4988 .4940 .4940

Minvar .0005 .0005 .0004 .0028 .0009 .0004

Scatter .0183 .0165 .0175 .0299 .0214 .0187

Bias .0351 .0473 .0525 .0308 .0833 .1323

2Cov .0178 .0092 .0091 .0636 .0284 .0134

1.Score is the Brier Score, lower values of which suggest “better” performance. The Yatesdecomposition is given by the five numbers below score in each column: Score = DVAR + Minvar +Scatter + Bias - 2 Cov

28


OLS-VAR Random Walk



score .6314 .6339 .6350 .7301 .7411 .7640

Dvar .6307 .6310 .6307 .6307 .6310 .6307

Minvar .0002 .0000 .0000 .0006 .0002 .0000

Scatter .0190 .0125 .0108 .0746 .0387 .0262

Bias .0028 .0029 .0032 .0428 .0811 .1121

2Cov .0212 .0127 .0097 .0186 .0099 .0050


Score .1520 .2193 .2698 .1474 .2281 .2834

Dvar .6471 .6471 .6471 .6471 .6471 .6471

Minvar .3628 .2720 .2143 .3810 .2819 .2253

Scatter .1064 .1285 .1382 .1101 .1483 .1674

Bias .0016 .0025 .0031 .0003 .0003 .0002

2Cov .9659 .8308 .7331 .9911 .8496 .7566

(Interest rates)

Score .6060 .6708 .7207 .6257 .6998 .7184

Dvar .7319 .7334 .7349 .7319 .7334 .7349

Minvar .0293 .0083 .0022 .0341 .0108 .0061

Scatter .1293 .0760 .0547 .1617 .1078 .0788

Bias .0032 .0048 .0060 .0076 .0160 .0249

2Cov .2876 .1517 .0771 .3095 .1682 .1263

29

Table 6. Average Probabilities Assigned to Events which Occur minus Average Probabilities Assignedto Events which do not Occur, by series (h, d/p, r), forecast horizon (t+1, t+2, t+3) and time period, forOLS-VAR and Random Walk.________________________________________________________________________________________________________________________________________________________________________

1927 - 1988 1921 - 1951 1952 - 1988

series OLS RW OLS RW OLS RW

h t+1 .0039 .0071 .0121 .0020 .0168 .0147

h t+2 .0100 .0007 .0084 -.0009 .0101 .0078

h t+3 .0035 .0016 .0059 .0059 .0077 .0040

d/p t+1 .5217 .5436 .4289 .4717 .7463 .7658

d/p t+2 .3667 .4095 .2500 .2948 .6419 .6565

d/p t+3 .2880 .3200 .1605 .2143 .5665 .5846

r t+1 .1257 .2028 .0178 .0638 .1965 .2114

r t+2 .0645 .1133 .0093 .0287 .1034 .1147

r t+3 .0337 .0792 .0092 .0136 .0525 .0859____________Numbers in this table are derivable from those contained in Tables 3,4, or 5 and the formula: slope =cov/dvar, where cov and dvar are defined as in tables 3,4, and 5. The slope is the difference in averageprobabilities associated with events which obtain versus the average probabilities associated with eventswhich do not obtain. It can be found statistically as the “slope” found by regressing probabilityjudgements on outcome indexes (see Yates (1988, page 284) for further discussion).

30

Table 7. Indicators of dominance: VAR(1) versus the Random Walk (0) for chi-squared statistics, Brierscores, and its components.________________________________________________________________________________________________________________________________________________________________________

series chi sq Br Sc Covar Sct Mvar Bias

steps ahead

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

- - - - - - - - - - - - - - - - - -

h (27-88) 0 1 1 1 1 1 0 1 1 1 1 0 1 0 - 1 1 1

h (27-51) 0 1 1 1 1 1 1 1 - 1 1 0 1 0 - 1 1 1

h (52-88) 0 1 1 1 1 1 1 1 1 1 1 1 1 1 - 1 1 1

d/p (27-88) 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1

d/p (27-51) 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1

d/p (52-88) 0 0 1 0 1 1 0 0 0 1 1 1 1 1 1 0 0 0

r (27-88) 0 0 1 0 1 1 0 0 0 1 1 1 1 1 1 1 1 1

r (27-51) 0 1 1 0 1 1 0 0 0 1 1 1 1 1 - 0 1 1

r (52-88) 0 0 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1

_________________A one (1) indicates the VAR outperforms the random walk on the particular measure for forecasts at thehorizon listed at the head of the column. A zero indicates the random walk outperformed the VAR onthe measure. A "-" indicates the two models have the same score (the models tied with respect to thatmeasure).The variables h, d/p, and r refer to stock market returns, dividend - price ratio and interestrates, respectively.

31

0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles

Campbells Model Random Walk Model

0

0.2

0.4

0.6

0.8

1 R

elat

ive

Fre

quen

cy

0 0.2 0.4 0.6 0.8 1 Fractiles


0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


Dividend Price Ratio 1952-1988.

Stock Returns 1952-1988.

Interest Rates 1952-1988

Figure 1. Calibration Plots on Probabilistic Forecasts of Dividend Price ratio, Stock Market Returns andInterest Rates, 1952 - 1988 Data, by model -- OLS-VAR (Campbell’s model) and Random Walk.

32

0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


Dividend Price Ratio 1926-1951

Stock Returns 1926-1951



33

0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


0

0.2

0.4

0.6

0.8

1

Rel

ativ

e F

requ

ency

0 0.2 0.4 0.6 0.8 1 Fractiles


Dividend Price Ratio 1926-1988

Stock Returns 1926-1988



34

References

Brier, Glen W., "Verification of forecasts expressed in terms of probability," Monthly Weather Review,78(1950): 1-3.

Bunn, Derek W. Applied Decision Analysis. New York: McGraw-Hill.

Campbell, John Y., "A variance decomposition for stock returns," Economic Journal, 101(1991): 157-179.

Campbell, John Y., "Stock returns and the term structure," Journal of Financial Economics, 18(1987):373-399.

Campbell, John Y., and Robert Shiller, "The dividend-price ratio and expectations of future dividendsand discount factors," Review of Financial Studies, 1(1988): 195-228.

Covey, Ted, and David A. Bessler, "Testing for Granger's full causality," Review of Economics andStatistics, 74(1992): 146-153.

Dawid, A.P., "Statistical theory: A prequential approach," Journal of Royal Statisticical Society,147(1984): 278-297.

Dawid, A.P., "Probability forecasting," in Encyclopedia of Statistical Sciences, Vol. 7. New York:Wiley.

deFinetti, B. “La Prevision: ses lois logiques, ses sources subjectives” Annales de l’Institute HeniPoincare, 7(1937):1 -67.

deFinetti, B. “Methods for Discriminating Levels of Partial Knowledge Concerning a Test Item,” BritishJournal of Mathematical and Statistical Psychology 18(1965, part I):87-123.

deFinetti, Bruno. Theory of Probability, New York: Wiley 1974.

Diebold, F., J. Han and A. Tay. “Multivariate Density Forecast Evaluation and Calibration in FinancialRisk Management: High-Frequency Returns on Foreign Exchange,” Review of Economic and Statistics81(1999):661-673.

Dow, Charles, H., "Scientific stock speculation", The Magazine of Wall Street, (New York), 1920.

Fair, Ray, C., "Evaluating the predicative accuracy of models," in Z. Griliches and M.D. Intriligator(eds.) Handbook of Econometrics Vol. III(New York: Elsevier Science Publishing Co., 1986), 1980-1995.

Fama, Eugene, "Efficient capital markets: A review of theory and empirical work," Journal of Finance,25(1970): 383-417.

Fama, Eugene, F. and Kenneth R. French, "Dividend yields and expected stock returns," Journal ofFinancial Economics, 22(1988): 3-25.

35

Fama, Eugene, F. and Kenneth R. French, "Business conditions and expected returns on stocks andbonds," Journal of Financial Economics, 22(1989): 23-49.

Fama, Eugene, F. and Robert G. Schwert, "Assert returns and inflation," Journal of FinancialEconomics, 5(1977): 115-46.

Hanson, Lars, Peter, "Large sample properties of generalized method of moments estimators,"Econometrica, 50(1982): 1029-1054.

Kim, Myung Jig, Charles Nelson, and Robert Startz, "Mean reversion in stock prices? A reappraisal ofthe empirical evidence," Review of Economic Studies, 58(1991): 515-528.

Kling, John, L., "Predicting the turning points of business and economic time series," Journal ofBusiness, 60(1987): 201-238.

Kling, John, L., and David A. Bessler, "Calibration based prediction distributions: An application ofprequential analysis to interest rates, money and prices," Journal of Business, 62(1989): 477-499.

Murphy, A.H., "A new vector partition of the probability score," Journal of Applied Meteorology,12(1973): 595-600.

Nelson, R. G. and D. A. Bessler. 1989. "Subjective Probabilities Elicited Under Proper and ImproperScoring Rules: A Laboratory Test of Predicted Responses." American Journal of AgriculturalEconomics 71:363-369.

Sanders, F., "On subjective probability forecasting," Journal of Meteorology, 2(1963): 191-201.

Savage, L. “Elicitation of Personal Probabilities and Expectations,” Journal of the American StatisticalAssociation 66(1971):783-801.

Yates, Frank, "Analyzing the accuracy of probability judgements for multiple events: An extension ofthe covariance decomposition," Organizational Behavior and Human Decision Processes, 41 (1988):281-299.

Zellner, A., Chansik Hong and Chung-Ki Min, "Forecasting turning points in international outputgrowth rates using Bayesian exponentially weighted autoregression, time-varying parameter, and pollingtechniques," Journal of Econometrics, 49 (1991): 275-304.

36

Documents

Prequential Analysis of Stock Market Returns · 2003. 2. 18. · David A. Bessler and Robert Ruffley Abstract The paper considers the Brier score and a covariance partition due to