Forecasting Elections from Voters’ Perceptions of Candidates’ Ability to Handle Issues

A Test of the Index Method

Andreas Graefe, Karlsruhe Institute of Technology

J. Scott Armstrong, Wharton School, University of Pennsylvania

The full paper to this talk can be downloaded at: tinyurl.com/issueindex

Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective

November 21, 2010

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Evolution of election forecasting

1978: Economist Ray Fair publishes regression model that focuses on economic growth and inflation as predictor variables.

[Fair, 1978]

Over the next three decades, others would follow with models that use slightly different variables.

Status-quo in election forecasting

In a review of 14 quantitative models by economists and political scientists

All were regression models12 used a measure of the state of the economy 7 used a measure of the incumbent’s popularity5 used both

[Jones & Cuzan, 2008]

Common view in election forecasting

Most models are economic vote models.

On average, these models perform well.

Widely-held view that a presidential election is a referendum on- the incumbent president’s popularity or- his ability to handle the economy

Presidential campaigns, individual differences among candidates, and parties are assumed to have no – or only little – impact on the election outcome.

But what about the candidates?

Candidates play a vital role in election campaigns and are extensively discussed in the media, e.g. their

- Biography (experience)- Personality- Stands on the issues- Endorsed policies

Yet, no existing model uses such information.

Models are unable to aid decision-making in campaigns.

Research with decision-making implications

Can better forecasting help to advise…

A candidate’s decision on whether to run for office?

A party’s decision about who to nominate?

Decisions as to what issues a candidate should emphasize in a campaign?

Decisions as to which policies to endorse?

This talk

Why do all existing models ignore individual differences between candidates?

Insufficient use of forecasting methods?

Multiple regression

Almost all existing forecasting models use multiple regression.

Multiple regression is useful to estimate the relative impact of certain variables on the outcome variable for a given data set, i.e. data fitting.

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Limitations of multiple regression

Multiple regression is limited for predicting new data.

Variable weights are typically estimated from the dataset.

- Limited in the number of variables- Limited ability to incorporate prior domain

knowledge- Extracts too much information (i.e. noise) from

given data, which does not generalize when predicting new data

Conditions for multiple regression

The performance of multiple regression depends to a large extent on the ratio between predictor variables and the number of observations available – especially when using non-experimental data.

[Einhorn & Hogarth, 1975]

Regression should not be used for social science predictions unless sample size is larger than 100 observations per predictor variable.

[Dana & Dawes, 2004]

Conditions for forecasting U.S. Presidential Elections

Data for the majority of regression models is limited to about 25 elections. Many models use - no more than 15 observations and- from two to five predictor variables

Multiple regression is limited for forecasting U.S. Presidential Elections.

In particular, if one wants to incorporate individual differences between candidates: too many variables.

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Index method

Structured approach for summarizing domain knowledge (e.g., prior research, expert knowledge).

Long history in forecasting and decision-making.

Index method procedure

(1) Identify variables from domain knowledge (i.e. prior empirical studies and/or subjective judgment by experts)

(2) Use prior evidence to determine their directional influence on the outcome (e.g. 1: favorable; 0: unfavorable)

(3) Sum up scores. Weight variables equally unless strong evidence exists for differential weighting.

(4) Select the option that is favored by most variables.

Early use of the index method

Predicting the success of paroling individuals from prison.[Burgess, 1939]

Based on a list of 25 factors, an index score was calculated for each individual to determine the chance of successful parole.

This method was criticized as it did not consider- the relative importance of different variables or- how favorable the ratings were for a certain variable

However, a follow-up study did not find evidence that supported the use of regression over index scores for predicting parole.

[Gough, 1962]

Relative performance of the index method and multiple regression: Theoretical evidence

The index method will outperform regression if

- sample size is small and - the number of – and inter-correlation among –

predictor variables high.

[Einhorn & Hogarth, 1975]

Relative performance of the index method and multiple regression: Empirical evidence

In a summary of 8 studies, 5 favored the index method over regression.

[Armstrong, 1985]

Unit-weighting showed higher out-of-sample accuracy than multiple regression for 20 prediction problems taken from statistical textbooks.

[Czerlinski et al., 1999]

“Equal-weights” versions of three established election forecasting models yielded a lower MAE and higher hit-rate than the original econometric models for the 32 elections from 1880 to 2004.

[Cuzán & Bundrick, 2009]

Advantages of Index models

No limit in the number of variables (regression usually limited to 3 or 4 variables).

No need to estimate weights from the data and thus- No sample necessary for building the model.- Easy to incorporate new variables in the model

Index models can utilize all prior knowledge.

Disadvantages of Index models

Expensive to summarize prior knowledge.

Difficult to account for effect size (coefficients and the amount of change in predictor variables).

Difficult to base model on theory

The first index model for forecasting elections

Lichtman’s index model, the 13 Keys to the White House, has been published for years. He made forecasts of the past 38 presidential elections (7 prospectively).

[Lichtman, 2008]

In all cases, the model’s predictions of the popular vote winner have been correct. No other approach has come close to this record.

From 1984 to 2004, Lichtman’s “Keys” yielded forecast errors of the popular vote shares almost as low as three established econometric models.

[Armstrong & Cuzán, 2006]

The bio-index model

Predicts U.S. presidential election winners based on 59 variables that incorporate biographical information about candidates.

Examples: height, weight, birth order, married, beauty

Correctly predicted the winner in 27 of the 29 elections from 1896 to 2008 and thereby outperformed polls, prediction markets, and econometric models.

[Armstrong & Graefe, 2010]

Conditions favoring the index method

Many variables

Few observations

Much prior domain knowledge (based on expertise and empirical studies)

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Prediction problem

Forecast U.S. presidential election outcome from information about how voters’ perceive candidates to handle issues

Conditions:1. Large number of issues / variables (sometimes more than 40)2. Issues change or new issues arise over time

(e.g., global warming, wars, financial crisis)3. Few observations

Conditions favor the index method

Data

Polls which asked voters to name the candidate who would be more successful in solving a specific issue.

“Now I'm going to mention a few issues and for each one, please tell me if you think Barack Obama or John McCain would better handle that issue if they were elected president...the economy”

[CNN/Opinion Research Corporation Poll. July 27-29, 2008]

Examples for issues: terrorism, war in Iraq, economy, jobs, budget deficit, health care, immigration

Obtained data: 427 polls with voters‘ opinion on 314 issues for the 10 elections from 1972 to 2008

Coding and generation of the issue index

For each issue, candidate with higher voter support achieved a score of 1. Otherwise, 0.

The issue-index heuristic for predicting the winner

Calculate the overall index score for each candidate.

Decision rule (issue index heuristic)Predict the candidate with the higher index score to

win the popular vote.

Performance- Correctly predicted 9 of 10 popular vote winners. - Missed Reagan in 1980.

Issue-index model for predicting vote-shares

The simple heuristic performs well in predicting the winner.

But it does not allow for predicting the popular vote share.

Issue-index modelSimple linear regression to relate the relative index score (I)

of the incumbent to the popular vote (V)

Vote equation: V = 40.3 + 0.22 * I

Relative performance: Issue-index model vs. econometric models

We compared out-of-sample (ex ante) forecast accuracy of the issue index model and 8 established econometric models for predicting the three elections from 2000 to 2008.

Early September forecast: Issue index model provided a lower MAE than all 8 benchmark

models but did not predict the correct winner in the 2004 election.

Election Eve forecast: Issue index model also correctly predicted the 2004 election

winner.

Summary

Issue-index model yielded a higher hit rate and more accurate long-term vote-share forecasts than the IEM.

Issue-index model yielded more accurate out-of-sample forecasts of the vote shares than 8 econometric models (early September forecasts)

Benefits of issue indexes

Contribute to forecasting accuracy, in particular for long-term forecasting and the PollyVote.

Can help political candidates in deciding about which issues to focus on in their campaign. - Increase marketing effort to gain ownership of an issue.- Raise and promote issues that favor them but which have not

received attention in the public yet.- Adopt new or revised positions and diverge from traditional

party views.

Simple to use and easy to understand.

Limitations of issue indexes

CostsMust summarize prior knowledge about the field.

Acceptability Easy to understand and thus easy to criticize.

People wrongly believe that complex methods are necessary to solve complex problems. They exhibit a general resistance to simple solutions.

[Hogarth, in press]

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Future work on index models (1)

Predict the election outcome based on how voters perceive the candidates’ personalities.

E.g., which of the candidates is more likable, honest, etc.

Implications for decision-making:- Helps candidates to decide whether to run- Helps parties to decide who to nominate

Future work on index models (2)

Predict the election outcome based on how voters agree with candidates’ positions on policies.

Implications for decision-making:- Helps candidates to decide which policies to pursue.

Examine policies related to issues such as gun control, income taxes, free trade, abortion, government spending to see which candidate is closest to the opinions of the voters on more policies.

The Index Model Challenge

Index method will be more accurate than econometric models in situations with

- many variables- much prior knowledge (especially

experiments) and- lack of data, measurement errors, and

collinearity.

Examples: Selecting CEOs, drafting athletes, marriages, economic growth rates of nations, value of real estate, medical treatments, effectiveness of ads.

Outline

1. Status-quo in election forecasting

2. Limitations of multiple regression for social science predictions

3. Index method

4. Issue-index model

5. Future applications of the index method

6. Conclusions

Conclusions

We used the index method to develop the issue index model, which is based on information about how voters perceive the candidates to handle the issues

The model correctly predicted the winner in 9 of the last 10 U.S. Presidential Elections. Its out-of-sample forecasts for the past three elections outperformed established econometric models.

The model improves the accuracy of long-term election forecasting and can help candidates to decide about which issues to focus on in their campaign.

Further improvements in accuracy are expected based on the index method – which itself can be used in many other applications.

References

Armstrong, J. S. (1985). Long-range forecasting: From crystal ball to computer, New York: John Wiley.Armstrong, J. S. & Graefe, A. (2010). Predicting elections from biographical information about candidates, Journal of

Business Research, in press. Berg, J., Nelson, F. & Rietz, T. A. (2008). Prediction market accuracy in the long run. International Journal of Forecasting,

24, 285-300.Burgess, E. W. (1939). Predicting success or failure in marriage. New York: Prentice-Hall.Cuzán, A. G. & Bundrick, C. M. (2009). Predicting presidential elections with equally-weighted regressors in Fair's

equation and the fiscal model, Political Analysis 17, 333-340. Czerlinski, J., Gigerenzer, G. & Goldstein, D. G. (1999). How good are simple heuristics? In: G. Gigerenzer, G. & Todd, P. M.

(Eds.), Simple heuristics that make us smart. Oxford University Press, pp. 97-118.Einhorn, H. J. & Hogarth, R. M. (1975). Unit weighting schemes for decision-making, Organizational Behavior & Human

Performance, 13, 171-192.Fair, R. C. (1978). The effect of economic events on votes for president, Review of Economics and Statistics, 60, 159-173.Gough, H. G. (1962). Clinical versus statistical prediction in psychology. In: L. Postman (Eds.), Psychology in the making.

New York; Knopf, pp. 526-584.Hogarth, R. M. (2006). When simple is hard to accept. In: P. M. Todd & Gigerenzer, G. (Eds.), Ecological rationality:

Intelligence in the world (in press). Oxford; Oxford University Press, pp.Jones, R. J. & Cuzán, A. G. (2008). Forecasting U.S. Presidential Elections: A brief review, Foresight, Issue 10, 29-34.Lichtman, A. J. (2008). The keys to the white house: An index forecast for 2008, International Journal of Forecasting, 24,

299-307. Rhode, P. W. & Strumpf, K. S. (2004). Historic presidential betting markets, Journal of Economic Perspectives, 18, 127-

142.

Appendix

Relative performance: issue index model vs. prediction markets

We compared the issue index forecasts to the market prices from the Iowa Electronic Markets.

Historical prediction marketsPopular and accurate already in the late 19th and

early 20th century (Rhode & Strumpf 2004)

Performance of prediction markets

Iowa Electronic Markets (IEM) yielded a lower forecasting error than the typical poll 74% of the times from 1988 to 2004 (Berg et al. 2008)

Relative performance: Issue-index vs. IEM (hit rate)

For the last 150 days prior to election day, the issue-index model (forecasts derived by jackknifing) yielded a higher hit rate than the IEM (1988-2008).

Relative performance: Issue-index vs. IEM (forecast error)

Across the last 150 days prior to election day, the issue-index model and the IEM yielded an identical MAE of 2.3 percentage points (1988-2008).

Accuracy varied depending on forecast horizon.- issue index model more accurate for long-term forecasts- prediction markets superior for short-term forecasts

Procedure for selecting the issues

Operational definition“A political issue is a matter of public concern and is something

that the next president can be expected to take action about. An issue always focuses on a particular problem. Issues do not include policies for solving problems.”

Four coders decided independently on whether to include an issue.

In case of ties, the authors made the final decision.