Preso on nate silver

Consolidated Behaviors and Attitudes 1

Modeling the 2012 U.S. Presidential Election

Exploring the Forecasting Methods

Of Nate Silver and the Five-Thirty-Eight Blog Prepared for the 3/1/13 RTP Analysts Luncheon Meetup

By Bruce Conner Consolidated Behaviors and Attitudes


Purposes of This Discussion

• Walk through the elements of Nate Silver’s methods in predicting the 2012 Presidential election

• “Kick around” some outstanding questions about those methods

• Explore whether participating analysts are using similar methods in their day-to-day work – and how they are using them?

• Sources listed in a “rough bibliography” at the back of this presentation


Background: Nate Silver • Developed PECOTA, a highly-successful “sabremetric”

model used for forecasting the performance and career development of Major League players

• In 2008: – Correctly predicted 49 out of 50 states (missed on Indiana)

– Correctly predicted all U.S. Senate elections

– Made better predictions for some of the primaries than major polls, using a demographic prediction model

• In 2010, less good predictions in U.S. House races

• In 2012: – Correctly predicted all 50 states and the District of Columbia


Nate’s Approach to Forecasting the Elections (1)

• Forecast Each State – As we learned in 2000, the election is decided in the Electoral

College: – Nate decomposes the election into the 50 states and the District

of Columbia, and forecasts popular vote in each separately – National projections of popular vote are simply the aggregate of

the state popular votes – This approach is a type of hierarchical modeling

• Who is using hierarchical modeling techniques – and how are you using them?

• Leverage Multiple Data/Predictor Sources – Polls within each state – Voter registration and demographics – Economic indicators – “Borrowed information” from other (similar) states – National polling (used only for inferring trends in the individual

states) – Etc.



• Calibration of Each Poll (Offsetting “House Effects”) – Measure -- and offset -- systematic bias of each pollster

– This is done by a regression for each pollster of all their national polls against a weighted average of the national polls, as well as a regression of their polls in the various states against a weighted average of the polls in each state. Nate also lowered weight of polls that seem to be consistent outliers



• An Economic Index is Used as an “Additional Poll.” It is based on: – The four factors used to date recessions

• Job growth (nonfarm payrolls)

• Personal income

• Industrial production

• Consumption

– Inflation (CPI)

– Change in S&P 500 Index – Consensus forecast of gross domestic product growth over the next two

economic quarters, as taken from the median of The Wall Street Journal’s monthly forecasting panel

• The economic index is used as a predictor of trend in future polls – as a “gravitational factor” that is likely to influence polls in the coming periods (with a lag effect). It is gradually removed (deweighted) from the model as the election approaches.



• Adjustment to state-level polls based on trend – National polls and state polls show trends over time

– Particularly when some states do not have recent polls (or recent polls by a particular pollster), it is often reasonable to infer that if those polls would have been repeated today, they would be affected by trends

– The method to determine trends is a regression with (1) dummy variables for each week and (2) dummy variables for each pollster in each state.

– A LOESS regression is used to smooth the trend.

– A “trend” correction is then applied to the older polls to reflect the effects of the trend since the poll was taken

– An example of “Bayesian thinking”



• A multiple regression of demographics, registration, and similar factors is treated as an additional poll – Voter registration, race, gender, age, income, etc. as

predictors of how a state will vote



• Weighted Averaging of Polls – The predictions from the demographic regression and

from the economic index are treated as additional polls

– Weight of the economic index is reduced as election approaches, until it disappears in the final forecast

– Weighting of Polls Against Each Other Is Based On

• Sample size

• Recency (weighting is done using a “half life” formula – because polls closer to the election are more accurate)

• Ratings of pollsters (see following slides)



• Pollster Ratings* Reflect Accuracy of Each Pollster in Predicting Actual Election Results – Raw ratings scores are done based on a multiple

regression – Each data point is a published poll taken by a pollster in

one of: • A presidential, gubinatorial, senatorial, congressional, or

“generic house” race • A full range of elections/primaries/caucuses • Within 21 days of an election, primary, or caucus • For elections starting with the 1998 election cycle

– The rating of each pollster is based on all of the polls attributed to that pollster – across the years and in the different types of elections

– The dependent variable is the size of the error in predicting the gap between the top two candidates

*For more detail on pollster ratings methods, see additiona l slides near end of this presentation



• Pollster Ratings (Continued) – Some properties of the raw rating regression:

• Pollsters are “rewarded” for accurately predicting results (minimizing error) further away from the election

• Pollsters are also “rewarded” for “degree of difficulty” of predictions

– It’s harder to accurately predict state and local elections than national popular vote in a presidential election, and to predict primaries than “generals” – as demonstrated in the following average percent error bar chart



• Nearest Neighbor – Particularly useful for states that have low levels of

polling

• This tends to be true for states with lower population and for states “not in play”

– Nate Silver is a big fan of Bayesian approaches and Bayesian thinking – could he have used such approaches, instead of Nearest Neighbor, to “borrow” information from similar states?



• Modeling Uncertainty: based on a number of factors, each state forecast includes both a point forecast and an “uncertainty distribution” (a normal curve????). Uncertainty is influenced by: – Number of (reliable) polls – Sample sizes – Number of undecided voters – Consistency of the polls being averaged (size of standard

deviation) – Time until election

• Has Nate modeled levels of uncertainty in various kinds of elections at various removes from the election? How is this analysis done?

– NOTE: Nate provides both a “Now Forecast” (“If the election were held today …”) and a true forecast. These completely converge as election nears



• Monte Carlo Simulation: on a regular basis (eventually every night), 10,000 simulations of the election are run

– Results for each state in each simulation is arrived at separately, providing both the “expected” (mean) result and the uncertainty (a normal curve???)

– Some approach is used to take account that states are not independent of each other (e.g., Minnesota and Wisconsin tend to move in tandem) – nearest neighbor? Trending?

– For each simulation, a winner is picked – and electoral votes assigned accordingly

– For each simulation, electoral votes are totaled and a winner picked

– The thousands of simulations provide a distribution of probable electoral college outcomes and a distribution of popular vote outcomes (by state and national)



• Major event adjustments – Based on previous elections, certain predictable events are expected

to have a somewhat predictable effect over the course of the campaign cycle • In 2012, both candidates were “expected” to get a “convention bounce” of a

certain magnitude

• As challenger, Romney was also “expected “ to get a bounce of a certain magnitude from the 1st debate.

• Bounces from previous elections tend to be “noisy” but do show average effects

– The effect of the bounces s expected (predicted) to decay in predictable way

– Nate Silver: • Factored in expected bounces and decays in making forecasts of final results

• Recognized increased uncertainty during periods when bounces were most recent

• Temporarily penalized or rewarded candidates who under- or over-performed the expected bounces


Projected (Expected) Convention Bounces


Additional Issues (Related to Polling) Not Addressed in This Presentation

• Differing Sampling Approaches of Polls

• Differing Likely Voter Models of Polls

• Differing Survey Media of Polls – Web only?

– Include cell phones in sample?

• Differing Weighting Schemes of Polls – To correct for demographic bias

– To offset biases of particular survey media


How Did Nate Do?

Obama Won!


How Did Nate Do? Obama Got 332 Electoral Votes: Exactly the Mode (Most

Likely Outcome) of His Distribution Projection


How Did Nate Do?

Popular Vote: An Error of 1.3% • Silver predicted a 2.5% gap – final result was a 3.8% gap


How Did Nate Do? • He correctly called all 51 states – and called Florida as the only

“tossup” – with Obama having a 50.3% chance of victory


Forecast Errors of the States

• 2 of 51 states were outside of the expected margin of error (we would have expected 1 state)

• On average, Nate forecast the states 0.2% too Republican (skewed slightly Republican)

• Skewness = -0.6

• Kurtosis = 0.75

“Too Republican” “Too Democratic”


Errors in Key (Swing) States and States With Largest Errors

State Pct Too Republican

Colorado 2.2

Florida 0.6

Iowa 2.4

Michigan 1.4

Nevada 2.1

New Hampshire 2.2

North Carolina (0.5)

Ohio 1.7

Pennsylviania (0.8)

Virginia 1.0

State Pct Too Republican

West Virginia (10.6)

Mississippi 9.1

Hawaii 9.0

Alaska 7.6

Montana (5.1)

North Dakota (5.1)

Utah (5.1)

New Jersey 5.0

Arizona (4.6)

Swing States

States With Greatest Errors


Nate’s Approach to Forecasting the Elections

• Drilling Down Into Pollster Ratings (1) – Raw Scores Are Produced By a Multiple Regression

– Variables in the regression include: • Dummy variables representing each pollster (the “B” of these

dummy variables is the raw score for the pollster)

• The square root of the number of days between the median polling date and the election (separate variables for primaries and general elections – because primaries are harder to predict)

• Sample size (this variable is only marginally significant)

• Dummy variables to represent the type of election and the cycle (e.g., a single variable represents a 2000 senatorial election)

• A separate dummy variable to indicate primary vs. caucus

• A set of dummy variables indicating particular races – for those races that have “robust” amounts of polling

– Non-robust races lack these dummy variables, and are therefore compared against each other



• Drilling Down Into Pollster Ratings (2) – Regression data points are weighted based on:

• How many surveys the pollster did for each particular election

• How recent the election was (e.g., in 2010, 2008 elections were weighted twice as much as 1998 elections)

– Raw rating scores are “regressed against the mean” to produce a final pollster rating • For the 2012 cycle, two raw regressions were done – one for the

election cycles through 2008, and another for the 2010 election cycle.

• The purpose was to understand how well the earlier regression predicted the 2010 regression for each pollster – and to “discount” each rating to account for how much of the rating was the result of “signal” and how much “noise” (don’t fully understand this)

• The resulting formula provided a “reversion parameter” that calculated how much the raw score should be discounted, based on the following formula: reversionparameter = 1 - (0.06 * sqrt(number of surveys))



• Drilling Down Into Pollster Rations (3): – Example of regression against the mean of raw rating

scores: • A particular pollster has a raw score of -0.50 (i.e., on a

weighted average, their polls produce a 0.5% reduction in error compared to the average for all pollsters)

• The same pollster has 25 polls in the sample

• reversionparameter = 1 - (0.06 * sqrt(number of surveys)) = 1 – (0.06 * sqrt(25)) = 0.7

• Final rating score = 0.50 * (1-0.7) = 0.15

– In the regression against the A dummy variable representing whether the pollster has made one of two commitments to methodological transparency (a predictor of accuracy)


Principal Sources (1) • http://fivethirtyeight.blogs.nytimes.com/2012/06/07/election-forecast-obama-begins-with-

tenuous-advantage/

• http://fivethirtyeight.blogs.nytimes.com/2012/06/25/the-problems-with-forecasting-and-how-to-improve/

• http://fivethirtyeight.blogs.nytimes.com/2012/07/05/measuring-the-effect-of-the-economy-on-elections/

• http://fivethirtyeight.blogs.nytimes.com/2012/08/29/measuring-a-convention-bounce/

• http://www.boston.com/sports/blogs/statsdriven/2012/11/nobodys_perfect_nate_silver_an.html

• http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/

• http://jwrteaching.blogspot.com/2012/11/election-results-fivethirtyeight-and.html

• http://www.guardian.co.uk/science/grrlscientist/2012/nov/08/nate-sliver-predict-us-election

http://fivethirtyeight.blogs.nytimes.com/2012/06/07/election-forecast-obama-begins-with-tenuous-advantage/














http://fivethirtyeight.blogs.nytimes.com/2012/06/25/the-problems-with-forecasting-and-how-to-improve/
















http://fivethirtyeight.blogs.nytimes.com/2012/07/05/measuring-the-effect-of-the-economy-on-elections/
















http://fivethirtyeight.blogs.nytimes.com/2012/08/29/measuring-a-convention-bounce/








http://www.boston.com/sports/blogs/statsdriven/2012/11/nobodys_perfect_nate_silver_an.html




http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/






















http://jwrteaching.blogspot.com/2012/11/election-results-fivethirtyeight-and.html








http://www.guardian.co.uk/science/grrlscientist/2012/nov/08/nate-sliver-predict-us-election










Principal Sources (2) • http://venturebeat.com/2012/11/07/data-science/

• http://en.wikipedia.org/wiki/Bayes_factor

• http://fivethirtyeight.blogs.nytimes.com/2012/06/22/calculating-house-effects-of-polling-firms/

• http://fivethirtyeight.blogs.nytimes.com/2012/10/18/gallup-vs-the-world/

• http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/#more-37396

• http://www.fivethirtyeight.com/2010/06/pollster-ratings-v40-methodology.html

• http://fivethirtyeight.blogs.nytimes.com/fivethirtyeights-2012-forecast/

• http://www.fivethirtyeight.com/2008/03/frequently-asked-questions-last-revised.html

• http://en.wikipedia.org/wiki/FiveThirtyEight

• http://www.fivethirtyeight.com/2008/06/we-know-more-than-we-think-big-change-2.html

• mitch.watnik.com/symposium/silver%20nate.ppt

http://venturebeat.com/2012/11/07/data-science/



http://en.wikipedia.org/wiki/Bayes_factor

http://fivethirtyeight.blogs.nytimes.com/2012/06/22/calculating-house-effects-of-polling-firms/











http://www.fivethirtyeight.com/2010/06/pollster-ratings-v40-methodology.html











































http://fivethirtyeight.blogs.nytimes.com/fivethirtyeights-2012-forecast/






http://www.fivethirtyeight.com/2008/03/frequently-asked-questions-last-revised.html










http://en.wikipedia.org/wiki/FiveThirtyEight

http://en.wikipedia.org/wiki/FiveThirtyEight

http://www.fivethirtyeight.com/2008/06/we-know-more-than-we-think-big-change-2.html


















mitch.watnik.com/symposium/silver nate.ppt

Documents

Preso on nate silver