Can$Oﬃcial$Records$Correct Errors$in$TurnoutSelf6reports?$

Can Official Records Correct Errors in Turnout Self-‐reports?

Ma:hew K. Berent, Stanford University

Jon A. Krosnick, Stanford University

Arthur Lupia, University of Michigan

AssumpJons

•  IF: EffecJve sampling.

•  PLUS: EffecJve survey design.

•  THEN: Accurate data.

•  TO: Understand the populaJon.

Theory vs. PracJce

Apparent Self-‐report Errors

EsJmates from sample data do not match known populaJon values.

Reasons for Discrepancies

QuesJon: Did you vote? •  Non-‐response.

•  The quesJon is not answered. •  Survey effects.

•  ParJcipaJon affects behavior. •  MisreporJng.

•  Answers are factually incorrect.

Types of MisreporJng

QuesJon: Did you vote? •  MisinterpretaJon.

•  Interpreted: Is poliJcs important to you? •  Misremembering.

•  Recalled: I wanted to vote so I must have. •  IntenJonal lying.

•  Reported: (I didn’t, but saying that would not look good so I’ll say) “Yes”.

When Voters Lie

“It’s a given that people fib in surveys.”

The Wall Street Journal, August 2, 2008

Apparent problem: Respondents lie.

Proposed soluJon: Use official records.

What to Do

Using Official Records

•  Obtain official records of turnout histories.

•  Match each respondent to his or her official record.

•  Determine the “correct” turnout for each respondent.

Two Ways Thought to IdenJfy Lying

A respondent reported one thing

BUT

•  An official record shows another

OR

•  No official record is found

Fail-to-match

A Case Study 2008-‐2009 ANES Panel Study

Total Sample

•  N=4,206 respondents recruited via RDD sampling.

•  US ciJzen •  VoJng age •  Landline households •  Completed monthly online surveys

Turnout

Which one of the following best describes what you did in this elecJon?

__ Definitely did not vote __ Definitely voted in person at a polling place on elec2on day __ Definitely voted in person at a polling place before elec2on day __ Definitely voted by mailing a ballot to elec2ons officials before elec2on day __ Definitely voted in some other way __ Not completely sure whether you voted or not

The QuesJon

ANES Self-‐report Turnout Rates



ValidaJng Turnout Self-‐reports

The ValidaJon Task •  Match government records to survey respondents.

•  Determine “correct” turnout for each respondent.

Two problems •  200,000,000+ people eligible to vote in the 2008.

•  Some government records are difficult to obtain.

Commercial Vendors

•  Several are available for records matching.

•  Maintain naJonal databases of government records.

•  Use records from mulJple sources to increase data accuracy.

•  Developed technology for records matching.

Our Requirements

•  Full disclosure about the vendor’s records. •  When they were obtained.

•  From whom were they obtained.

•  How the records are cleaned.

•  Full disclosure about the matching process.

•  Details about their matching algorithm.

•  Their criteria for determining a match.

What We Learned

•  Vendors use names, addresses, and dates of birth to match records.

•  Some vendors are more transparent than others.

•  Some vendors can match records at different levels of confidence.

•  No vendor was willing to provide complete details about their matching algorithm.

In-‐house Records Matching

•  Obtain government registraJon and turnout records from a sample of states.

•  Use a publicly available computer applicaJon to match respondents to government records.

•  LinkPlus – Developed by the CDC

2008-‐2009 ANES Panel Study Our Subsample (N=761)

Respondents with a residenJal address in: •  California •  Florida •  New York •  North Carolina •  Ohio •  Pennsylvania

Weighted to reflect within-‐state landline populaJons:

•  Age •  Sex •  Race •  EducaJon •  Income •  Marital status

Matching Respondents to Records

Three methods of matching


Last name First name Address DOB

Strict Iden2cal Iden2cal Iden2cal Iden2cal





Moderate Same or similar

Same or similar

Same or similar

Iden2cal





Moderate Same or similar

Same or similar

Same or similar

Iden2cal

Least Same or similar

Same or similar

Same or similar

Same or similar

Same or similar

Same or similar

Iden2cal

Iden2cal Iden2cal Iden2cal


Strict Moderate Least All target states

43% 59% 69%

Turnout Rates According to Government Records

Strict Moderate Least Popula2on All target states

43% 59% 69% 62%

“Correct” Turnout Turnout Rates According to Government Records

Strict Moderate Least Popula2on All target states

43% 59% 69% 62%

Using Government Records

If lying causes turnout over-‐repor2ng

•  Moderate matching is the most valid

•  Least matching is somewhat valid

•  Strict matching is not valid

•  Self-‐reports cannot be trusted

Turnout Rates According to Government Records

Problem Solved?

Problem: Respondents Lie

SoluJon: Use government records

Proof: Sample turnout=Popula2on turnout

DerivaJve: Sample registra2on=Popula2on registra2on

“Correct” registraJon

Popula2on All target states 83%

•  PopulaJon registraJon rate based on staJsJcs published by states.

•  RegistraJon rate among voJng age US ciJzens in landline households.

•  Adjusted for deadwood.

RegistraJon Rates According to Government StaJsJcs

Popula2on Strict All target states 83% 49%

•  Strict matching underesJmates populaJon registraJon rate.

“Correct” registraJon RegistraJon Rates According to Government StaJsJcs

Popula2on Strict Moderate All target states 83% 49% 66%

•  Moderate matching underesJmates populaJon registraJon by 17%.

•  Moderate matching appears invalid.


Popula2on Strict Moderate Least All target states 83% 49% 66% 78%

•  Even Least matching underesJmates populaJon registraJon.


Compared to the PopulaJon

Strict Moderate Least Turnout Way Too Low Closest Too High RegistraJon Way Too Low Way Too Low Closest

What is going on?

CompeJng Biases

•  Downward bias – Using government records incorrectly idenJfies some respondents as having not turned out.

•  Upward bias – People who parJcipate in surveys are more likely to vote.

Evidence of an upward bias?

Turnout among the Registered

Among voJng-‐age US ciJzens in landline households WHO ARE REGISTERED:

Turnout=75%

Based on staJsJcs published by states, adjusted for deadwood



Turnout=75%

Strict Moderate Least Turnout 88%

Strict matching suggests higher turnout among respondents



Turnout=75%

Strict Moderate Least Turnout 88% 90%

Moderate matching suggests higher turnout among respondents



Turnout=75%

Strict Moderate Least Turnout 88% 90% 88%

Least matching suggests higher turnout among respondents

Why the downward bias?

Self-‐Reported Voters

Truthful Record shows no turnout

Fail-‐to-‐match

Strict 50% 3% 47% Moderate 68% 4% 27% Least 80% 5% 15%

Self-report confirmed by government records



Fail-‐to-‐match


Self-report disconfirmed by government records



Fail-‐to-‐match


Self-report cannot be confirmed by government records

Problem: Respondents lie.

Problem: Sample turnout > PopulaJon turnout.

SoluJon: Use government records.

Using government records create an illusion of accuracy.

SoluJon: Use self-‐reports.

Turnout self-‐reports appear extremely accurate.

Sample Turnout ≠ PopulaJon Turnout

Thank You!

Documents

Can$Oﬃcial$Records$Correct Errors$in$TurnoutSelf6reports?$