The Cheater Problem Revisited:Lessons from Six Decades of Polling

Regina Faranda, Acting Director (FarandaRD@state.gov)Office of Opinion Research, U.S. Department of StateNEAAPOR, February 2015

HUMAN BEINGS, it seems, remain human beings when they are asking questions* for polling organizations.

Source: Leo P. Crespi (1945). The Cheater Problem in Polling. Public Opinion Quarterly, 9(4), 431-445.

Overview

What is the State Department’s Office of Opinion Research?

The “Morale Problem” and Falsification• A case study from international research

Tackling Falsification – a holistic approach

Don’t blame the mirror for your own ugly mug. - Gogol, The Government Inspector

OPN – A Brief History

Initially created in the 1950s as the research office within the U.S. Information Agency• An initial goal of the Marshall Plan was to better

understand the causes of war in Europe, and how European publics would respond to U.S. assistance

Leo P. Crespi, Chief Methodologist 1954-1986

Integrated into State Department in 1999 as the Office of Opinion Research• The Overseas Pollster of the U.S. Government

Scope and Stakes

OPN conducts roughly 200 surveys annually in some 100 countries and produces about 450 reports per year based on our original polling.

The office has a staff of about 30 analysts conducting policy-relevant survey research as well as vetting other available polling of interest to our customers.

FIDELITY is critical—Main audience: Secretary of State, White House, U.S. Government

What We Do

Contract with local firms (usually multiple firms in a country over time) to implement surveys

Train local firms where there is no capacity

Vast majority is face-to-face

Responsible for the entire process from project conceptualization to sample design and selection to questionnaire construction to post-survey adjustments and quality control to final analysis and reporting

Often in difficult environments

“Difficult” means… There and back again

In Afghanistan, the average sampling point is miles away from the nearest road. Often, respondents have no cell phone, so checks or re-interviews mean a second trip. Computer tablets and taking photos of households draw immediate attention.

Photo: Bamyan Province

“Difficult” means… Where to start?

A devastating 2010 earthquake displaced 1.5 million Haitians from their neighborhoods. Sampling points were impromptu camps. OPN used satellite imagery and GPS key chains to implement the skip pattern. Re-checks were still complicated by spotty cell access and movement of residents.

Photo courtesy of Blake Congdon

Falsification – A Morale ProblemA Case Study

That doesn’t look right…

Suspicious data patterns, one country, two field firms over two years.

One of the many difficult environments in which OPN conducts surveys

• Only viable mode -- face to face interview

• Impossible to audit or observe fieldwork

• Environment – interviewer error more likely

Contracted with a third party to pinpoint data quality issues

Techniques: Comparative analysis:

Deviation Analysis and Duplication Analysis

Internal analysis

• Straights

• Inconsistencies

• Rare combinations

Inconsistencies and Duplicates

“Top Duplicates” pairs of surveys sharing half or more of their responses in sequence

Top Duplicates Clustered by Supervisor:

All other supervisors (15+) have no top duplicates

Moving Forward

Source: Thomas Hertzog, Fritz Scheuren, and William Winkler, Data Quality and Record Linkage Techniques. 2nd ed. (Springer, 2015), 234 p.

Data Quality is Everyone’s Responsibility

Data Quality Begins at Home

QC Team Ethos:

Data checking is a necessary step in ensuring that our reporting is based on data having the highest integrity.

Threats to data integrity arise from various sources, including the following:• Some field firms make mistakes which may go undetected in

the fielding of the survey as well as in the data processing.

• Some field firms may be victims of intentional or undetected data fabrication by staff in one or more positions within the organization or within a subcontracted segment of the survey.

Data Quality Begins at Home

Consolidate efficient ways to execute each Quality Control check (e.g., SPSS syntax, logical/systematic approaches);

Establish thresholds to help identify “problems” (which may require record keeping of QC results to pinpoint deviations from norms); and,

Define appropriate courses of action based on the QC analysis (ranging from addressing smaller issues to dumping the data entirely).

The Checklist

Institutionalizing Deterrence

In addition to metadata needed for quality control, OPN analysts have the option of requiring two additional SPSS files among final deliverables:

1. Containing all incomplete cases (such as break-offs and refusals);

2. Containing cases rejected because they did not meet the field firm’s quality control standards.

The fidelity of musical recordings was greatly increased by unremitting attention to the point of contact between the record and the amplifier—the pickup. May it not be true that the fidelity of opinion recording can be similarly augmented by close attention to the point of contact—the interviewer?

High FIDELITY in recording the voice of the people is worth not a little to achieve.

Truer words…

Source: Leo P. Crespi (1945). The Cheater Problem in Polling. Public Opinion Quarterly, 9(4), 431-445.

Regina FarandaActing DirectorOffice of Opinion ResearchU.S. Department of StateFarandaRD@state.govThank you.