Political Science Scopeand Methods

Introduction to Research Design and The Experimental Method

Introduction to Research Design

Scientific method (again) Theory development

What qualifies as a theory? Van Evera’s view

Theory testing Van Evera: Positivist approach

Inductive vs. deductive theorizing

Some Terminology

Dependent Variables (DV) and Independent Variables (IV) Van Evera definition Alternative conception:

We explain particular phenomenon – our DV – as a function of specific explanations – our IVs.

Examples Strategies

Terminology (continued)

Internal vs. External validity Internal validity: the “real effect.” External validity: “generalizability.” Threats to validity

Example (internal): School vouchers Example (external): Social psychology

Bottom Line (76 pages of Campbell and Stanley later): Be Careful!

Testing theories

Van Evera: 2 ways to test theories: Experimentation Observation

Case studies “Large N” (statistical) analysis

Experiments

Experimentation: Lab experiments

Effect of negative advertisements (Ansolabehere) Study of political cognition (Berinsky)

Field experiments Effect of canvassing, telephone calls, and mailing on

turnout (Green and Gerber) Effectiveness of “franking” – baby books and ballots

(Cover and Brumberg) Survey Experiments

War in Iraq (Berinsky)

Example: Iraq War

Please give your best guess to this next question, even if you are not sure of the correct answer. As you know, the United States is currently involved in a war in Iraq. Do you happen to know how many soldiers of the U.S. military have been killed in Iraq since the fighting began in March 2003?

Log (Base 10) of Estimates of U.S. Troop Deaths in Iraq, 2004

0.1

.2.3

.4.5

Fra

ctio

n

0 1 2 3 4 5Log(10) Casualties

Table 3:Predicted Probability of Causality

Estimates

Pr (Underestim

ate)

Pr (Correct Answer)

Pr (Overestima

te)Information

Low Information 0.51 0.31 0.18

High Information 0.36 0.56 0.07

Difference -0.15 +0.25 -0.11

Follow News About Iraq?

Not At All Closely 0.66 0.24 0.11

Very Closely 0.25 0.67 0.08

Difference -0.41 +0.43 -0.03

Partisanship

Strong Republican 0.48 0.44 0.08

Strong Democrat 0.35 0.54 0.12

Difference -0.13 +0.10 +0.04

Effect of Information Treatment on Support for War in IraqAmong Under-EstimatorsDid The U.S. Make The Right Decision in Using Military Force against Iraq?

Estimate War Deaths

Corrected Information

Right Decision 48% 44%

Wrong Decision 52% 56%

N=252; 2(1)=0.40 Pr=0.53Has The Current War in Iraq Been Worth Fighting?

Estimate War Deaths

Corrected Information

Worth Fighting 58% 53%

Not Worth Fighting

42% 47%

N=253; 2(1)=0.71 Pr=0.40

Among Over-estimatorsDid The U.S. Make The Right Decision in Using Military Force against Iraq?

Estimate War Deaths

Corrected Information

Right Decision 42% 42%

Wrong Decision 58% 58%

N=57; 2(1)=0.00 Pr=0.95Has The Current War in Iraq Been Worth Fighting?

Estimate War Deaths

Corrected Information

Worth Fighting 42% 48%

Not Worth Fighting

58% 52%N=572(1)=0.26 Pr=0.61

The Practice of Experimentation

Campbell and Stanley: The hard sellLimitation of experiments

Experimental work as the plutonic ideal

Experiments are about control Payoff in causal inference Maximize internal validity (if do them correctly) Random AssignmentRandom Assignment

Note: Random assignment random selection

Other Concerns

Construct validity Why does the treatment work? Is the treatment what we say it is?

Experiments vs. Quasi Experiments

Experiments: C&S – p.8: If you use random assignment, you don’t need to worry about internal validity

Quasi-Experiments: C&S – p. 40,56 – things are not so neat Specific threats to worry about Designs that control for all threats to validity

might be hard to operationalize

Example

Enid wants to investigate the effect of saliency of message on attitude change. From an old Ph.D. she finds a swell communication on the importance of physical sciences in a liberal education. Fortunately for her, Widget University conducts separately - English classes for engineers and liberal arts majors. Within this limitation, however, the university has matched the classes carefully on age, sex composition, socioeconomic background, and College Entrance Board Scores (both verbal and mathematical ability as well as on scores in specific subjects). Enid checks on the dean's records and is happy to find that the classes have indeed been matched to the best possible extent. Enid then delivers the message to the engineers (the salient group) and to the liberal arts students (the non-salient group). The engineers show much more attitude change. Enid concludes that message saliency increases attitude change.