Upload
dokien
View
217
Download
2
Embed Size (px)
Citation preview
Statistical Analysis of Endorsement Experiments:Measuring Support for Militant Groups in Pakistan
Kosuke Imai
Department of PoliticsPrinceton University
Joint work with Will Bullock and Jacob Shapiro
May 6, 2011
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 1 / 23
Motivation
Survey is used widely in social sciencesValidity of survey depends on the accuracy of self-reportsSensitive questions =⇒ social desirability, privacy concernse.g., racial prejudice, corruptionsLies and nonresponses
How can we elicit truthful answers to sensitive questions?Survey methodology: protect privacy through indirect questioningStatistical methodology: efficiently recover underlying responses
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 2 / 23
Survey Techniques for Sensitive Questions
Randomized Response TechniqueMost extensively studied and commonly usedUse randomization to protect privacyDifficulties: logistics, lack of understanding among respondents
List ExperimentsAlso known as block total response and item count techniqueUse aggregation to protect privacyNew estimators to enable multivariate regression analysisNew methods to detect and correct list experiment failures
Endorsement ExperimentsUse randomized endorsements to measure support levelsDevelop a measurement model based on item response theoryApplications:
1 Pakistanis’ support for Islamic militant groups2 Afghanis’ support for Taliban and ISAF (joint with J. Lyall)3 Nigerians’ support for insurgents (joint with G. Blair)
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 3 / 23
Endorsement Experiments
Measuring support for political actors (e.g., candidates, parties)when studying sensitive questionsAsk respondents to rate their support for a set of policiesendorsed by randomly assigned political actors
Experimental design:
1 Select policy questions
2 Randomly divide sample into control and treatment groups
3 Across respondents and questions, randomly assign political actorsfor endorsement (no endorsement for the control group)
4 Compare support level for each policy endorsed by different actors
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 4 / 23
The Pakistani Survey Experiment
6,000 person urban-rural sample
Four militant groups:Pakistani militants fighting in Kashmir (a.k.a. Kashmiri tanzeem)Militants fighting in Afghanistan (a.k.a. Afghan Taliban)Al-Qa’idaFirqavarana Tanzeems (a.k.a. sectarian militias)
Four policies:WHO plan to provide universal polio vaccination across PakistanCurriculum reform for religious schoolsReform of FCR to make Tribal areas equal to rest of the countryPeace jirgas to resolve disputes over Afghan border (Durand Line)
Response rate over 90%
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 5 / 23
Endorsement Experiment Questions: Example
The script for the control groupThe World Health Organization recently announceda plan to introduce universal Polio vaccinationacross Pakistan. How much do you support such aplan?
The script for the treatment groupThe World Health Organization recently announceda plan to introduce universal Polio vaccinationacross Pakistan, a policy that has receivedsupport from Al-Qa’ida. How much do you supportsuch a plan?
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 6 / 23
Distribution of ResponsesPolio Vaccinations Curriculum Reform FCR Reforms Durand Line
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Not At All
A LittleA Moderate Amount
A LotA Great Deal
Control Group
Pakistani militant groups in Kashmir
Afghan TalibanAl−Qaida
Firqavarana Tanzeems
Control Group
Pakistani militant groups in Kashmir
Afghan TalibanAl−Qaida
Firqavarana Tanzeems
Control Group
Pakistani militant groups in Kashmir
Afghan TalibanAl−Qaida
Firqavarana Tanzeems
Control Group
Pakistani militant groups in Kashmir
Afghan TalibanAl−Qaida
Firqavarana Tanzeems
Pun
jab
Sin
dhN
WF
PB
aloc
hist
an
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 7 / 23
Methodological Challenges and Proposed Solutions
1 How to combine responses from multiple questions?=⇒ item response theory
2 How to recoup loss of statistical efficiency?=⇒ hierarchical modeling
3 What is the key assumption?=⇒ learning vs. support
4 How to select policy questions?Policies should belong to a single dimensionPolicies should be known to respondentsRespondents should not have strong views
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 8 / 23
Endorsement Experiments Framework
N respondents
J policy questions
K political actors
Yij ∈ {0,1}: response of respondent i to policy question j
Tij ∈ {0,1, . . . ,K}: political actor randomly assigned to endorsepolicy j for respondent i
Yij(t): potential response given the endorsement by actor t
Covariates measured prior to the treatment
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 9 / 23
The Proposed Model
Quadratic random utility model:
Ui(ζj1, k) = −‖(xi + s∗ijk )− ζj1‖2 + ηij
Ui(ζj0, k) = −‖(xi + s∗ijk )− ζj0‖2 + νij
where xi is the ideal point and s∗ijk is the “influence” of anendorsementThe statistical model (item response theory):
Pr(Yij = 1 | Tij = k) = Pr(Yij(k) = 1)
= Pr(Ui(ζj1, k) > Ui(ζj0, k))
= Pr(αj + βj(xi + s∗ijk ) > εij)
Support level: ∂∂sijk
Pr(Yij = 1 | Tij = k) > 0 where
sijk =
{s∗ijk if βj ≥ 0−s∗ijk otherwise
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 10 / 23
The Proposed Model (Continued)
Hierarchical modeling:
xiindep.∼ N (Z>i δ, σ
2x )
sijkindep.∼ N (Z>i λjk , ω
2jk )
λjki.i.d.∼ N (θk ,Φk )
“Noninformative” hyper prior on (αj , βj , δ, θk , ω2jk ,Φk )
Interpretation:spacial model vs. factor analysislearning vs. support
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 11 / 23
Quantities of Interest and Model Fitting
Average support level for each militant group k
τjk (Zi) = Z>i λjk for each policy j
κk (Zi) = Z>i θk averaging over all policies
Standardize them by dividing the (posterior) standard deviation ofideal points
Bayesian Markov chain Monte Carlo algorithmMultiple chains to monitor convergenceImplementation via JAGS (Plummer)
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 12 / 23
Model for the Division Level Support
Ordered response with an intercept αjl varying across divisionsThe model specification:
xiindep.∼ N (δdivision[i],1)
sijkindep.∼ N (λk ,division[i], ω
2k )
δdivision[i]indep.∼ N (µprovince[i], σ
2province[i])
λk ,division[i]indep.∼ N (θk ,province[i],Φk ,province[i])
Averaging over policiesPartial pooling across divisions within each province
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 13 / 23
Estimated Division Level Support−
1.0
−0.
50.
00.
51.
0
Sta
ndar
dize
d Le
vel o
f Sup
port
Pakistani militant groups in KashmirMilitants fighting in AfghanistanAl−QaidaFirqavarana Tanzeems
Bah
awal
pur
n=
118
Der
a G
hazi
Kha
n n
=0
Fais
alab
ad
n=31
3
Guj
ranw
ala
n=
403
Laho
re
n=57
9
Mul
tan
n=
495
Raw
alpi
ndi
n=
208
Sar
godh
a n
=13
1
Hyd
erab
ad
n=20
3
Kar
achi
n=
473
Lark
ana
n=
311
Mir
purk
has
n=
0
Suk
kur
n=
293
Ban
nu
n=0
Der
a Is
mai
l Kha
n n
=84
Haz
ara
n=
287
Koh
at
n=50
Mal
akan
d n
=0
Mar
dan
n=
215
Pes
haw
ar
n=28
8
Kal
at
n=10
3
Mak
ran
n=
0
Nas
iraba
d n
=21
0
Que
tta
n=32
0
Sib
i n
=67
Zho
b n
=61
Punjab Sindh NWFP Balochistan
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 14 / 23
Model with Individual Covariates
Ordered response with an intercept αjl varying across divisionsThe model specification:
xiindep.∼ N (δdivision[i] + Z>i δ
Z ,1)
sijkindep.∼ N (λk ,division[i] + Z>i λ
Zk , ω
2k )
δdivision[i]indep.∼ N (µprovince[i], σ
2province[i])
λk ,division[i]indep.∼ N (θk ,province[i],Φk ,province[i])
Expands upon the division level model to include individual levelcovariates:
gender, urban/rural, education, incomeIndividual level covariate effects after accounting for differencesacross divisionsPoststratification on these covariates using the census
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 15 / 23
Estimated Effects of Individual Covariates
−0.
2−
0.1
0.0
0.1
0.2
Pakistani militant groups in KashmirS
tand
ardi
zed
Leve
l of S
uppo
rt
●
●
●
●
●
●●
● ● ●
Female Rural Income Education −0.
2−
0.1
0.0
0.1
0.2
Militants fighting in Afghanistan
Sta
ndar
dize
d Le
vel o
f Sup
port
●
●
●
● ●●
●
● ●
●
Female Rural Income Education
−0.
2−
0.1
0.0
0.1
0.2
Al−Qaida
Sta
ndar
dize
d Le
vel o
f Sup
port
●
●
●
●
●
● ●●
●
●
Female Rural Income Education −0.
2−
0.1
0.0
0.1
0.2
Firqavarana Tanzeems
Sta
ndar
dize
d Le
vel o
f Sup
port
●
●
●
●
● ●● ●
●
●
Female Rural Income Education
Demographics play a small role in explaining support for groupsKosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 16 / 23
Regional Clustering of the Support for Al-Qaida
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 17 / 23
Association between Support and Violence
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
0
50
100
150
200
Division−Level Estimated Support (Proposed Method)
Num
ber
of In
cide
nts
correlation = −0.574
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.2 0.0 0.2 0.4 0.6
0
50
100
150
200
Division−Level Estimated Support (Ordered Probit)
Num
ber
of In
cide
nts
correlation = −0.061
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
0
50
100
150
200
Division−Level Estimated Support (Proposed Method)
Num
ber
of In
cide
nts
correlation = −0.594
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.2 0.0 0.2 0.4 0.6
0
50
100
150
200
Division−Level Estimated Support (Ordered Probit)
Num
ber
of In
cide
nts
correlation = −0.365
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
0
50
100
150
200
Division−Level Estimated Support (Proposed Method)
Num
ber
of In
cide
nts
correlation = −0.468
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.2 0.0 0.2 0.4 0.6
0
50
100
150
200
Division−Level Estimated Support (Ordered Probit)
Num
ber
of In
cide
nts
correlation = 0.021
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
0
50
100
150
200
Division−Level Estimated Support (Proposed Method)
Num
ber
of In
cide
nts
correlation = −0.414
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.4 −0.2 0.0 0.2 0.4 0.6
0
50
100
150
200
Division−Level Estimated Support (Ordered Probit)
Num
ber
of In
cide
nts
correlation = −0.166
Pakistani militant groups in Kashmir
Militants fighting in Afghanistan
Al−Qaida
Firqavarana Tanzeems
Strong negativeassociation betweensupport and violence
Much weaker associationfor the standard orderedprobit model (divisiondummies, treatmentvariables interacted withdivision dummies)
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 18 / 23
Ideology, Support, and Violence
No strong relationship between:ideology and violenceideology and support
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−0.5 0.0 0.5
0
50
100
150
200
Division−Level Estimated Ideal Point
Num
ber
of In
cide
nts
correlation = −0.040
●
●
●
●
● ●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
−0.5 0.0 0.5
−0.25
−0.20
−0.15
−0.10
−0.05
0.00
0.05
0.10
Division−Level Estimated Ideal Point
Div
isio
n−Le
vel A
vera
ge E
stim
ated
S
uppo
rt fo
r M
ilita
nt G
roup
s correlation = 0.087
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 19 / 23
Simulation Studies
1 Based on the Pakistani DataSame 2 models plus province-level issue ownership modelTop-level parameters held constant across simulationsSample sizes and distribution same as beforeIdeal points, endorsements and responses follow IRT models
2 Varying sample sizesModel for division-level estimates with no covariatesModel for province-level estimates with no covariates but supportvarying across policiesN = 1000,1500,2000Again, top-level parameters held constant across simulations whileideal points, endorsements and responses follow IRT models
100 simulations under each scenario (3 chains, 60000 iterations)Frequentist evaluation of Bayesian estimators
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 20 / 23
Monte Carlo Evidence based on the Pakistani Data
Den
sity
−0.10 −0.05 0.00 0.05 0.10
0
10
20
30
40
Den
sity
0.80 0.85 0.90 0.95 1.00
0
5
10
15
20
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Effect Size
Pro
port
ion
Sta
tistic
ally
Sig
nific
ant
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
Den
sity
−0.10 −0.05 0.00 0.05 0.10
0
10
20
30
40
Den
sity
0.80 0.85 0.90 0.95 1.00
0
5
10
15
20
●●●●●●
●
●●●●●
●
●
●
●●
●●● ●
●
●
●
●
●
●
●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●●
●●
●
● ●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Effect Size
Pro
port
ion
Sta
tistic
ally
Sig
nific
ant
The
Div
isio
n M
odel
W
ith In
divi
dual
Cov
aria
tes
The
Div
isio
n M
odel
Bias Coverage Rate of the 90% Confidence Intervals
Statistical Power
α level = 0.10
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 21 / 23
Monte Carlo Evidence with Varying Sample Size
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
N=1000 N=1500 N=2000
−0.2
−0.1
0.0
0.1
0.2
Bias
Bia
s
●
● ●
N=1000 N=1500 N=2000
0.75
0.80
0.85
0.90
0.95
1.00
1.05
Coverage Rate of the 90% Confidence Intervals
Cov
erag
e R
ate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Statistical Power
Effect Size
Pro
port
ion
Sta
tistic
ally
Sig
nific
ant
●
●●●●●●
●●●●●●●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
N=2000N=1500N=1000
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
N=1000 N=1500 N=2000
−0.2
−0.1
0.0
0.1
0.2
Bia
s
●
● ●
N=1000 N=1500 N=2000
0.75
0.80
0.85
0.90
0.95
1.00
1.05
Cov
erag
e R
ate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Effect Size
Pro
port
ion
Sta
tistic
ally
Sig
nific
ant
●
●●●●●●
●●●●●●●●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
N=2000N=1500N=1000
The
Div
isio
n M
odel
The
Div
isio
n M
odel
W
ith In
divi
dual
Cov
aria
tes
α level = 0.10
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 22 / 23
Concluding Remarks
Survey methodology to study sensitive questions
Endorsement ExperimentsMost indirect form of questioningApplicability limited to measuring supportAnalysis based on the ideal points frameworkMultilevel modeling to efficient estimation of spatial patterns
Design considerations:Policy positions should not be well-knownResponse distribution should not be skewedPolicies should belong to a single dimensionMeasure policy positions and political knowledge separately
Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 23 / 23