Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
The Total Survey Error Paradigm andChallenges to its Application in the Arab
World
Kathrin Thomas1 Michael Robbins1,2
1 Princeton University
2University of Michigan
5 June 2018
Where to begin...
"Honest discussions about the extent, sources of, ansolutions for quality issues [...] are needed. Regionallyspecific issues [...] raise concerns about social desirabilityand underscore the need for methodological research.Technological advances [...] o�er possibilities forreal-time monitoring and [...] research [...]. Yet apart froma handful of studies on interviewer e�ects, anchoringvignettes, and a few other topics, almost no researchsystematically assesses the impact of the surveymethods used on data quality in the Arab world."
(Benstead, 2018)
Puzzle
Is the Arab survey world just a few decades behind?Surveys and polling fairly new: beginning in 1980s,but real interest since the Arab uprising 2011(Benstead, 2018)While core principles of Total Survey Error (TSE) arefollowed, holistic concept not (yet) recognisedSurvey practice in the Arab world may also makeapplication of TSE more di�cult
The TSE
Construct
Measurement
Response
Edited Data
Population
Sampling Frame
Sample
Respondents
Survey Statistic
Validity
Measurement Error
Processing Error
Coverage Error
Sampling Error
Nonresponse Error
M E A S U R EM E N T R E P R E S E N T A T I O N
(Groves & Lyberg, 2010)
Representation
Potentially poor coverage due to past or ongoingsecurity treatsAbsence of transparent sampling frames and/oroutdated censusesLimited attention given to nonresponse andincomplete filesData fabrication, i.e, "Intentional departure from [...]sampling procedures by any member of the surveyproject" (Robbins, 2018)
Measurement
Translation: Arabic 6= Arabic 6= ArabicInterviewer e�ects on response or item nonresponse(e.g., Benstead and Maluche, 2014; Benstead,2014a, 2014b; Blaydes and Gillum, 2013; Koker,2009)Third party presence (Benstead, 2014)Data fabrication, i.e., intentionally deviating frominstructions; intentionally misrecording the answergiven by a respondent; or intentionally creating data(Robbins, 2018)
The TSE...again
Construct
Measurement
Response
Edited Data
Population
Sampling Frame
Sample
Respondents
Survey Statistic
Validity
Measurement Error
Processing Error
Coverage Error
Sampling Error
Nonresponse Error
New/different kinds of I & R Effects
Outdated Censuses;
mobile populations
Alternative Sampling Strategies
Absence of
Sampling Frames
M E A S U R E M E N T R E P R E S E N T A T I O N
Fabrication Fabrication
(Groves & Lyberg, 2010)
Data and Methods
Arab Barometer III and IVRepresentation: Contact attempts, data fabricationMeasurement: INR and response bias due tointerviewers and third party presence
Interviewer Gender
Example JordanInterviewer gender not included in the AB IV, butinterviewer names available...in ArabicThree coders (2 Arabic speakers, 1 non-Arabicspeaker) coded names into gender
κ varied from 0.37 to 0.70Non-Arabic speaker seemed to be least reliable; butalso the Arabic speakers had discrepanciesMajor challenges unisex names and context
Interviewer GenderAB IV
Interviewers in the Arab region predominantlyfemale (AB IV: 72 percent)
51
55
63
78
80
87
89
0 20 40 60 80 100Percent
Algeria
EgyptMoro
ccoTu
nisiaLe
bano
nJo
rdanPale
stine
Source: Arab Barometer Wave IV
Proportion of interviews conductedby female interviewers
Interviewer ReligiosityApprox. three quarters of the female interviewerswore a hijab during the survey interview (77.0percent)
Country Percent
Algeria N/AEgypt 100Jordan 96.3Lebanon 28.5Morocco N/APalestine 94.2Tunisia 29.1
Table: AB IV Proportion of Female Interviewers wearing aHijab
Third party presence
Almost every other interview was attended by athird person (44.6 percent)
12
24
48
55
58
59
62
0 20 40 60 80 100Percent
Leba
nonMoro
ccoEgy
ptAlge
riaJo
rdan
Tunis
iaPalestin
e
Source: Arab Barometer Wave IV
Proportion of interviews conductedwith third person presence
Representation: Coverage, Sampling Error, &Unit Nonresponse
Little information about any of these aspects of theTSE
Coverage problematic, esp. in less stable countriese.g., field report for Iraq AB III indicates that only"more than 66 percent of the population of Iraq" arerepresented
Sampling error problematic, esp. due to lack oftransparent informationNo collection/no provision of UNR filesResponse rate largely AAPOR standard, but not forall countries
e.g., field report for AB IV Palestine suggest 91percent response rate (response/response+refusal)
Representation: Contact Attempts
Example TunisiaUp to 8 contact attempts before UNR is loggedOn average, female interviewers need approximatelyone fewer contact attempt to interview comparedto male interviewersT-test reveals statistically significant di�erence incontact attempts across interviewer gender (t=2.7;p<0.01)
Representation: Suspected Data Fabrication
AB III AB IV
100 percent match 4 095 percent match 161 090 percent match 336 485 percent match 565 18
Total no. of suspicious cases 1,066 22
Total no. of observations in survey 14,809 9,000
Table: Suspected Data Fabrication AB III and AB IV
Representation: Implications
Keep more detailed records of sampling proceduresand framesRequest UNR files by contractRequest calculation of AAPOR RR by contractDevelopment of a live data checking tool that flagsup suspicious cases to prevent fabrication
Measurement: Interview Length
Example TunisiaShortest interview just over 30 minutes; longestinterview almost 2 hoursInterviews conducted by female interviewers tend tobe longer
T-test reveals statistically significant di�erence intotal interview length across interviewer gender(t=2.7; p<0.01)
Measurement: Item Nonresponse: Americansgood people? (1)
Overall, quite high INR (16.7 percent), substantivecountry variation
6
10
11
16
21
28
29
0 10 20 30 40 50Percent
Leba
nonPale
stine
Jorda
nAlge
riaTu
nisiaMoro
ccoEgy
pt
Source: Arab Barometer Wave IV
Proportion of INR
Measurement: Item Nonresponse: Americansgood people? (2)
Interviewer GenderMale interviewers appear to have higher INR(χ2=22.7; p<0.01) – all AB IV countries, but Algeria
Third Person PresenceThird person presence appears to increase INR(χ2=17.2; p<0.01) – esp. Algeria, Morocco, and Tunisia
Measurement: Interviewer Gender & Women’sRights (1)
While female interviewers seem to evoke moreagreement with the question whether women couldbecome prime minister/president (χ2=67.8, p<0.01)– esp. Jordan, Palestine, Tunisia.......male interviewers seem to evoke more agreementwhen asking if men should have the final say infamily decisions (χ2=61.6, p<0.01) – all AB IVcountries, except Tunisia
Measurement: Interviewer Religiosity & Women’sRights (1)
It seems to be more likely that respondents disagreewith the statement "women should be able tobecome prime minister/president" when the femaleinterviewer wears a headscarf (χ2 =23.4, p<0.01) –esp. Jordan, Palestine, TunisiaRespectively, wearing a hijab appears to evoke moreagreement with the statement that men shouldhave the final say in family decisions (χ2=8.5, p<0.01)– all AB IV countries, except Tunisia
Measurement: Third Person Presence &Women’s Rights (1)
Respondents seem to be more inclined to disagreewith the statement that women should be able tobecome prime minister/president when a thirdperson is present (χ2 =5.0, p<0.05) – esp. Algeria,Jordan, MoroccoBut, presence of a third person seems to lead tomore agreement with the statement that menshould have the final say in family decisions (χ2=6.3,p<0.01) – esp. Algeria, Morocco, Tunisia
Measurement: Implications
Design of a more extensive interviewerquestionnaire by the end of the survey interviewRequest for provision of core information aboutinterviewers (ID, name, gender, etc.) from the fieldagenciesInterviewer questionnaire capturing similarinformation (sealed envelope approach) atinterviewer trainingSwitch to Computer Assisted Personal Interviews in(almost) all countriesRequest for more detailed field reports detailingfieldwork procedures plus relevant records
Is the Arab survey world just a few decadesbehind?
Arab survey world is in transitionBenstead (2018) proposes an extensive agenda formethodological research on the Arab world, but arewe ready to walk the walk...yetThe AB has started collecting relevant data toempirically and systematically investigaterepresentation and measurement errorThe AB V will collect most of these data, so staytuned...
Contact Information
Kathrin Thomas:[email protected]
Michael Robbins:[email protected]
www.arabbarometer.org@arabbarometer@ArabBarometer