JWS Kappelhof

Sociaal en Cultureel Planbureau/ SCP

Measurement equivalence vs. Representativeness: The influence of interviewer characteristics and interview setting on the comparability of answers among difficult to survey populations.

Overview

• Background: difficult to survey populations in the Netherlands: Ethnic minorities

• Total Survey Error (TSE)-approach: Coding the problems with survey research among ethnic minorities

• The case study: the SIM survey• Steps to reduce nonresponse• Consequences of response enhancing measures

• Estimating the effect of interviewer characteristics and interview setting on the comparability of answers among different groups

• Considerations

ITSEW 2009, SWEDEN

ITSEW 2009, SWEDEN

Ethnic minorities in the Netherlands. Who are they?

• Government policy aimed at non-western non-natives because of societal backlog.

• Mainly focused on Antilleans, Surinamese, Moroccans and Turks for 2 reasons.

• Lower social-economic position in general.• They make up 2/3 of the total non-western

non-native population in the Netherlands.

TSE (Kish, 1964; Groves, 1989): Coding the problem areas with regard to surveying among ethnic minorities

• Measurement error

• Sampling error

• Non-observation error

ITSEW 2009, SWEDEN

Measurement error

• Interviewer• Culture• Customs

• Respondent• Social desirability• Language• Interviewer-respondent interaction

ITSEW 2009, SWEDEN

Measurement error (2)

• Instrument• Type of instrument• Length• Wording, question phrasing and

coding• Sensitivity of the topic

• Mode• Type of mode

ITSEW 2009, SWEDEN

Sampling and Non-observation error

• Sampling frame errors

• Nonresponse• Mode• Approach• Socio-demographic characteristics• Language• Attitude towards surveys

ITSEW 2009, SWEDEN

ITSEW 2009, SWEDEN

SIM survey

• Goal: to measure the integration of ethnic minorities

• Target population: Antilleans, Surinamese, Moroccans, Turks plus Dutch control group (age 14+) living in the Netherlands.

• Desired sample size: N=5000 (1000 per group). Achieved Ss (N= 5247)

• Sample frame : Municipality records of all municipalities in the Netherlands

• Sample design per group: Stratified 2-stage sample design with pps allocation.

• Fieldwork periods: 3/2006-7/2006 and 8/2006-12/2006

• Instrument: structured questionnaire

• Mode: Capi

ITSEW 2009, SWEDEN

Response enhancing measures in the SIM survey

• Questionnaire translated in Turkish and Arabic

• CAPI

• Conditional non-monetary incentives

• Interviewer bonus

• Interviewers (bi-lingual and Dutch)

• Minimum of 4 contact attempts (max 8)

• Unconditional non-monetary incentives (experiment)

• Re-contact of soft refusals

Additional measures in SIM: reducing and mapping causes for measurement error

• Simple language

• Interview control form

ITSEW 2009, SWEDEN

Response enhancing measures: Results and consequences

• Nonresponse decreased• No more language problem• Lower refusal rates • Lower non contact rates• Response more equally distributed between

groups•  But,

• Reciprocal effect of language (Turks and Moroccans)

• Cultural differences• Interview setting (alone during interview)• Interviewer effect (ethnicity, gender, gender

match) ITSEW 2009, SWEDEN

Sequential approach for estimating the effect of interview language,interview setting & sx_i

• CFA • Multiple endogenous indicators of the latent

factor• Thresholds (in case of categorical data)

• MGCFA • Test for measurement invariance

• Model “Interview language, interviewer gender & alone”

• Multiple exogenous indicators upon latent factor under investigation

• Multiple exogenous indicators upon latent factor “Interview language”.

• Feedback loop between “latent variable” and “interview language” ITSEW 2009, SWEDEN

ITSEW 2009, SWEDEN

Mun S.

Gender_r

L. Skill

Gender_i

Generat

Educat

Age

Alone

Etn_Int

FACTOR

LANG.

IND 1

IND 2

IND 3

IND 4

IND 5

Pararameter Unst. Est. (SE) (T-value) Parameter Unst. Est. (SE) (T-value)

λ11 0.73 (0.05) (14.30) * γeducat2.1 0.14 (0.06) (2.19)*

λ 21 1.00 (-----) (-------) γ educat3.1 0.25 (0.07) (3.51)*

λ 31 0.92 (0.05) (19.64) * γ educat4.1 0.45 (0.10) (4.30)*

λ 41 0.95 (0.07) (14.40) *

λ 52 1.00 (-----) (-------) β12 1.19 (0.22) (5.43)*

β21 -0.06 (0.03) (-1.71)

γ l.skill.1 -0.17 (0.04) (-4.88)*

γ l.skill.2 -0.45 (0.06) (-6.95)* Θ11 0.77 (0.03) (22.73)*

γalone.1 0.11 (0.05) (2.25)* Θ22 0.57 (0.04) (14.42)*

γalone.2 0.10 (0.08) (1.21) Θ33 0.62 (0.05) (13.37)*

γ etn_int.2 1.96 (0.11) (17.42)* Θ44 0.65 (0.04) (15.49)*

γ sx_i.1 0.15 (0.05) (2.92)* Θ55 0.00 (-----) (-------)

γ sx_i.2 -0.69 (0.09) (-7.53)* Θ23 0.08 (0.04) (2.18)*

γ sx_r.1 -0.35 (0.05) (-7.00)* Fit measures

γ sx_r.2 0.64 (0.12) (5.23)* Chisq 54.49

γag2.1 0.03 (0.06) (0.58) Df 53

γag3.1 -0.03 (0.09) (-0.36) RMSEA 0.005

γag4.1 -0.40 (0.13) (-3.09)* CI RMSEA 0.000;0.020

γms2.1 -0.14 (0.06) (-2.27)* CFI 1.00

γms3.1 -0.29 (0.07) (-4.34)* SRMR 0.023

γgenerat.1 0.06 (0.09) (0.61) *= (p<0.05)

γgenerat.2 -0.81 (0.15) (-5.42)* N 1119

Preliminary results: Direct Estimates

ITSEW 2009, SWEDEN

Mun. S.

SX_R

SX_I

Generat

Educat

Age_cat

Alone

Language

Factor

Lang.

SXINT

Alone

IND 1

IND 2

IND 3

IND 4

Considerations

• Translation errors• Heterogeneity • Nonresponse• Quasi-experimental• Sample frame errors• Simultaneity• Model limitations • Interviewer as a cluster variable• Other ways?

ITSEW 2009, SWEDEN