View
213
Download
0
Tags:
Embed Size (px)
Citation preview
How to evaluate the cross-cultural equivalence of single items
Melanie Revilla, Willem SarisRECSM, UPF
Zurich – 15/16 July
Cross-cultural equivalence• Usually
– Discussed in the frame of cross-national research– Idea: in different countries people can express themselves in
different ways– Different cultures can also be defined on other criteria (e.g. language)– Procedure: can be applied in similar way to all kinds of different
groups
• “Equivalence” measurement equivalence – 2 persons with the same opinion will give the same answer (whatever
their group)
• Important– because observed differences may result from non equivalent
measures and not be real differences
• If measurement equivalence does not hold– cannot make comparison across groups!!
Important distinction (Northrop, 1947)
• Concept by Intuition (CI)
– Simple concepts that can be measured directly– Single item– Ex: trust in the parliament
• Concept by Postulation (CP)
– Complex concepts that cannot be measured directly– Also called “construct”– Need several CI to measure them– Ex: political trust: trust in parliament + legal system + police
…
• Classic procedure to test for equivalence for CP but not for CI
start with a reminder of the procedure for CP
Basic Confirmatory Factor Analysis model
CP1
Y1
Y2
Y3
λ11
λ21
λ31
e1
e2
e3
τ1
τ3
τ2
Yi = τi + λij CP1 + ei i = 1,2,3
Political trust
Answer Trust in the parliament
Answer Trust in the legal system
Answer Trust in the police
intercepts slopes error terms
≈ regression equation [
Dependent variable
Independent variable
Multiple Group CFA approach
Group 1 Group 2
• Multiple group: – possible to test for equality of the parameters in the
different groups – constraints across groups
• Can be extended to more groups
Different levels of invariance (Meredith, 1993)
• Configural– Same model holds in all
groups
• Metric– Configural + Slopes (λij) the
same in all groups– Sufficient for comparison of
relationships
• Scalar– Metric + Intercepts (τi) the
same in all groups– Sufficient for comparison of
means
• More: error terms, etc…
Group 1
Group 2
In practice
• Analyses can be done with standard SEM softwares– LISREL/Mplus– based on covariance matrices & means– recommended sample size: >200 in each
group– 3-step procedure: configural, metric, scalar– syntax quite easy to get estimates
• More tricky but crucial step: testing
Testing the model• Assessing global fit
– Chi2 test / Fit indices: RMSEA (<.05), CFI (>.9), etc…– Limits: Depends on sample size / Sensitive to deviations from
normality
• Assessing local fit
– Saris & Satorra should test at the parameter level + take into account type II errors (H0 not rejected despite being false)
– JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski)• See next presentation!
• Always check if estimates are really different
– Difference may be statistically significant but not substantially meaningful
• Partial invariance
– What if some indicators are equivalent but not all?– Consistent estimates of the means of the latent variables if at least
2 indicators are scalar invariant (Byrne, Shavelson, Muthén, 1989)
Single items
Y1
e1
τ1CI1 λ11
• Testing equivalence single items
• Testing equivalence for CI
Yi = τi + λij CIi + ei
Y1
e1
τ1CI1 λ11
Group 1 Group 2
Single multiple indicators?
• Problem: model just presented not identified
• “Single indicator” = single trait in fact
• But possible to use multiple methods
• So for CI:– Only one trait, but we can always have more
than one method– Several indicators = same trait asked using
different methods
Can apply again MGCFA
Group1 Group 2
• Similar at the previous model (for CP) but now different methods instead of different traits measuring a same concept
Trust in theparliament
11 points
6 points
4 points
Same procedure
• Different levels of invariance as for CP
– Configural – Metric– Scalar– etc
• Same procedure to get the estimates and test the model
– Multiple group analyses– Test of the model: global / local fit– Partial equivalence
Problem for the CI
• Fix the scale?
– As before, necessary to fix the scale of the LV– Usually, fix the first loading to 1– Can be done here too– Other loadings are relative to the first one– But need to be done in all groups– If there are differences for the method whose
loading is fixed to 1 across groups, may be problematic
– Should try to use methods that have been shown to be the most similar across groups: e.g. fixed reference points
General model
CP1
Y31
Y22
Y13
c1
c2
c3
e11
e12
e33
τ11
τ33
τ12
CI1
CI2
CI3
v21
v22
v23
u1
u2
u3
α1
α2
α3
• Even when working on CP: better to use different methods
Y21
Y12
Y23
Y33
Y11
Y32
e21
τ21
e31
τ31
e23
τ23
e13
τ13
e22
τ22
e32
τ32
1
v31
v32
v33
1
1
Equivalence single items
• Need to repeat the same item with different methods
– 3 or more repetitions
– Multi Methods (MM)?• Same persons get the question several times using different
methods • Limit: 20 minutes at least to avoid memory effects (Van Meurs &
Saris, 1990)
– Mix with Split-Ballot (SB) design? • Random assignment of respondents to different versions of the
questionnaire• “SB-MM” (CI) or SB-MTMM (CP)?
Conclusion
• Measurement equivalence can be assessed both for CP and CI using Multiple Group Confirmatory Factor Analysis
– For CP, process already well-known and used a lot– For CI, possible do similar analyses
But necessary to repeat questions!! specific data
• So testing equivalence of simple item can be done using a (SB)-(MT)MM approach– Similar to what exists in the ESS for CP: main +
supplementary questionnaires (different versions)– With extension for concepts by intuition
In summary
• To test single item equivalence
– Use multiple methods
– Do everything as for multiple items equivalence