Predicting the quality of a survey question from its design characteristics: SQP
Preview:
Citation preview
- 1. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski (joint work with Willem Saris)
U N I V E R S I T A T P O M P E U F A B R A Predicting the quality
of a survey question from its design characteristics: SQP Daniel
Oberski
- 2. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Measurement Representation Construct Measurement Response Edited
data Validity Processing error Measurement error Inferential
population Target population Sampling frame Sample Respondents
Survey statistic Coverage error Sampling error Nonresponse error
(Groves et al. 2004). Predicting the quality of a survey question
from its design characteristics: SQP Daniel Oberski
- 3. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error
ConclConstruct Measurement Response Edited data Validity Processing
error Measurement error Predicting the quality of a survey question
from its design characteristics: SQP Daniel Oberski
- 4. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Assume the step from construct to measurement is already acceptable
Assume that the question measures an intended construct: respondent
knows the answer, can interpret the question, ... Predicting the
quality of a survey question from its design characteristics: SQP
Daniel Oberski
- 5. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Assume the step from construct to measurement is already acceptable
Assume that the question measures an intended construct: respondent
knows the answer, can interpret the question, ... reaction of
respondent to the question depends on some unobserved
value/opinion, which is in turn a measure of construct. Predicting
the quality of a survey question from its design characteristics:
SQP Daniel Oberski
- 6. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Assume the step from construct to measurement is already acceptable
Assume that the question measures an intended construct: respondent
knows the answer, can interpret the question, ... reaction of
respondent to the question depends on some unobserved
value/opinion, which is in turn a measure of construct. We focus
only on the degree to which the response is a good measure of this
unobserved score/opinion, measurement error. Predicting the quality
of a survey question from its design characteristics: SQP Daniel
Oberski
- 7. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Assume the step from construct to measurement is already acceptable
Assume that the question measures an intended construct: respondent
knows the answer, can interpret the question, ... reaction of
respondent to the question depends on some unobserved
value/opinion, which is in turn a measure of construct. We focus
only on the degree to which the response is a good measure of this
unobserved score/opinion, measurement error. (NOT the degree to
which the question is interpretable, measures some construct, etc.)
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 8. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Assume the step from construct to measurement is already acceptable
Assume that the question measures an intended construct: respondent
knows the answer, can interpret the question, ... reaction of
respondent to the question depends on some unobserved
value/opinion, which is in turn a measure of construct. We focus
only on the degree to which the response is a good measure of this
unobserved score/opinion, measurement error. (NOT the degree to
which the question is interpretable, measures some construct, etc.)
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 9. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: Predicting the quality of a survey question from its
design characteristics: SQP Daniel Oberski
- 10. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: var() = 1 2 /n, where (0, 1) is reliability Predicting
the quality of a survey question from its design characteristics:
SQP Daniel Oberski
- 11. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: var() = 1 2 /n, where (0, 1) is reliability
Unreliability reduces apparent strength of relationships between
variables: Predicting the quality of a survey question from its
design characteristics: SQP Daniel Oberski
- 12. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: var() = 1 2 /n, where (0, 1) is reliability
Unreliability reduces apparent strength of relationships between
variables: xy = x y XY , where XY is the true correlation and xy
the observed correlation. Predicting the quality of a survey
question from its design characteristics: SQP Daniel Oberski
- 13. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: var() = 1 2 /n, where (0, 1) is reliability
Unreliability reduces apparent strength of relationships between
variables: xy = x y XY , where XY is the true correlation and xy
the observed correlation. Correlated measurement errors will make
variables look more related than they really are; e.g. How many
minutes does it take to... questions correlate partly because they
are all asked in the same way. Predicting the quality of a survey
question from its design characteristics: SQP Daniel Oberski
- 14. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Reasons to study measurement error Reliability is an upper bound on
validity; responses can never measure underlying construct better
than the single indicator. Unreliability increases the variance of
estimators: var() = 1 2 /n, where (0, 1) is reliability
Unreliability reduces apparent strength of relationships between
variables: xy = x y XY , where XY is the true correlation and xy
the observed correlation. Correlated measurement errors will make
variables look more related than they really are; e.g. How many
minutes does it take to... questions correlate partly because they
are all asked in the same way. Predicting the quality of a survey
question from its design characteristics: SQP Daniel Oberski
- 15. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Public health ranking: Correction of regression coefcients for
Country Educationaldifferentialsinsubjectivehealthwith2s.e.interval
-0.4-0.3-0.2-0.10.0 GR CZ PT SI FI HU PL SK LU ES EE DK DE TR IS NO
CH BE IE FR UA AT NL SE Uncorrected regression coefficient
Measurement error-corrected coefficient 0.82 0.85 0.78 0.73 0.56
0.75 0.71 0.81 0.86 0.85 0.95 0.84 0.91 0.70 0.81 0.87 0.81 0.82
0.92 0.85 0.91 0.81 0.93 0.99 Predicting the quality of a survey
question from its design characteristics: SQP Daniel Oberski
- 16. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Design characteristics of questions Social Desirability Centrality
Reference period Question formulation WH word used Use of gradation
Balance of the request Encouragement Showcards present Showcards
have pictures ... Emphasis on subjective opinion in request
Information about the opinion of other people Use of stimulus or
statement in the question Absolute or comparative judgment Response
scale: basic choice Number of categories Labels full, partial, or
no Labels full sentences Knowledge provided Survey mode ... Order
of the labels Correspondence between labels and numbers of the
scale Theoretical range of the scale Neutral category Number of xed
reference points Dont know option Interviewer instruction
Respondent instruction Extra motivation, info or denition
available? Agree-disagree scale . . . (Saris & Gallhofer 2007)
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 17. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; Predicting
the quality of a survey question from its design characteristics:
SQP Daniel Oberski
- 18. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; It is clear
that what is a good method depends strongly on the topic, for
example Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 19. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; It is clear
that what is a good method depends strongly on the topic, for
example The frequency and importance of an event or series of
events asked about determine: reasonable reference periods;
reasonable categories - wide or deep; approximately or exactly
(Tourangeau et al. 2000). Predicting the quality of a survey
question from its design characteristics: SQP Daniel Oberski
- 20. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; It is clear
that what is a good method depends strongly on the topic, for
example The frequency and importance of an event or series of
events asked about determine: reasonable reference periods;
reasonable categories - wide or deep; approximately or exactly
(Tourangeau et al. 2000). But are some methods generally better
than others? Predicting the quality of a survey question from its
design characteristics: SQP Daniel Oberski
- 21. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; It is clear
that what is a good method depends strongly on the topic, for
example The frequency and importance of an event or series of
events asked about determine: reasonable reference periods;
reasonable categories - wide or deep; approximately or exactly
(Tourangeau et al. 2000). But are some methods generally better
than others? If so, what about those methods makes them better?
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 22. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices There are a great number of question design
characteristics for which it has at some point been found or
suggested that they inuence the response; Any question in a
questionnaire represents a series of choices (conscious or not) on
those characteristics: a method of asking the question; It is clear
that what is a good method depends strongly on the topic, for
example The frequency and importance of an event or series of
events asked about determine: reasonable reference periods;
reasonable categories - wide or deep; approximately or exactly
(Tourangeau et al. 2000). But are some methods generally better
than others? If so, what about those methods makes them better?
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 23. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Question design choices But are some methods generally better than
others? If so, what about those methods makes them better?
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 24. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl
Talk outline 1 Question design The inuence of the method Variation
in inuence of the method 2 Modeling measurement error Denitions
Formal model and assumptions 3 Estimating measurement error Design
requirements Estimation of the model 4 Predicting measurement error
Description of the data Meta-analysis of the MTMM experiments
Program demonstration Predicting the quality of a survey question
from its design characteristics: SQP Daniel Oberski
- 25. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl The
inuence of the method The method inuences the answers Predicting
the quality of a survey question from its design characteristics:
SQP Daniel Oberski
- 26. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl The
inuence of the method European Social Survey, 2002 Predicting the
quality of a survey question from its design characteristics: SQP
Daniel Oberski
- 27. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl The
inuence of the method European Social Survey, 2002 Method A: ENTER
START TIME: 1 TvTot CARD 1 On an average weekday, how much time, in
total, do you spend watching television? Please use this card to
answer. No time at all Less than hour hour to 1 hour More than 1
hour, up to1 hours More than 1 hours, up to 2 hours More than 2
hours, up to 2 hours More than 2 hours, up to 3 hours More than 3
hours (Dont know) A2 TvPol STILL CARD 1 And again on an average
weekday, how much of your time watching television is spent
watching news or programmes about politics and current affairs1 ?
Still use this card. 00 GO TO A3 01 02 03 04 ASK A2 05 06 07 88
Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 28. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl The
inuence of the method European Social Survey, 2002 Method A: ENTER
START TIME: 1 TvTot CARD 1 On an average weekday, how much time, in
total, do you spend watching television? Please use this card to
answer. No time at all Less than hour hour to 1 hour More than 1
hour, up to1 hours More than 1 hours, up to 2 hours More than 2
hours, up to 2 hours More than 2 hours, up to 3 hours More than 3
hours (Dont know) A2 TvPol STILL CARD 1 And again on an average
weekday, how much of your time watching television is spent
watching news or programmes about politics and current affairs1 ?
Still use this card. 00 GO TO A3 01 02 03 04 ASK A2 05 06 07 88
Method B:! !""#$%&'()*%)+!)&,%$# ! -&.#
!"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#)$32.4"(#
3&5&%464/"89 :## # # # # ,$/+%#/)#;!0#### ###?@A#BC@0# # #
# # # # # -&1#
#!"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#5463&"4"(#3/#
3.'$+4/8F :## # # # # ,$/+%#/)#;!G## ?@A#BC@G# # # # # # # # # # #
# -&2#
!"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#'&$+4"(#3.
"&)67$7&'688 :## # # # # ,$/+%#/)#;!G# #?@A#BC@G# # # #
#Predicting the quality of a survey question from its design
characteristics: SQP Daniel Oberski
- 29. Introduction Question design Modeling measurement error
Estimating measurement error Predicting measurement error Concl The
inuence of the method TV watching: method A versus method B 0
h