1 MEASUREMENT Measurement Error & Survey Construction

1

MEASUREMENT

Measurement Error &

Survey Construction

2

Measurement: Measurement Error in Survey Research

• What is measurement Error?

Measurement Error = Observed Score - True Score

• What are the sources of measurement error in survey research and how do we minimize it?

•One Requirement of Effective Research Design:Minimizing measurement error (Remember MAXMINCON Principle!)

3

Measurement Error in Survey Research

• Instrumentation Error/Bias• Error that results from lack of fidelity of the

measurement instrument (e.g., poor instructions, wording, scales, response options)

• Response Error/Bias• Error that stems from the mentality and/or

predisposition of study participants

Types of Measurement Error/Bias:

4

Measurement Error in Survey Research--Instrumentation Bias

Sources of Instrumentation Bias -- How to Minimize It?A. Poor Wording of Items/Instructions:

– Guideline #1: Are some words in the item ambiguous? Spell out acronyms, define difficult to understand terms, and reword the items using clearer and more commonly recognized vocabulary.

– Guideline #2: Does the question state the necessary/ appropriate frame of reference/criterion for answering? If not, the criterion must be clearly indicated.

– Guideline #3: Does the question overemphasizes/ exaggerates/dramatizes some condition. If so, it must be toned down and stated in less dramatic terms.

5


– Guideline #4: Is the question as specific as it can reasonably be? If the item is too general, state it more specifically.

– Guideline #5: When possible, avoid using negatives and double-negatives in statements--they are confusing.

– Guideline #6: Avoid putting a blank in the middle of an item.

6


– Guideline #7: Use your pronouns carefully when writing both, the items as well as the instructions.

– Guideline #8: Choose proper and clear wording and sentence structure for the items and instructions.

– Guideline #9: Is the question likely to lead respondents toward a particular answer (i.e., is it a leading question)? If so, the leading phrase must be removed.

7


B. Poor Content of Items/Questions:– Guideline#10: Is the question applicable to all respondents? If not,

reword it and/or use a detour around it for those to whom it does not apply.

– Guideline#11: Avoid branching if at all possible. The two items above can easily be rewritten as one question that applies to everyone.

– Guideline #12: Does the item contain an example that is also a possible answer? If so, change or discard the example.

8


– Guideline #13: Does the question require respondents to remember too much detail or to recall distant events? If so, it must be modified or generalized to make recall easier.

– Guideline #14: Is the question as free from threat to respondents as possible? If not, change it to reduce the threat; consider depersonalizing it and/or asking about hypothetical situations.

– Guideline #15: Does the question include more than one issue (i.e., a double-barreled item)? If so, it must be split or modified to include only a single question .

– Guideline #16: Is the question “lladed” with a reason for responding in a particular way? If so, the reason must be deleted.

9


– Guideline #17: Underline, italicize, or use bold print to draw respondent’s attention to important terms in the instructions.

C. Poor Response Format/Options:C. Poor Response Format/Options:

– Guideline #18: Make sure the list of response options is exhaustive.

– Guideline#19: Are “Yea-sayers” or “nay-sayers” likely to always choose one answer? Reword the item to avoid using “Yes” and “No” as choices.

– Guideline #20: Use “Always” and “Never” with caution; “Almost Always” and “Almost Never” may be much better substitutes.

10


– Guideline #21: Note that not all rating scales are bipolar or have to be balanced/symmetrical with a neutral mid-point (e.g., importance, likelihood, etc.).

– Guideline #22: When the respondent is to select only one choice from a list of several options, make sure the choices are mutually exclusive.

– Guideline #23: Make sure that the formats of your items and your response options match (are consistent).

11


D. Organization and Order:

– Guideline #24: Be cognizant of position/order bias.

– Guideline #25: When appropriate, use the funnel approach to avoid the potential assimilation effect. That is:

The more general first, followed by the more specific

– Guideline #26: Use duller demographic questions in a section at the end of the questionnaire (the opposite in interviews)

– Guideline #27: Use more interesting and attention-grabbing set of questions earlier in the survey.

12


– Guideline #28: Use more sensitive, personal and/or threatening items later in the survey.

– Guideline #29: Organize questionnaire by arranging items into subsections (based on content or/and response format), each with a clear and short instruction.

– Guideline #30: At the end of the survey, indicate what your respondents are expected to do next—e.g., mail the survey to whom, at what address, by when.

13

Survey Research

MEASURMENT ERROR:

SOURCES OF RESPONSE ERROR/BIAS

14

Measurement Error in Survey Research--Response Bias

Sources of Response Bias -- How to minimize it?– SOCIAL DESIRABILITY:

Response based on what is perceived as being socially acceptable or respectable.

Response Error/Bias: Error that stems from the mentality and/or predisposition of study participants.

15


– YEA- AND NAY-SAYING:Response influenced by the global tendency toward

positive or negative answers.

–ACQUIESCENCE:Response based on respondent’s perception of what would be desirable to the sponsor.

16


– THREAT:

Response influenced by anxiety or fear instilled by the nature of the question.

–PRESTIGE:Response intended to enhance the image of the respondent in the eyes of others.

17


– AUSPICES:Response dictated by the image or opinion of the

sponsor rather than the actual question.

–HOSTILITY:Response arising from feelings of anger or resentment engendered by the response task.

18

18

– ORDER:The sequence in which a series is listed affects the

responses to the items.

– EXTREMITY:Clarity of extremes and ambiguity of midrange

options may encourage extreme responses.

–MENTAL SET:Cognitions or perceptions based on previous items influence response to later ones (the “priming” problem).

Documents

1 MEASUREMENT Measurement Error & Survey Construction