SPECA Regional Wrokshop on Disability StSPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006atistics, Dec 13-15, 2006

Issues Related to Issues Related to Disability Disability Measurement: Measurement: Cognitive testing and Cognitive testing and modemodeJennifer Madans for the Jennifer Madans for the Washington Group on Disability Washington Group on Disability StatisticsStatistics

Part 1: Part 1: Cognitive testingCognitive testing

Adapted from work of Kristen Adapted from work of Kristen Miller, NCHS, USA Miller, NCHS, USA

Cognitive testingCognitive testing

Designed to help us understand how Designed to help us understand how respondents comprehend, retrieve, respondents comprehend, retrieve, judge, respond to questionsjudge, respond to questions

Illustrate how aspects of respondents’ Illustrate how aspects of respondents’ lives impact their understanding of the lives impact their understanding of the questionquestion

Through this examination, can identifyThrough this examination, can identify– Potential response errors Potential response errors – Patterns of interpretationPatterns of interpretation

Components of the Components of the processprocess

A good question isA good question is:: Relevant to the Relevant to the

research agendaresearch agenda Relevant to each Relevant to each

potential potential respondents’ respondents’ experience and experience and knowledgeknowledge

RespondentsRespondents:: Are informantsAre informants Reference aspects Reference aspects

of their livesof their lives Do not know or Do not know or

understand the understand the research questionresearch question

Do not use Do not use scientific or scientific or abstract conceptsabstract concepts

Response errorsResponse errors

When questions are not optimal or When questions are not optimal or the respondent does not the respondent does not understand, error is introduced understand, error is introduced into the datainto the data– False positivesFalse positives– False NegativesFalse Negatives– An entirely different phenomena is An entirely different phenomena is

measured than intended by the measured than intended by the research agendaresearch agenda

Cognitive stages: Cognitive stages: responding to questionsresponding to questions Comprehension– the respondent Comprehension– the respondent

interprets the questioninterprets the question Retrieval– the respondent searches Retrieval– the respondent searches

memory for relevant informationmemory for relevant information Judgment– the respondent assesses the Judgment– the respondent assesses the

completeness and relevance of completeness and relevance of memories and makes an estimationmemories and makes an estimation

Response: Maps judgment onto Response: Maps judgment onto response category; may need to edit response category; may need to edit response to fit the categoryresponse to fit the category

Cognitive response Cognitive response problemsproblems

Respondent cannot remember the Respondent cannot remember the correct answercorrect answer– ““In the past month, how many times did you In the past month, how many times did you

eat rice?”eat rice?” Respondent makes an error when Respondent makes an error when

forming an estimateforming an estimate– ““What is the total value of all your financial What is the total value of all your financial

assets?”assets?” Respondent’s response does not match Respondent’s response does not match

response categoriesresponse categories– ““When you go to work, do you drive, bicycle When you go to work, do you drive, bicycle

or walk?”or walk?”

Socio-cultural factors Socio-cultural factors impact question impact question responseresponse Level of education Level of education Socio-economic statusSocio-economic status Cultural beliefsCultural beliefs Racial, ethnic, gender identitiesRacial, ethnic, gender identities Access to knowledge and Access to knowledge and

resourcesresources

Translation ProblemsTranslation Problems

Bad literal translationsBad literal translations Words vary by regionWords vary by region Formal words may not be Formal words may not be

understood understood Multiple meanings of wordsMultiple meanings of words

Cognitive InterviewsCognitive Interviews

Semi-structuredSemi-structured Core questionsCore questions

– the questions that will be testedthe questions that will be tested– interview organized by the core questions interview organized by the core questions

Probe questionsProbe questions– the questions that will provide information the questions that will provide information

about respondents’ understanding of the about respondents’ understanding of the core questionscore questions

– Probe questions are open-ended, Probe questions are open-ended, spontaneous, not pre-scripted, based on spontaneous, not pre-scripted, based on information that respondent provides information that respondent provides

Cognitive InterviewsCognitive Interviews

Data are qualitativeData are qualitative Qualitative methodologies used to Qualitative methodologies used to

analyze dataanalyze data Small samples of roughly 10 – 30 Small samples of roughly 10 – 30

participants are usedparticipants are used

ProbesProbes

Probing for storyProbing for story:: Why did the Why did the

respondent respondent answer the answer the question the way question the way that they did?that they did?

Does this story Does this story match with the match with the intent of the intent of the question?question?

Common probesCommon probes:: Why did you Why did you

answer that way?answer that way? Can you tell me Can you tell me

about that?about that? When the question When the question

asks about asks about “difficulty “difficulty walking,” how are walking,” how are you understanding you understanding that?that?

Most Common ProbesMost Common Probes

How so?How so? In what way?In what way? Can you say more?Can you say more? What do you mean by that?What do you mean by that? [Silence][Silence]

Issues related to Issues related to disabilitydisability Conceptual confusion around Conceptual confusion around

disabilitydisability Cognitive testing allows you to Cognitive testing allows you to

gain insight into what respondent gain insight into what respondent is thinking and whether this is thinking and whether this varies by socio-demographic varies by socio-demographic characteristics such as education, characteristics such as education, age, and genderage, and gender

Part 2: Part 2: Mode of data Mode of data collectioncollection Mail (paper and pencil)Mail (paper and pencil) TelephoneTelephone In-personIn-person Web-based Web-based

Issues related to Issues related to disabilitydisability Mail (paper and pencil)Mail (paper and pencil)

– Literacy levelLiteracy level– Insure availability of Braille, large print, or Insure availability of Braille, large print, or

assistant if neededassistant if needed– Cannot clarify questions if respondent Cannot clarify questions if respondent

unclearunclear TelephoneTelephone

– Presence of phone in the household may Presence of phone in the household may bias who respondsbias who responds

– TTY/TDD or assistant for persons with TTY/TDD or assistant for persons with hearing impairmenthearing impairment

– Can clarify questions if neededCan clarify questions if needed

Issues related to Issues related to disabilitydisability In-personIn-person

– May require assistant if respondent has May require assistant if respondent has communication or cognitive difficulties (allows for communication or cognitive difficulties (allows for including respondents who would usually be including respondents who would usually be excluded)excluded)

– Sign language interpreter or alternate Sign language interpreter or alternate accommodations for persons with hearing accommodations for persons with hearing impairmentsimpairments

– Can offer additional explanation of question if neededCan offer additional explanation of question if needed Web-basedWeb-based

– Presence of computer in the household may bias who Presence of computer in the household may bias who respondsresponds

– Bias in educational level of respondents who are able Bias in educational level of respondents who are able to respond to web-based surveyto respond to web-based survey

Issues related to Issues related to disabilitydisabilityLevels of interview assistanceLevels of interview assistance Direct personal interview: Respondent Direct personal interview: Respondent

participates directly; no interpretation or participates directly; no interpretation or facilitation is neededfacilitation is needed

Interpreted interview: Someone interprets the Interpreted interview: Someone interprets the questions to the respondent and interprets the questions to the respondent and interprets the responses back to the interviewer; interpreter responses back to the interviewer; interpreter is an intermediaryis an intermediary

Facilitated interview: Someone else helps the Facilitated interview: Someone else helps the respondent understand the questions and even respondent understand the questions and even answers some of the questions on their behalfanswers some of the questions on their behalf

Proxy interview: Someone else completes the Proxy interview: Someone else completes the questionnaire for the respondent because he / questionnaire for the respondent because he / she is unable to do soshe is unable to do so

Refusals and non-Refusals and non-responsesresponses Disabled people may not be seen as Disabled people may not be seen as

valid respondents by family members or valid respondents by family members or ignored and therefore not enumerated ignored and therefore not enumerated in the householdin the household

Not acceptable to have non-response on Not acceptable to have non-response on basis of disability in disability surveys - basis of disability in disability surveys - losing out on target population losing out on target population

Training of interviewers and provision of Training of interviewers and provision of support is crucial (e.g. sign language support is crucial (e.g. sign language interpreters, assistants, Braille versions, interpreters, assistants, Braille versions, etc.)etc.)