Q-BANK: Development of a Tested- Question Database

Questionnaire Design Research Lab

Office of Research and MethodologyNational Center for Health Statistics

Cognitive Interviewing

at NCHS

A Brief Overview

Questionnaire design problems

Some possible threats to accuracy derived from questionnaires: Questions not understood as intended Don’t adequately capture respondent

experience Pose an overly challenging response task Don’t provide a legitimate option for an answer

Questionnaire design problems (2)

Errors can be systematic (bias) or random (noise; high variance)

Problems may not be visible in the actual data, inviting delusions about the quality of self-reports that have been collected

“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward

“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:

What do people think of as their “abdomen”?

What, to you, is your abdomen?

“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:

What do people think of as their “abdomen”? What sort of threshold do people use for

answering “yes”?

“In the last year, have you been bothered by pain in the abdomen?” Seems to be straightforward But suppose we ask:

What do people think of as their “abdomen”? What sort of threshold do people use for

answering “yes”? What period of time are people thinking about

while answering?

“In the last year, have you been bothered by pain in the abdomen?” Possible revisions:

Show shaded picture of abdomen Drop “bothered” Use “In the past 12 months”

“In the past 12 months, have you had pain in the abdomen? By abdomen, we mean the shaded area in this picture.”

Clear alternatives address these problems, with no apparent drawbacks

Shaded picture of the abdomen

What is cognitive interviewing? 1) An administration of a draft

questionnaire…2) With additional verbal information

collected about the responses…3) And used to either:

Evaluate the quality of responses Or help to determine whether the question is

generating the sort of information its author intends.

Within these general parameters, there are a number of variants:

Practices of cognitive interviewing The “purist” version– encourage

participants to think out loud while answering questions; interviewer “prompts” participants without much other activity.

Practices of cognitive interviewing (2) Newer versions rely much more heavily on

direct probing. But even so, there can be much variation: Probes scripted or based on

interview/participant particulars (more “active” role for the interviewer)

Probes that are “cognitive” or more narrative in nature

A relatively realistic administration of the questionnaire (with debriefing at the end) or intensive discussion throughout

Some general practices We generally opt for:

Designing a semi-scripted interviewing protocol. It asks some probes in anticipation of potential problems, but allows improvisation.

Selecting a group of interviewees who would be typical recipients of the questionnaire and collectively cover a range of situations. Get a demographic/educational mix as much as possible.

Considering the interviewers to be investigators and not simply data collectors (methodologists with subject matter knowledge).

What sorts of probes should you ask?Consider this simple model of the response

process as a starting point:

Comprehension: Respondent interprets the question

Retrieval: Respondent searches memory for relevant information

Estimation/Judgment: Respondent evaluates and/or estimates response

Response: Respondent provides information in the format requested

Probes in cognitive interviews

Traditionally the majority of attention focuses on comprehension

Direct comprehension of terms: What does 'dental sealant' mean to you?"

But we also find that the most efficient way to learn about frame of reference and interpretation is to ask for narrative about the basis for the respondent’s answer Why did you answer that way? You said ___; tell me more about that.

Or, ask for paraphrase of the question.

Probes in cognitive interviews (2)To get at other aspects of the response process: Recall: How did you figure your answer to that? How

do you remember that? Confidence: How certain are you about that? (can be

used to identify judgment issues/uncertainty) Response categories: How did you pick an answer to

that? Was it difficult to find a response category? You can’t go wrong with:

“Tell me more about that?” “How so?”

The end result Qualitative analysis follows one or more rounds of

interviews Project reports include rich data on

Participant experiences and how these line up with question wording

Participant interpretations of the questions Problems participants had answering Potential ambiguities, trade-offs involved in asking

question a certain way Reports provided to sponsor, including

recommendations for changes when appropriate Then what?

Q-BANK Origination 2001 QUEST conference

Losing knowledge Repeat testing

Lack of “place” Implications:

Personal Administrative Survey Research

Impact on Question Evaluation Inability to compare studies

Hinders methodological conclusions Prevents insight Stagnant method

Limited use to survey research

Database Structure National Center for Health Statistics

original developers1. Access to past evaluation projects2. Methodological research in question design

Q-BANK houses: Individual questions Test reports

Test findings Methods used

Question characteristics

Question Subject Matter Numerous, open categories Accessed by drop-down list Examples:

Demographic: Income, Race, Gender, NationalityHealth: Insurance, Chronic conditions, SmokingEmployment: Status, Occupation, Job change,

Salary, Benefits

Question Information Type1. Events, Actions and Behaviors2. Objective Characteristics3. Subjective Characteristics4. Speculations5. Attitudes/Opinions6. Knowledge Tests7. Other

Response Category Type

1. Yes/no2. Select one textual3. Select one numeric4. Mark all that apply5. Allocation6. Open-ended textual: Delineated & Non-

delineated7. Open-ended textual: Delineated & Non-

delineated8. Other

Response Error Indicator

1. Interviewer difficulty2. Visual design problem3. Missed instructions4. Unknown terms5. Ambiguous concepts 6. Overly complex7. Assumption8. Double barrel9. Questionnaire effects10. Recall & estimation error11. Inadequate response options12. Problematic answer space13. None as tested

Value of Q-BANKSurvey Research Tool for:

Questionnaire Designers Subject Analysts Survey Methodologists

Interagency Effort Co-Sponsors:

Office of Behavioral and Social Science Research,

Bureau of the Census, National Science Foundation, National Cancer Institute, Bureau of Labor Statistics

NCHS provides internet access

Current Status On-line to steering committee members Testing and refinement Broader access possibly by end of year Looking for sponsors and partners

ChallengesPast Integrating interagency needs

Additional fields for establishment surveys Additional fields for self-administered surveys Reworking existing categories: response error

typologies, question type typologies, question topic categories

Establishing procedures Methodological requirements of evaluation

reports

ChallengesCurrent Coding consistency Populating the database Staffing limitations Funding limitations

ChallengesFuture Maintaining a methodological research

agenda Continual eye toward development

Searching in Q-BANK- Agency Name- Survey Title- Response Error- Advanced Searches

Q-BANK Search Fields

Agency Name Search

Agency Name Search

Agency Name Search

Q-BANK Reports

Searching on Survey Title

Searching on Response Error

Advanced Searches

Advanced Searches

Cognitive Testing Final Reports Don’t get lost in the weeds Important information about a tested

question is lost without referencing the final report

Conclusions about specific questions can be distorted without the final report

Contents of a Typical Report Introduction Method Procedures Summary of findings Question-by-question analysis

Why refer to the final report?

Search categories may not be straightforward Example: “At this time, is (CHILD) covered by

health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question

Why refer to the final report?

Search categories may not be straightforward Example: “At this time, is (CHILD) covered by

health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question

The interpretive process and underlying construct can’t be gleaned from the search category Example: “Have you ever taken any herbal

supplements for yourself?” Type of response error = ambiguous concept

Final Reports Q-Bank is a good tool for quickly finding

relevant reports of different cognitive testing projects

Searches not meant to replace final reports

Questions?

Kristen MillerKSMiller@CDC.GOV