11

Getting a Grip on Assistive Getting a Grip on Assistive Technology: Technology:

Cognitive Testing Disability Cognitive Testing Disability QuestionsQuestions

Barbara F. Wilson, Barbara Altman, and Karen Barbara F. Wilson, Barbara Altman, and Karen WhitakerWhitaker

National Center for Health StatisticsNational Center for Health Statistics Vicki A. FreedmanVicki A. Freedman, Jennifer C. Cornman, Lisa , Jennifer C. Cornman, Lisa

LandsbergLandsbergPolisher Research InstitutePolisher Research Institute

Emily M. AgreeEmily M. AgreeJohns Hopkins Bloomberg School of Public HealthJohns Hopkins Bloomberg School of Public Health

22

Project GoalsProject Goals

To develop a set of instruments To develop a set of instruments for national surveys on health for national surveys on health and aging to collect information and aging to collect information on on assistive technologiesassistive technologies (AT) (AT) and the and the environmentsenvironments in which in which they are used.they are used.

33

Methodology of the Methodology of the Cognitive Testing Cognitive Testing

Recruited 28 participants with Recruited 28 participants with assorted conditions (ALS, assorted conditions (ALS, Blindness, Diabetes)Blindness, Diabetes)

Participants used a great variety Participants used a great variety of assistive technology items (e.g. of assistive technology items (e.g. cane, grab bars, hearing aid, cane, grab bars, hearing aid, walker, wheelchair, etc.)walker, wheelchair, etc.)

44

MethodologyMethodology

Cognitive interviews were conducted Cognitive interviews were conducted over the telephone from one room to over the telephone from one room to another in the NCHS another in the NCHS Questionnaire Questionnaire Design Research Laboratory (QDRL)Design Research Laboratory (QDRL)..

Interviews were videotaped with Interviews were videotaped with permission.permission.

Interviewing techniques used were Interviewing techniques used were think aloud with probes and debriefing.think aloud with probes and debriefing.

55

MethodologyMethodology

Three rounds of testing.Three rounds of testing. Revised instrument after each Revised instrument after each

round.round.

66

Selected Findings: Selected Findings:

Language and comprehensionLanguage and comprehension Response option scalesResponse option scales Reference periodsReference periods

77

Findings about Findings about comprehensioncomprehension

Some words were misunderstood, Some words were misunderstood, either because they could not be either because they could not be heard over the telephone or heard over the telephone or because they were unfamiliar because they were unfamiliar (curb cut, corridor, health, pill (curb cut, corridor, health, pill reminder, stall shower, vision)reminder, stall shower, vision)

88

Findings about Findings about interpretationinterpretation

• People couldn’t decide what constituted People couldn’t decide what constituted “convenient” public transportation if “convenient” public transportation if they couldn’t walk.they couldn’t walk.

• or “adequate” lighting if they couldn’t or “adequate” lighting if they couldn’t see well.see well.

• Couldn’t decide whether “How often do Couldn’t decide whether “How often do you walk around your neighborhood?” you walk around your neighborhood?” was intended to find out if participant was intended to find out if participant socializedsocialized or or exercisedexercised..

• One man who uses a wheelchair said, “I One man who uses a wheelchair said, “I rollroll around my neighborhood.” around my neighborhood.”

99

Findings about response Findings about response optionsoptions

• Many participants did not Many participants did not understand that they were being understand that they were being offered five possible response offered five possible response options and should choose one.options and should choose one.

1010

Transcript of one Transcript of one questionquestion

““You said there is a computer in your home. You said there is a computer in your home. And how hard or easy is it for you to do the And how hard or easy is it for you to do the following…is it Very hard, Hard, Not hard or following…is it Very hard, Hard, Not hard or easy, Easy, or Very easy to use the keyboard?”easy, Easy, or Very easy to use the keyboard?”

Shrugs shoulder…“It’s OK.”Shrugs shoulder…“It’s OK.” ““And so you say that it’s ‘Not hard or easy’?”And so you say that it’s ‘Not hard or easy’?” ““It’s Not hard.”It’s Not hard.” ““Is it Easy?”Is it Easy?” ““Yes.”Yes.” ““Is it Very easy?”Is it Very easy?” ““It’s Easy. That’s what it is. Easy.”It’s Easy. That’s what it is. Easy.”

1111

Findings about Findings about response optionsresponse options

• Many participants did not understand Many participants did not understand that “Not hard or easy” was a single that “Not hard or easy” was a single option, the third of five. option, the third of five.

• To solve this, the response options To solve this, the response options could be read aloud with numbers could be read aloud with numbers preceding.preceding.

• This was a problem over the telephone. This was a problem over the telephone. In another mode (self administered by In another mode (self administered by computer or paper and pencil), this computer or paper and pencil), this may not have been a problem.may not have been a problem.

1212

Response optionsResponse options

Participants did not naturally use Participants did not naturally use the requested metric.the requested metric.

1313

Transcript of second Transcript of second questionquestion

““And how hard or easy is it to see And how hard or easy is it to see the screen?”the screen?”

““I see it very wellI see it very well.”.” ““Would you say then it’s Easy to Would you say then it’s Easy to

see the screen?”see the screen?” ““Yes.”Yes.” ““Is it very easy?”Is it very easy?” ““Very easy.”Very easy.”

1414

Response optionsResponse options

Questionnaire designers intended Questionnaire designers intended that stem of question with five that stem of question with five response options be carried forward response options be carried forward to four follow-up questions. to four follow-up questions. However, questions 2, 3, and 4 However, questions 2, 3, and 4 abbreviated the scale to just “hard abbreviated the scale to just “hard or easy” . Interviewer had to or easy” . Interviewer had to repeat options to arrive at a valid repeat options to arrive at a valid answer.answer.

1515

Transcript of third Transcript of third questionquestion

““And how hard or easy is it to And how hard or easy is it to control the Pointer?”control the Pointer?”

““Easy.”Easy.” ““Very easy?”Very easy?” ““Very easy.”Very easy.”

1616

Transcript of fourth Transcript of fourth questionquestion

And how hard or easy is it to And how hard or easy is it to follow the instructions?”follow the instructions?”

““You mean Email?”You mean Email?” ““Yes.”Yes.” ““I learned it, and it’s no I learned it, and it’s no

problem nowproblem now.”.” ““So it’s Very Easy?”So it’s Very Easy?” ““Yes.”Yes.”

1717

Problem with bipolar Problem with bipolar scalesscales

One woman said that a numbered One woman said that a numbered scale would be easier for her than scale would be easier for her than a a

5-point 5-point bipolarbipolar verbal scale from verbal scale from Very hard to Very easy. Very hard to Very easy.

In asking for a numbered scale In asking for a numbered scale from one to five she is asking for from one to five she is asking for a a unipolarunipolar one-directional scale. one-directional scale.

1818

Time framesTime frames

Cognitive testing showed that it was Cognitive testing showed that it was important for questions to include a important for questions to include a reference period (In the past 30 reference period (In the past 30 days) for use of assistive technology days) for use of assistive technology items.items.

Without the time frame, people Without the time frame, people offered long lists of equipment that offered long lists of equipment that was no longer used, confounding was no longer used, confounding subsequent questions.subsequent questions.

1919

Frequency of AT useFrequency of AT use

In the first round, frequency of In the first round, frequency of use was asked generically for all use was asked generically for all AT items, beds, canes, hearing AT items, beds, canes, hearing aids. The original question was: aids. The original question was: “How often do you currently use “How often do you currently use itit …All the time, some of the …All the time, some of the time, or never? time, or never?

2020

Frequency of AT use, Frequency of AT use, contd.contd.

When asked about a cane, this When asked about a cane, this question got responses such as “90 question got responses such as “90 percent of the time, except when I percent of the time, except when I sleep, eat or read the paper” or “most sleep, eat or read the paper” or “most of the time”. of the time”.

Good answers, but not in the Good answers, but not in the requested metric, which was All the requested metric, which was All the time, some of the time, or never.time, some of the time, or never.

2121

Frequency of AT use, Frequency of AT use, contd.contd.

When asked about a hospital bed, When asked about a hospital bed, “How often do you currently use “How often do you currently use it…All the time, some of the time, it…All the time, some of the time, never?”never?”

Response: “Am I in bed 24 hours Response: “Am I in bed 24 hours a day?”a day?”

2222

Frequency of AT use, Frequency of AT use, contd.contd.

Cognitive testing showed that, in Cognitive testing showed that, in order to make sense, the order to make sense, the frequency of using an AT item frequency of using an AT item (e.g. hospital bed) had to be (e.g. hospital bed) had to be linked to a specific activity (e.g. linked to a specific activity (e.g. sleeping). sleeping).

2323

Questions to develop Questions to develop Measures of Measures of EffectivenessEffectiveness

EffectivenessEffectiveness Concepts: Concepts:– Participation Participation – SafetySafety– ControlControl– IndependenceIndependence– Pain, fatigue, extra time Pain, fatigue, extra time

needed to do thingsneeded to do things

2424

Original Original EffectivenessEffectiveness QuestionsQuestions

Response options offered Response options offered Balanced, forced choice, 5-point Balanced, forced choice, 5-point bipolar semantic differential scale:bipolar semantic differential scale:

““Please tell me whether you Please tell me whether you strongly disagree, disagree, strongly disagree, disagree, neither agree nor disagree, agree, neither agree nor disagree, agree, or strongly agree with each of the or strongly agree with each of the following statements.”following statements.”

2525

Example: One of 15 Example: One of 15 Effectiveness Effectiveness questionsquestions

““Because I use these items, Because I use these items, I I have less painhave less pain than I used to. than I used to. Do you Strongly disagree, Do you Strongly disagree, Disagree, Neither disagree nor Disagree, Neither disagree nor agree, Agree, or Strongly agree ?”agree, Agree, or Strongly agree ?”

2626

ConclusionsConclusions

Researchers and QDRL staff jointly Researchers and QDRL staff jointly reviewed tapes from first two rounds.reviewed tapes from first two rounds.

The project demonstrated that The project demonstrated that telephone surveys of elderly persons telephone surveys of elderly persons present special problems.present special problems.

The overly long survey was shortened The overly long survey was shortened by dropping unworkable questions.by dropping unworkable questions.

Synonyms were found for Synonyms were found for misunderstood terms.misunderstood terms.

2727

Conclusions, contd.Conclusions, contd.

Purpose of confusing questions was Purpose of confusing questions was clarified.clarified.

Bipolar response option scales were Bipolar response option scales were replaced with unipolar scales that replaced with unipolar scales that included a “Does not apply” option.included a “Does not apply” option.

Assistive technology questions were Assistive technology questions were limited to 30-day reference period.limited to 30-day reference period.

Third round went smoothly.Third round went smoothly. Revised instrument will be pilot tested.Revised instrument will be pilot tested.

2828

Balanced, forced choice, Balanced, forced choice, 5-point bipolar scales5-point bipolar scales

Meaning of the midpoint was Meaning of the midpoint was unclear. The midpoint might mean unclear. The midpoint might mean that they that they neitheneither disagreed nor r disagreed nor agreed, that they agreed, that they bothboth disagreed disagreed and agreed, or that and agreed, or that sometimessometimes the the statement was true and sometimes statement was true and sometimes false. false.

Moreover, for some items “Neither Moreover, for some items “Neither agree nor disagree” just did not fit.agree nor disagree” just did not fit.

2929

ExampleExample ““I have less pain than I used to.”I have less pain than I used to.” One participant said she never One participant said she never

had pain. If she had pain. If she agreedagreed that she that she had less, then it would seem as if had less, then it would seem as if she previously had pain.she previously had pain.

If she If she disagreeddisagreed that she had less that she had less pain, it would sound as if the pain, it would sound as if the Assistive Tech did not help. Assistive Tech did not help.

3030

Problems with forced Problems with forced choice,Disagree/Agree, choice,Disagree/Agree,

response scaleresponse scale There was a hidden assumption There was a hidden assumption

that there had been pain. that there had been pain. She wanted the option to say She wanted the option to say

“Does not apply.”“Does not apply.”

3131

Revised question Revised question (Unipolar, 3 point, (Unipolar, 3 point, nonforced choice)nonforced choice)

““Because you use these items, Because you use these items, how much less painful is it for you how much less painful is it for you to do your daily activities….. to do your daily activities…..

No less, a little less, or a lot less? No less, a little less, or a lot less? Or does it not apply?”Or does it not apply?”

3232

ConclusionConclusion

Researchers and QDRL staff Researchers and QDRL staff reviewed tapes from first two reviewed tapes from first two rounds.rounds.

Researchers revised, deleted, or Researchers revised, deleted, or fine-tuned the questions.fine-tuned the questions.

Third round went more smoothly.Third round went more smoothly. Revised instrument will be pilot Revised instrument will be pilot

tested.tested.