Developing a Context-Sensitive Approach to AT Outcomes Measurement

Developing a Context-Sensitive Approach to AT Outcomes Measurement

OT 541Fall 2003

James A. Lenker, MS, OTR/L, ATP

What are Outcomes?

Outcomes are the result of an intervention (Scherer, 1998)

Outcomes answer the question, "Does it work?" (Oldridge, 1996)

Outcomes are the impact produced by AT devices in the lives of users and their environments (Fuhrer, 2001)

What is Outcomes Measurement?

"The evaluation process in the service delivery system that is designed to measure and establish a baseline of what works; how well something works; for which clients it works; and at what level of economic efficiency it works” (DeRuyter, 1997)

Purposes of Outcomes Measurement

Funding justification for equipment and services (assessment & training)

Measure individual's change over time

Program evaluation: "Who do we serve? How well do we serve them?"

Domains of Outcomes Measurement

Many domains of inquiry can be considered an "outcome" Medical status Activity performance Quality of Life (QOL) impact Community impact AT usage & usability

AT Outcomes Research – Developmentally Speaking

Blissful ignorance (Lenker, 2000)Denial ( “ )Exhortative (Fuhrer, 1999)Sober appraisal ( “ )Infrastructure building ( “ )Model demonstration ( “ )Getting on with it ( “ )

AT Outcomes Research: Challenges

Heterogeneity of…. End users: age, abilities, needs, support AT devices Contexts of use Reimbursement models Practitioner skills

Limitations of Previous Studies

Incomplete information about reasons for device abandonment

Inconsistent definitions for prevalence of use that have been difficult to compare across studies because use and abandonment are often defined differently

Changes in functional status when evaluating long-term device use and disuse, which bias the results toward high rates of ‘abandonment’.

Limitations of Previous Studies

Incompletely defined outcomes indicators (e.g. comfort, safety, and function), hindering replication by others

Incomplete information about device success

Measurement tools that were not sensitive to improvements in function

Low numbers of participants

Research Needs

Measurement of impact on social participation and on work (Fuhrer, 1999)

Consideration of ATD usage within context of use service delivery model, and user characteristics (Fuhrer, 1999)

Methods that embrace the user's perspective and assess impact on quality of life (Fuhrer, 1999; Jutai et al., 1996; Scherer, 1998)

Approaches that evaluate underlying factors contributing to overall success/failure of AT interventions (De Ruyter, 1994; Scherer, 1996)

Research Needs (con’t.)

Measurement of the hassle experienced by end users, comparing trade-offs against the functional gains (Sprigle & Abdelhamied, 1998)

Measurement for reduced dependence (Fuhrer, 1999)

Description of devices that are achieving desirable outcomes, which would provide the rationale for funding ATD devices and/or training services (Scherer, 1996).

The Chain of Outcomes Research (Keith, 1995)

Conceptual Models => relationships among domains

Domains => measurement toolsMeasurement tools => dataResearch design links treatment to

outcome, i.e. interpretation of data, i.e. “outcomes”

Outcomes influence practice and help build theory

So, Where Do I Start?

Know the context of AT useIdentify all relevant stakeholdersDevelop specific research questionsIdentify indicators that will answer your

questionsIdentify measurement tools

Useful Conceptual Perspectives

Quality of LifeProduct UsabilityFunctioning and HealthHAAT: Human, Activity, Assistive

TechnologyMPT: Matching Person & Technology

Quality of Life: The Bottom Line

Definition: “(Quality of Life is) the degree to

which the person enjoys the important possibilities of his or her life” (Renwick & Brown, 1996).

Quality of Life (Renwick & Brown, 1996)

Quality of Life

Physical• Personal safety • Health• Fitness• Mobility

Social• Relationships family, friends• Community

Material• Finance• Housing• Transport• Security & tenure

Development & Activity• Competence independence, choice & control• Productivity job, education home life, leisure

Emotional Well Being• Mental health• Self-esteem• Faith• Sexuality• Fulfillment

Product Usability Perspective

Effectiveness Does it achieve the desired goal?

Efficiency What are the costs associated with

achieving the desired goal? Ex: $$, time, speed, errors effort, comfort, pain, hassle

User satisfactionInternational Standards Organization. (1993). ISO CD 9241-11.2

Smith’s Parallel Interventions Model

Reduce the Impairment (Therapy)

Change technique / Increase Skill

Use AT Devices and Services

Redesign the Activity

Redesign the Environment

Use Personal Assistance

ICF – AT Model (Lenker & Jutai, 2002)

HAAT: Human, Activity, Assistive Technology (Cook & Hussey,

2002)

CONTEXTSocial ContextFamilyPeersStrangersSettingHomeSchoolEmploymentCommunityPhysical ContextSpaceLightSoundTemperature

MPT:Matching Person &

Technology (Scherer,

1998)

Getting Down to Business

What question(s) do you want to answer? This the most important consideration when

designing your study & picking a toolWho are the stakeholders?

Consumers, clinicians, administrators, manufacturers, policy makers

What is your conceptual orientation?What are your resources?

Develop Specific Research Question(s)

Must be a match between Stakeholder interests Reimbursement model User characteristics AT device Context of use

Identify Stakeholders

Who is your target audience? End users Practitioners / direct service providers Program administrators Manufacturers of AT devices Funding / policy agencies

State VR agenciesFederal agencies that sponsor rehabilitation

R&D programsState, local departments of Education A

Describe Context of Use

Reimbursement Model Educational, vocational, medical Practitioners involved in service delivery User involvement in service process

AT User Physical, cognitive, learning, sensory Ages and environments of use Needs / goals / roles

Context of Use (con’t.)Assistive Technology Devices

Wheelchair mobility, postural support, seating Adaptive computer software and hardware Alternative & augmentative communication aids Job accommodations, often ‘ergonomic’ Aids for daily living activities

Important!! AT devices are uniformly defined but they should not be uniformly studied

What data will answer your research question?

The domain of data may include Functional performance, QOL impact, pain,

usability, user satisfaction, etc.The perspective of data may be

objective (by observation) subjective (self-reported)

The nature of data may be quantitative (data are numeric) qualitative (data are language-based)

Outcomes research requires measurement tools!

The most important consideration when picking a tool? Knowing what question (s) you want to

answer

What other considerations matter when choosing a measurement tool?

Does it measure what you want it to measure? Does it measure what it purports to measure? Is it easy for participants to answer consistently?

Test-retest reliability Is it easy for multiple observers to rate

consistently? Inter-rater reliability

What demands does it place on respondents? What demands does it place on administrators?

Is tool-specific training required? Is it time consuming to administer? How difficult is it to score the results?

Accessibility of format for people with disability?

A Useful Model: ICF

Presents a functional continuum of human performance in life situations

Provides framework for outcomes research Interventions Reimbursement / service delivery models Assessment tools Outcomes questions

ICIDH-2 / WHO Model in the Context ofReimbursement Models and Assistive Technology Interventions

ICIDH-2Domain

Body Functions and Structures (Body level):  

Activities (Individual level):  

Participation (Role level):  

Contextual Factors (Societal level):  

Domain-relevant examples

Functional and structural integrity vs. impairment. Normal physiologic processes and organ systems function:Cognition, sensory, perception, motor, physiologic and psychological processes 

Performance of action or activity (cognitive, sensory, motoric, and/or behavioral) that facilitates performance in multiple roles: - initiate, organize, sequence, judge, attend- sit, lift, stand, climb, reach, pinch, grasp, hold, release- relate, interact, cope- hear, speak, see, taste move- read, write, learn, understand 

Involvement in life situations and performance in roles (family, work, and educational; self-care; leisure) that take place in specific contexts (physical, social, cognitive).

Features of the physical and social world that affect the individual but are not specific to the individual's ability or disability; e.g. physical barriers in public spaces, employer attitudes; funding for social service benefits; public policy;

Domain-relevant Assistive Technology

Heart pacemaker, hip/knee replacement, cochlear implant, baclofen pump

Mobility aids: wheelchairs, crutches, canes, walkers; UE orthotics; eyeglasses, hearing aid, AAC device

Job modifications of-task/device/ environment; home modification;vehicle modification; computer-based AT

Accessible public venues: buildings, transportation, communications, recreation; 3rd party reimbursement for AT services & equipment 

Funding Model

Medical Rehabilitation Vocational RehabEducationalIndependent LivingEmployer

Independent LivingPublic Policy

 Adapted from:Baum, C. M., & Law, M. (1997). Occupational therapy practice: Focusing on occupational performance. American Journal of Occupational Therapy, 51, 286.Gray, D. B., Quatrano, L. A., & Lieberman, M. L. (1998). Moving to the next stage of assistive technology development. In D. B. Gray & L. A. Quatrano & M. L.

Lieberman (Eds.), Designing and Using Assistive Technology: The Human Perspective (pp. 299-309). Baltimore, MD: Paul H. Brookes.Moyers, P. A. (1999). The guide to occupational therapy practice. American Journal of Occupational Therapy, 53, 247-322.World Health Organization. (1999). ICIDH-2: International classification of functioning and disability. (Beta-2 draft). Geneva: Author.

Example Research Questions

Do computer-based ATDs improve the psychological and emotional well-being of end-users?

Do computer-based ATDs increase end-user

participation in work, school, and/or leisure activities?

Do users with clearly defined goals become more successful user of computer-based ATDs?

Does impact of ATD use vary across disability populations?  

Research Questions (con’t.)

Are there societal barriers that preclude or interfere with participation in school, work, or leisure activities?

Are there significant technical problems with computer-based ATDs that are limiting performance for users of these products?

Are there any themes or variables that are associated with successful or unsuccessful computer-based ATD use?

Indicators of AT “Outcome”

Psychosocial impact on the individual User satisfaction with device User satisfaction with services Achievement of user goals Prevalence of long-term AT use Tasks and roles in which AT is used Environmental influences that contribute to

functional AT use/abandonment Unmet technology needs Employment / educational success

AT Outcome Measurement Tools

PIADS: Psychosocial Impact of Assistive Device Scale

QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology

COPM: Canadian Occupational Performance Measure

MPT: Matching Person & Technology

PIADSAttribute: Quality of Life

3 sub-scales: Competence, adaptability, self-esteem

26 item, 7-point, self-reportStrengths

Not device-specific Identifies potentially useful points of departure

for exploration Quick, easy to administer Useful retrospectively or pre/post

PIADS

Additional considerations Items are abstract, some users will

struggle -- especially kids/adolescents Better to administer in person Open-ended probing questions can

provide insight about specific scores

COPM

Semi-structured interview format, approx. 30-40 minutes

Process Identify problem areas Prioritize 5 areas of greatest importance User rates each of the 5 areas on

Current performance levelSatisfaction with that performance level

COPM

Strengths User centered

Philosophically soundEffective as intervention approach; increased

consumer ‘stake’ in the outcome (?) Sensitive to clinically significant changes in

performance and satisfaction with performance Useful for variety of devices and contexts of use Outcome indicators are built-in to the

assessment process

COPM

Additional considerations Goals are likely to be interviewer-

dependent, which will affect inter-rater reliability

The specificity of goals to the end user's circumstances hinders generalizability of results to other users

Goals may not suit all stakeholder perspectives

QUEST

12 items, 5-point scale Device satisfaction Services satisfaction

Quick, straightforward to administer (10-15 mins.)

Suitable for phone or distance surveysSuitable for wide range of AT products and

user demographics

QUEST

Additional considerations Expectations often color the perceptions

of satisfaction, therefore expectations are a potential confounding factor

Satisfaction judgments are best supplemented with information that explains underlying predictors

User performance, or function, with use of AT device is not captured

MPT

Several instruments Survey of Technology Use ATD Predisposition Assessment (PA) Educational Technology PA Workplace Technology PA Health Care Technology PA

ATD Predisposition Assessment

Two perspectives are captured Professional assesses ...

Match between user abilities and ATD requirements

Psychosocial incentives/disincentives to AT use End user self-rates ...

Sensory & motor capabilitiesLife satisfactionPerceived value of device

MPT

Additional considerations AT ‘benefits’ to user are not explicitly captured Professional’s ratings of psychosocial incentives

and disincentives to AT use require knowing the consumer really well -- beyond the scope of a typical AT evaluation period

Professional’s ratings of psychosocial factors is subjective, subject to bias

Overall, does a fine job of identifying factors that could predispose user to frequent or infrequent use

Conclusion

Acknowledge the futility of identifying a single “best” AT outcomes tool that is appropriate for all contexts and research questions.

Accept the challenge of matching research methods and measurement tools to your research area