More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research

More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research

John E. Ware, Jr., PhD, Professor and Chief

Division of Measurement Sciences, Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA

Track A - Patient Reported Outcome Measurement and Comparative Effectiveness Research to Reform: Achieving Health System Change

AHRQ 2009 Annual Conference, Bethesda MD September 13-16, 2009

2

What is the Relationship Between Health Care Expenditures &

Outcomes?

Expenditures for Health Care ($)

HealthOutcome

3

More Health Care is Not Always Better

HealthOutcome

“Flat of the Curve”

Expenditures for Health Care ($)

Health Insurance Experiment Revealed:

4

When the Same Outcome Costs More,

HealthOutcome

Payers & Consumers:

Expenditures for Health Care ($)

Want to Pay Less

5

Who is Most Vulnerable with Aggressive Cost Containment?

HealthDecline

Expenditures for Health Care ($)

Cost Containment

WellWell offYoung

Health Insurance Experiment (HIE)(1974-1981)Medical Outcomes

Study (MOS)(1986-1990)

Most vulnerable inthe MOS:• Chronically ill• Elderly• Poor• Non- white

6

4-Year Physical Health Outcomes Favored FFS > HMO for Chronically-

Ill Medicare in the MOS

Source: Ware, Bayliss, Rogers et al., JAMA, 1996; 276:1039-1047

These percentages, better & worse would be only about 5% due to measurement error

7

When Outcomes Vary at the Same Price

Expenditures for Health Care ($)

HealthOutcome

Payers & ConsumersWant the Best Outcomes

8

To Compare Health Care EffectivenessWe Need Health Outcomes “Rulers”

Expenditures for Health Care ($)

Better

Same

Worse

HealthOutcome 4

6

7

1

2

3

5

9

Continuum of Disease-specific and Generic Health Measures

9

Specific Symptoms

Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995

(1) (2) (3) (4)

1010

Specific Symptoms

Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995

(1) (2) (3) (4)

dd

Spirometry

Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all

Shortness of Breath

Continuum of Disease-specific and Generic Health Measures

1111

Specific Symptoms

Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995

(1) (2) (3) (4)

dd

Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all

How much did your lung/respiratory problems limit your usual activities or enjoyment of everyday life? Not at all A little Moderately Extremely

Spirometry Shortness of Breath

Respiratory-specific

Continuum of Disease-specific and Generic Health Measures

1212

Specific Symptoms

Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995

(1) (2) (3) (4)

dd

Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all

How much did your lung/respiratory problems limit your usual activities or enjoyment of everyday life? Not at all A little Moderately Extremely

In general, would you say your health is…

Excellent Very good Good Fair Poor

Spirometry Shortness of Breath

Respiratory-specific

Generic

Continuum of Disease-specific and Generic Health Measures

13

Specific Symptoms

(1) (2) (3) (4)

There is More to the Continuum

14

Prediction and Risk Management:

PROs are among the Best Predictors

Health-Related QOL (HR-QOL)

Future health

Inpatient expenditures

Outpatient expenditures

Job loss

Response to treatment

Return to work

Work productivity

Mortality

(3) (4)

What Do We Need for Comparative Effectiveness

Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation

15

Physical activity limitationsSymptoms of psychological distressPhysical well-beingLife satisfactionEmotional behaviorRole disability due to physical problemsPsychological well-beingGeneral health perceptionsPhysical mobilityRole disability due to emotional problemsSatisfaction with physical conditionSocial activities with friends/relatives

16

Content of Widely-Used Patient-Reported Outcome Measures

Source: Adapted from Ware, 1995

Reported health transition

SIP = Sickness Impact Profile (1976)HIE = Health Insurance Experiment surveys (1979)NHP = Nottingham Health Profile (1980)QLI = Quality of Life Index (1981)COOP = Dartmouth Function Charts (1987)DUKE = Duke Health Profile (1990)MOS FWBP = MOS Functioning and Well-Being

Profile (1992)

MOS SF-36 = MOS 36-Item Short-Form Health Survey (1992)

QWB = Quality of Well-Being Scale (1973)EUROQOL = European Quality of Life Index (1990)HUI = Health Utility Index (1996)SF-6D = SF-36 Utility Index (Brazier, 2002)

Psychometric Utility RelatedSIP HIE NHP COOP DUKE MOS

FWBPMOSSF-36

QWB EURO-QOL

HUI SF-6DCONCEPTS

Physical functioning Social functioning Role functioning Psychological distress Health perceptions (general) Pain (bodily) Energy/fatigue Psychological well-being Sleep Cognitive functioning Quality of life

PROMIS = Patient Reported Outcomes Measurement Information System

= Quality of Well-Being Scale (1973)

PROMIS

What Do We Need for Comparative Effectiveness Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation17

What Do We Need for Comparative Effectiveness Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation 18

“Ceiling Effect”

19Ware JE, Jr, et al. Med Care. 2000;38:1173-82.

Skewed 5-Item Headache Pain Measure

“Ceiling Effect”

r = 0.536N = 1016

Dynamic 5-Item HeadachePain Measure

r = 0.938N = 1016

CriterionScore

CriterionScore

A Practical Solution in 1999: Computerized Dynamic Health

Assessment

3 SD units

NoNoDisabilityDisability

IRT/CAT will spawn a new generation of

static tools

What Do We Need for Comparative Effectiveness Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation 20

Cri

teri

on

Cri

teri

on

VASVAS

What Do We Need for Comparative Effectiveness Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation 21

22

Practical Solution in 2000:Cross-Calibration of Headache Pain Disability Measures

Note: Direction of scoring shown with arrows Source: Ware, Bjorner & Kosinski, Medical Care, 2000

Scales 20 30 40 50 60 70

HDI 16 43 73 91 98 100

HIMQ 74 53 31 17 8 2

MIDAS 58 28 5 1 0 0

MSQ 31 53 79 92 96 99

DYNHA-5 (+) 23 32 41 51 58 66

Theta (θ) [Best Possible Estimate]

23

52 centimeters = 20.5 inches

and Public-Private Partnerships That Meetthe Needs of Research and Business

We Need the Health Equivalent of a Two-Sided Tape Measure

What Do We Need for Comparative Effectiveness Research?

• Outcomes that matter to patients

• Practical measures

• Coverage of a wide range

• Greater precision

• Comparability of scores

• Ease of interpretation24

What do the results mean?

25

PRO Validation Must be Comprehensive

• Diagnosis

• Disease severity

• Responders

• Treatments

MeasuresIn

Question

GoldStandard

• Work productivity• Costs of care• Mortality• Self- evaluated

health

OtherMeasures& Methods

• Diagnosis

• Disease severity

• Responders

• Treatments

• Work productivity• Costs of care• Mortality• Self-evaluated

health

GoldStandard

OtherMeasures& Methods

Adapted from: Ware JE, Jr. and Keller SD: Interpreting general health measures, in: Quality of Life and Pharmacoeonomics in Clinical Trials. Philadelphia, PA: Lippincott-Raven Publishers; 1995: Chapter 47.

Causes Consequences

26

What Do Differences in Treatment Effectiveness Mean?

30 40 50

Chronic Lung Disease

Physical Component Summary (PCS)

DiabetesType II

Congestive Heart Failure

Average Adult

Asthma After

Rx

Asthma Before

Rx

Treatment

Average Well Adult

50% reduction in disease burden 33% reduction in hospitalization Substantial increase in work productivity Subsequent cost savings

Matching Methods to Applications:“Choosing the Right Horse for the Course”

• Population monitoring

• Group-Level outcomes monitoring

• Patient-level measurement/management

27

28

1

2

3

4

5

6

7

1

3

5

7

Single-Item

1

2

3

4

5

Multi-ItemScale

PopulationMonitoring

Group-LevelOutcomesMonitoring

Patient-LevelManagement

“Item Pool” (CAT Dynamic)

7

6

Most Functionally Impaired

NoisyIndividual

Classification

Very AccurateIndividual

Classification

Matching Methods to Applications

Solutions

• Improved psychometrics (Item response theory – IRT)

• Computerized adaptive testing (CAT) software

• The Internet (and other connectivity)

29

Business Week. November 26, 2001.

30

First, Construct Better Metrics

Source: Business Week 11/26/01

• Comprehensive Item “Pools”• IRT Cross Calibration of Items

ADLSIPFIM

Physical Functioning (PF)

PF-10

+ =

NEWPF

% @ Ceiling:

1980 “PF Ruler” > 75% @ Ceiling

1990 “PF Ruler” > 30% @ Ceiling

2008 “PF Ruler” < 3 % @ Ceiling

31

RheumatoidArthritis

Physical Function (PF), Mean = 50 Source: Rose M, Bjorner JB, Becker J, Fries JF and Ware JE. Evaluation of a preliminary physical function item bank supported expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology, 2008, 61, 17-33.

6.0

5.0

4.0

3.0

2.0

1.0

0 10 20 30 40 50 60 70 80

StandardError

0.75

0.90

0.95

Reliability

PF “Criterion”(Item Bank)

PF-10 (“Static”)

PF-2 (“Static”)

PF CAT-10

PF-1 (“Static”)

Precision Varies Across “Static” and Dynamic Forms and Across Score Levels

32

CAT

2nd Solution, Assess Health Dynamically

Patient scores

here

CAT = Computerized Adaptive Testing

33

What are the Advantages of Dynamic Assessments?

• More accurate risk screening • Reliable enough to monitor individual

outcomes• Brevity of a short form –

90% reduction in respondent burden• Elimination of “ceiling” & “floor” effects • Can be administered using various data

collection technologies• Markedly reduced data collection costs• Monitor data quality in real time

35

Reference – Headache Impact: MS Bayliss, JE Dewey, R Cady etal., A.Study of the Feasibility of Internet Administration of a computerized health survey: The Headache Impact Test (HIT), Quality of Life Research, 2003, 12: 953-961

References – Asthma Control: Nathan RA, Sorkness CA, Kosinski M et al., “Development of the Asthma Control Test: A survey for assessing asthma control. Journal of Allergy and Clinical Immunology. 2004;113: 59-65.

www.amIhealthy.com www.asthmacontroltest.com

/

3rd Solution: The Internet

September 15, 2009

More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research (john.ware@umassmed.edu)

Conclusions

• Patient-reported outcomes (PROs) are very useful

• Standardization of concepts & metrics is enabling comparisons across treatments & settings

• Increasing widespread use proves that more practical tools will be adopted

• Promising technological advances include: item response theory (IRT), computerized adaptive testing (CAT) and Internet-based data capture