Measurement theory and provider profiling

Timothy P. Hofer MD

Dept. Medicine, University of

Michigan

VA Ann Arbor HSR&D Center of Excellence

The measurement problem

Quality

i(jk)

construct indicator

Site (clinic, hospital)

Provider (physician)

Patient

Levels of care

Indicatorsi i i

Implications of the measurement model

The indicator is a fallible measure of the construct Some indicators are less precise than others Quality indicators are very imprecise for a

variety of reasons You need to account for the measurement error

The location of the construct variability can suggest different causes, interventions and measurement procedures

Intra-class correlation(=reliability)

222

2

patientphysiciansite

physicianxR

Ability to distinguish between physicians (or sites)

single observation under a specified set of conditions of measurement.

Vol. 281 No. 22, pp. 2065-2160, June 9, 1999

MD laboratory utilization profiles

-60

-20

20

60

Dev

iatio

ns o

f MD

pro

files

from

mea

n la

bora

tory

util

izat

ion

($)

Adjusted for reliabilityUnadjusted for reliability

Table 3: Amount of variation in hospitalization and outpatient visit rates attributable

to a physician practice style effect

Variable

Age-Sex Adjusted

Casemix Adjusted

Visits

% of variation associated w/ physician

Unadjusted for reliability(R2) * 13% 10%

Reliability adjusted (ICC) † 7% 4%

Reliability‡ 0.51 0.40

Hospitalizations

% of variation associated w/ physician

Unadjusted for reliability (R2) * 8% 8%

Reliability adjusted (ICC) † 2% 1%

Reliability‡ 0.24 0.17

VA Network 11 Diabetes Care Project

Resources available

VA Diabetes Registry Project (1998-2001) Automated Clinical Databases

Data warehouse (VA Healthcare and analysis group) Database Components

Encounter records (OPC/PTF ) Outpatient Pharmacy Lab primary care provider database (PCMM () Vitals

Cohort identification procedure Data quality and measure validation

Kerr EA , et al. Journal on Quality Improvement 2002; 28(10):555-65.

Selected Measures:Resource Use

Cost of hypoglycemic medications Cost of home glucose monitoring

for patients not on insulin Cost of calcium channel blockers

Quality

ProcessesOutcomes Intermediate Outcomes

Selected Measures :Intermediate Outcomes

Last A1c value A1c 9.5% Last LDL value LDL 3.6 mmol/L (140mg/dl)

Quality

ProcessesOutcomes Intermediate Outcomes

Selected Measures:Process Measures

Hemoglobin A1c obtained LDL-C obtained Lipid profile obtained

Quality

ProcessesOutcomes Intermediate Outcomes

Selected Measures:Mixed or Linked Measure

LDL 3.6 mmol/L (140mg/dl) or

on a statin

Are there differences between physicians?

What are the sources of variation? Noise Unmeasured differences Physician effects Clinic or group effects Health System/payor effects

Noise

Physician

Clinic

System/Payor

Outcomes

Level of Care

Diabetes Care Indicator

Facility Team PCP Resource Use

Cost of hypoglycemic medications

1% 0 2%

Cost of home glucose monitoring for patients not on insulin

18% 3% 3%

Cost of home glucose monitoring for patients on insulin

8% 2% 1%

Cost of Calcium Channel Blockers

1% 0 0

Intermediate outcomes

Level of Care

Diabetes Care Indicator

Facility Team PCP Intermediate Outcomes

Last HbA1c value

12% 0 1%

Last HbA1c value 9.5%

16% 0 0

Last LDL-C value‡

7% – 1%

Last LDL-C value 3.6 mmol/L (140mg/dL)‡

2% 1% 1%

Last LDL-C value < 3.6 mmol/L (140 mg/dL) or on a statin (%)‡

2% 2% 5%

Process measures

Level of Care

Diabetes Care Indicator

Facility Team PCP Process Measures†

Hemoglobin A1c (HbA1c) obtained

- 1% 8%

Low Density Lipoprotein Cholesterol (LDL-C) obtained‡

– 2% 8%

Lipid profile obtained

7% 2% 9%

Physician effect size

Physician effect size

Pan

el s

ize

Variance attributable to level of care

.02 .04 .08 .10

0

50

100

150

200

Median PCP Panel size in study sample

Last LDL-C Value (1%)

Cost of homeglucosemonitoring forpatients not on Insulin

Last LDL-C value <3.6 mmol/L or on a statin (5%)

Hemoglobin A1c obtained (8%)

PCP Effect Negligible Small Moderate

Implicit chart review – site level

Trained physician reviewers

621 records 26 clinical sites

0.2

.4.6

.81

Rel

iabi

lity

of s

ite le

vel q

ualit

y sc

ore

0 10 20 30 40 50cases

HTN 0.17

DM 0.10COPD 0.11

Site level reliability

(to detect differences between sites)How many reviews are needed?

Conclusions

Measurement models are fundamentally important to measuring and profiling quality.

There is often little reason or capability to profile at the physician level.

Profiles that ignore measurement error Misrepresent the variability in quality Are difficult (or impossible) to validate

Example – the imprecise thermometer

Budget cuts inspire innovation in the clinic

Observed temperature

8590

9510

010

5B

ody

tem

pera

ture

(F)

observed

Observed vs. true temperature

8590

9510

010

5B

ody

tem

pera

ture

(F)

true observed

Strength in numbers

8590

9510

010

5B

ody

tem

pera

ture

(F)

true observed average

Scale transformation0

.2.4

.6.8

1P

roba

bilit

y of

pre

vent

ion

1 2 3 4 5 6Rating of preventability

Reliability

“A person with one watch knows what time it is”

“A person with two watches is never quite sure”

1991 1992

-0.4

-0.2

0.0

0.2

0.4

-0.4

-0.2

0.0

0.2

0.4

Dev

iatio

ns

of M

D p

rofil

es f

rom

mea

n H

gb

A1c

leve

ls

These physicians eliminate from their 1992

panels the patients who in 1991 had HgbA1c levels

in the top 5%.

Outlier Physicians (1991)

Non-outlier Physicians (1991)

These physicians have the same patient panels in 1992

as in 1991.

Effect of gaming