Embed Size (px)
Citation preview
Marion R. Sills, MD, MPHSAFTINet co-Investigator
LEVERAGING SAFTINET RESOURCES TO ENHANCE VALUE IN PERFORMANCE MEASUREMENT
Questions• In general, what makes a performance measure more/less actionable? (or valuable)?• What makes UDS adult weight measure, specifically, more/less
actionable?• How generalizable to other UDS, meaningful use measures?
•What does actionable usually mean? • What kinds of actions would you take? • How would we detect action/follow-up?
•What features of a measure or a report would affect actionability?
Value of Performance Measures• Attributes of performance measures related to their value* include:• the relevance of the measure to a topic of importance* that
will inform important* healthcare decisions • the scientific soundness of the measure, including validity,
accuracy and reproducibility• the feasibility of the measure, including logistical and financial
burden
• relevance• scientific soundness• feasibility*to stakeholders
Limitations of Current Measure• Relevance• Is the measure easily interpreted? • What is the significance of the measure to stakeholders? • What are the policy and financial implications of implementing
the measure? Does it encourage activities that use resources efficiently to maximize health?• What is the prevalence and overall impact (health, financial) of
the condition in the population? {selected for this in choosing measure}• What control does the measured entity have over the
condition? • Will there be wide variations across systems?• How much room is available for measured entities to improve
performance? • relevance• scientific soundness• feasibility
Limitations of Current Measure• Scientific soundness• Does the measure precisely evaluate what is actually
happening? What measures of data quality are reported?• What is the strength of the evidence linking the clinical
processes and outcomes that the measure addresses?• Does the measure produce reproducible results when repeated
in the same population and setting?• Does the measure make sense logically and clinically?
(validity)• Is it appropriate to risk-adjust the measure by age or some
other variable?• Are accuracy, reproducibility and validity consistent across
different data systems and settings? • relevance• scientific soundness• feasibility
Limitations of Current Measure• Feasibility• Does the measure impose an inappropriate burden on health
care systems?• Does the measure have clear specifications for data sources
and methods for data collection and reporting?• Does the data collection violate accepted standards of
member confidentiality?• Is the required data logistically feasible to access?• Is the measure susceptible to manipulation that would be
undetectable in an audit?
• relevance• scientific soundness• feasibility
Selected UDS Measure, Table 6B
Limitations of Current Measure• Relevance• Is the measure easily interpreted?
• relevance• scientific soundness• feasibility
UDS process measure
Composite measure obfuscates what portion of the compliance rate reflects documentation of• BMI• follow-up• underweight or
overweight• by age• by type: nutrition,
exercise
Limitations of Current Measure• Parsing the composite measure may help stakeholders identify which components contribute to the compliance rate• Easier to interpret
Adults with > 1 visit
BMI documented
BMI > 30
Follow-up documented
UDS process measure
Limitations of Current Measure• Relevance• What is the significance of the measure to stakeholders? • Does it target measures of value?Stakeholder-valued process
measures (hypothetical)
Adults with > 1 visit
BMI documented
Stakeholder-valued outcome measures
BMI > 30
Follow-up documented
?
UDS process measure
Limitations of Current Measure• Relevance• What is the significance of the measure to stakeholders? • Does it target measures of value?Stakeholder-valued process
measures (hypothetical)
Adults with > 1 visit
BMI documented
?
Stakeholder-valued outcome measures
BMI > 30
Follow-up documented
Do stakeholders give equal value to follow-up documentation in • a 65 year old with
a BMI of 21 as in • a 21 year old with
a BMI of 65?
UDS process measure
Limitations of Current Measure• Relevance• What is the significance of the measure to stakeholders? • Does it target measures of value?Stakeholder-valued process
measures (hypothetical)
Adults with > 1 visit
BMI documented
?
Stakeholder-valued outcome measures
BMI > 30
Follow-up documented
Measure could have enhanced value if made concordant with a BMI cut-off of value to stakeholders
Limitations of Current Measure• Relevance• Does the measure encourage activities that use resources
efficiently to maximize health?• Is the data useful for improving improved performance?• Aggregate nature of measure makes it hard to use for process
improvement• cannot compare providers or practices• thus cannot measure improvement associated with practice-level
interventions
• cannot target individual patients for case management or other services
• relevance• scientific soundness• feasibility
Addressing Relevance-Related Limitations• Addressing lack of concordance with measures of value• Add relevant variables to measure
Org 1 Org 2 Org 3 Org 40
10
20
30
40
50
60
70
80
UDS Adult Weight Measure Compliance, 2012
Addressing Relevance-Related Limitations• Addressing composite nature of current measure
Parameter Selection
BMI documented Yes No
Follow-up documented Yes No
Age and BMI parameters < age 65 AND BMI > 25 > age 65 AND BMI > 30 < age 65 AND BMI < 18.5 > age 65 AND BMI < 22 Org 1 Org 2 Org 3 Org 4
0%
10%
20%
30%
40%
50%
60%
UDS Adult Weight Measure Compliance, 2012
Addressing Relevance-Related Limitations• Addressing lack of concordance with measures of value
Org 1 Org 2 Org 3 Org 40%
10%
20%
30%
40%
50%
60%
UDS Adult Weight Measure Com-pliance, 2012
Parameter Selection
BMI documented Yes No
Follow-up documented Yes No
Age and BMI parameters < age 65 AND BMI > 25 > age 65 AND BMI > 30 < age 65 AND BMI < 18.5 > age 65 AND BMI < 22 Age _____ to _____ BMI _____ to _____ Age _____ to _____
BMI _____ to _____
Number of visits in 12 months _____ to _____ Hgb A1C _____ to _____ Systolic BP _____ to _____ Diastolic BP _____ to _____ Hospitalizations in 12 months _____ to _____
Nutrition Exercise
Addressing Utility-Related Limitations• Addressing aggregate nature of current measure
Population
Practice Main St Clinic West Health Practice Whole Family Center Casey Cares Clinic Peaceful Practice Wellness Center
Provider Brett Almond Pat Grant Darby Eden Carson Brooke Harper Keegan Addison Keith Kelly Jordan
Main
St C
linic
Wes
t Hea
lth P
ract
ice
Who
le Fa
mily
Cen
ter
Casey
Car
es C
linic
Peac
eful
Pra
ctice
Wel
lnes
s Cen
ter
010203040506070
Brett Almond
Pat Grant Darby Eden
Carson Brooke
Harper Keegan
Addison Keith
Kelly Jordan
0
10
20
30
40
50
60
70
Addressing Utility-Related Limitations• Addressing aggregate nature of current measure: patient registry
Limitations of Current Measure• Scientific soundness• Does the measure precisely evaluate what is actually
happening? What measures of data quality are reported?• What is the strength of the evidence linking the clinical
processes and outcomes that the measure addresses?• Does the measure produce reproducible results when repeated
in the same population and setting?• Does the measure make sense logically and clinically?
(validity)• Is it appropriate to risk-adjust the measure by age or some
other variable?• Are accuracy, reproducibility and validity consistent across
different data systems and settings? • relevance• scientific soundness• feasibility
Limitations of Current Measure• Scientific soundness: data quality• If chart-review is used to derive UDS
measure• inter-relater variation: • measure elements—especially deciding what
constitutes follow-up—are not simple to identify
• observation bias, misclassification
• If EHR is used to derive UDS measure• clinicians vary• what justifies clicking a “follow-up done’
checkbox
• how often they forget to click the checkbox (completeness of data)
• EHRs vary in how easily data is recorded and extracted
Limitations of Current Measure• Scientific soundness• Lack of information about data quality in the measure reporting is a barrier to• interpreting and using the measure
• fixing data quality issues
• relevance• scientific soundness• feasibility
Addressing Scientific Soundness-Related Limitations• Addressing lack of information about data quality
Main
St C
linic
Wes
t Hea
lth P
ract
ice
Who
le Fa
mily
Cen
ter
Casey
Car
es C
linic
Peac
eful
Pra
ctice
Wel
lnes
s Cen
ter
010203040506070
Data Quality Elements
Out-of-range values Height > 260 cm Height < 60 cm Weight > 200 kg Weight < 20 kg Pregnant males
Missing values Missing height
Missing weight Missing BMI Missing follow-
up Missing
provider
Population
Practice Main St Clinic West Health Practice Whole Family Center Casey Cares Clinic Peaceful Practice Wellness Center
Provider Brett Almond Pat Grant Darby Eden Carson Brooke Harper Keegan Addison Keith Kelly Jordan
Limitations of Current Measure• Feasibility• Does the measure impose an inappropriate burden on health
care systems?• Does the measure have clear specifications for data sources
and methods for data collection and reporting?• Does the data collection violate accepted standards of
member confidentiality?• Is the required data logistically feasible to access?• Is the measure susceptible to manipulation that would be
undetectable in an audit?
• relevance• scientific soundness• feasibility
Limitations of Current Measure• Feasibility• Chart review of 70 records is resource
intensive• What would be the resource commitment
involved in transitioning to EHR-based reporting of this measure?
• relevance• scientific soundness• feasibility
R/QI Committee Process• Define priorities for addressing limitations • Refine and finalize scope-of-work for addressing key limitations of the current UDS measure• Develop reports that address these limitations• Define and pursue next steps• assess changes in relevance, scientific soundness and
feasibility• disseminate findings• to whom?
• what format?
• possible groundwork for pragmatic trial
• relevance• scientific soundness• feasibility
