Embed Size (px)
Citation preview
7/25/2012
1
How will you know that a change is an
improvement?Prepared and Presented
Robert Lloyd, Ph.D.Executive Director Performance Improvement
Institute for Healthcare ImprovementTuesday, August 14, 2012 ~ Chicago
Objectives
• To be clear about why you are measuring.
• To review the milestones along the Quality
Measurement Journey (QMJ).
• To assess where you and your organization
are in the QMJ.
2
7/25/2012
2
Exercise: Measurement Self-Assessment
This self-assessment is designed to help quality facilitators gain a better understanding of where
they personally stand with respect to the milestones in the Quality Measurement journey (QMJ).
What would your reaction be if you had to explain why using a run or control chart is preferable to
computing only the mean, the standard deviation or computing a p-value? Can you construct a run
chart or help a team decide which control is most appropriate for their data?
You may not be asked to do all of the things listed below today or even next week. But, if you are
facilitating a QI team or advising a manager on how to evaluate a process improvement effort,
sooner or later these questions will be posed. How will you deal with them?
The place to start is to be honest with yourself and see how much you know about the QMJ. Once
you have had this period of self-reflection, you will be ready to develop a learning plan for self-
improvement and advancement.
Use the following Response Scale. Select the one response which best captures your opinion.
1 I could teach this topic to others!
2 I could do this by myself right now but would not want to teach it!
3 I could do this but I would have to study first!
4 I could do this with a little help from my friends!
5 I'm not sure I could do this!
6 I'd have to call in an outside expert!Source: R. Lloyd, Quality Health Care: A Guide to
Developing and Using Indicators. Jones &
Bartlett Publishers, 2004: 301-304.
Exercise: Measurement Self-AssessmentSource: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators.
Jones & Bartlett Publishers, 2004: 301-304.
Measurement Topic or SkillResponse Scale
1 2 3 4 5 6
Moving a team from concepts to set of specific quantifiable measures
Building clear and unambiguous operational definitions
Developing data collection plans (including frequency and duration of data
collection)
Helping a team figure out stratification strategies
Explain and design probability and nonprobability sampling options
Explain why plotting data over time is preferable to using aggregated data and
summary statistics
Describe the differences between common and special causes of variation
Construct and interpret run charts (including the run chart rules)
Decide which control chart is most appropriate for a particular measure
Construct and interpret control charts(including the control chart rules)
Link measurement efforts to PDSA cycles
Build measurement plans into implementation and spread activities
7/25/2012
3
A Model for Learning and Change
When you
combine the 3
questions with
the…
PDSA cycle,
you get… …the Model for
Improvement.
5The Improvement Guide, API, 2009
Our focus today
Why are you measuring?
The answer to this question will guide your entire quality measurement journey!
Improvement?
7/25/2012
4
7
QualityBetter
Old Way
(Quality Assurance)
QualityBetter Worse
New Way
(Quality Improvement)
Action taken
on all
occurrences
Reject
defectives
Requirement,
Specification or
Threshold
No
action
taken
here
Worse
Healthcare Measurement:
Old Way, New WaySource: Robert Lloyd, Ph.D.
“The Three Faces of Performance Measurement:
Improvement, Accountability and Research”
“We are increasingly realizing not only how
critical measurement is to the quality
improvement we seek but also how
counterproductive it can be to mix
measurement for accountability or research
with measurement for improvement.”
byLief Solberg, Gordon Mosser and Sharon McDonald
Journal on Quality Improvement vol. 23, no. 3, (March 1997), 135-147.
7/25/2012
5
The Three Faces of Performance Measurement
Aspect Improvement Accountability Research
Aim Improvement of care
(efficiency & effectiveness)
Comparison, choice,
reassurance, motivation for
change
New knowledge
(efficacy)
Methods:
• Test ObservabilityTest observable
No test, evaluate current
performance Test blinded or controlled
• Bias Accept consistent bias Measure and adjust to
reduce bias
Design to eliminate bias
• Sample Size “Just enough” data, small
sequential samples
Obtain 100% of available,
relevant data
“Just in case” data
• Flexibility of
Hypothesis
Flexible hypotheses,
changes as learning takes
placeNo hypothesis
Fixed hypothesis
(null hypothesis)
• Testing Strategy Sequential tests No tests One large test
• Determining if achange is animprovement
Run charts or Shewhart
control charts
(statistical process control)
No change focus
(maybe compute a percent
change or rank order the
results)
Hypothesis, statistical tests
(t-test, F-test,
chi square), p-values
• Confidentiality ofthe data
Data used only by those
involved with improvement
Data available for public
consumption and review
Research subjects’ identities
protected
“Health Care Economics and Quality”by Robert Brook, et. al. Journal of the American Medical Association
vol. 276, no. 6, (1996): 476-480.
Three approaches to research:
• Research for Efficacy(experimental and quasi-experimental designs/clinical trials, p-values)
• Research for Efficiency
• Research for Effectiveness
Quality Improvement
Research
7/25/2012
6
0
0.1
0.2
0.3
0.4
0.5
0.6
Ja
n-0
1
Fe
b-0
1
Ma
r-0
1
Ap
r-0
1
Ma
y-0
1
Ju
n-0
1
Ju
l-0
1
Au
g-0
1
Se
p-0
1
Oc
t-0
1
No
v-0
1
De
c-0
1
Ja
n-0
2
Fe
b-0
2
Ma
r-0
2
Ap
r-0
2
Ma
y-0
2
Ju
n-0
2
Ju
l-0
2
Au
g-0
2
Se
p-0
2
Oc
t-0
2
No
v-0
2
De
c-0
2
Pro
po
rtio
n
Control Chart - p-chart
11552 - Vaginal Birth After Cesarean Section (VBAC) Rate
Data for
Improvement
0
0.1
0.2
0.3
0.4
0.5
0.6
Jan-0
1
Feb-0
1
Mar-
01
Apr-
01
May-0
1
Jun-0
1
Jul-01
Aug-0
1
Sep-0
1
Oct-
01
Nov-0
1
Dec-0
1
Jan-0
2
Feb-0
2
Mar-
02
Apr-
02
May-0
2
Jun-0
2
Jul-02
Aug-0
2
Sep-0
2
Oct-
02
Nov-0
2
Dec-0
2
Pro
port
ion
Data for Judgment
These data points are all
common cause variation
Better than or Equal to State Average
Worse than State Average
Legend
State
Average Q3 04 4Q 04 Q1 05 Q2 05 YTD 05
AMI 79 77 79 78 80 78
CHF 61 56 58 56 62 58
PN 46 16 16 20 31 20
SIP 52 41 43 43 49 44
63
54
81 79
60
47
Example of Data for JudgmentCMS/HQA Core Measures
Does this tabular display of data help us understand the variation in these measures?
(Perfect Care Bundles – all aspects of a bundle must be met in order to receive credit)
7/25/2012
7
So, how do you view the Three Faces of
Performance Measurement?
Or,
As… As a…Im
pro
vem
en
t
Ju
dg
men
t
Researc
h
Integrating the Three Faces of
Performance Measurement
The three faces of performance
measurement should not be seen as
mutually exclusive silos. This is not an
either/or situation.
All three areas must be understood as a
system. Individuals need to build skills in
all three areas.
Organizations need translators who and
be able to speak the language of each
approach.
The problem is that individuals identify with
one of the approaches and dismiss the
value of the other two.
7/25/2012
8
The Quality Measurement Journey
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
ACTION
The Quality Measurement Journey
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
7/25/2012
9
Category
Number
of
Measures
ADEs 14
CAUTI 7
CLABSI 11
FALLS 6
OB 16
PU 8
READMISSION 18
SSI 8
VAP 6
VTE 9
• 103 measures have been identified
• They all have operational definitions
• Potential data sources are referenced
AHA/HRET Hospital Engagement
Network Encyclopedia of Measures
The basic set of measures for the
HEN have already been identified
and the operational definitions
provided.
18
An Operational Definition...
… is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently.
• It gives communicable
meaning to a concept
• Is clear and unambiguous
• Specifies measurement
methods and equipment
• Identifies criteria
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
7/25/2012
10
AHA/HRET Hospital Engagement Network Encyclopedia of Measures
The Quality Measurement Journey
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
7/25/2012
11
Now that you have selected and defined
your measures, it is time to head out, cast
your net and actually gather some data!
22
Stratification
• Separation & classification of data according to
predetermined categories
• Designed to discover patterns in the data
• For example, are there differences by shift, time of
day, day of week, severity of patients, age, gender or
type of procedure?
• Consider stratification BEFORE you collect the data
7/25/2012
12
1C-23
0
500
1000
1500
2000
2500
3000
3500
3/1/
2008
3/8/
2008
3/15
/200
8
3/22
/200
8
3/29
/200
8
4/5/
2008
4/12
/200
8
4/19
/200
8
4/26
/200
8
5/3/
2008
5/10
/200
8
5/17
/200
8
5/24
/200
8
5/31
/200
8
6/7/
2008
6/14
/200
8
6/21
/200
8
6/28
/200
8
7/5/
2008
7/12
/200
8
7/19
/200
8
7/26
/200
8
8/2/
2008
8/9/
2008
8/16
/200
8
8/23
/200
8
8/30
/200
8
9/6/
2008
9/13
/200
8
9/20
/200
8
What does a stratification problem look like?
Measure: running calorie total
There are two distinct
processes at work here!
© Richard Scoville & I.H.I.
Tota
l C
alo
rie
s
The data should be divided
into two stratification levels
0
500
1000
1500
2000
2500
3000
3500
3/1/
2008
3/8/
2008
3/15
/200
8
3/22
/200
8
3/29
/200
8
4/5/
2008
4/12
/200
8
4/19
/200
8
4/26
/200
8
5/3/
2008
5/10
/200
8
5/17
/200
8
5/24
/200
8
5/31
/200
8
6/7/
2008
6/14
/200
8
6/21
/200
8
6/28
/200
8
7/5/
2008
7/12
/200
8
7/19
/200
8
7/26
/200
8
8/2/
2008
8/9/
2008
8/16
/200
8
8/23
/200
8
8/30
/200
8
9/6/
2008
9/13
/200
8
9/20
/200
8
Travel days
Home days
What factors might influence your process?
Track them in your data to provide insights about
variation and how to change the process!
© Richard Scoville & I.H.I.
Tota
l C
alo
rie
s
7/25/2012
13
Common Stratification Levels
• Day of week
• Shift
• Severity of patients
• Gender
• Type of procedure
• Payer class
• Type of visit
• Unit
• Age
What
stratification
levels are
appropriate for
your data?
Sampling
When you can’t
capture data on the
entire population
(an enumeration),
you can estimate its
characteristics by
sampling.
© Richard Scoville & I.H.I.
7/25/2012
14
The Relationships Between a Sample
and the Population
Population
Negative Outcome Positive Outcome
What would a “good”
sample look like?
Source: R. Lloyd. Quality Health Care: A Guide to
Developing and Using Indicators. Jones and Bartlett
Publishers, 2004, 75-94.
Population
Negative Outcome Positive Outcome
Ideally a “good” sample will have the same shape and center as the total
population but have fewer observations.
A representative
sample
The Relationships Between a Sample
and the Population
Source: R. Lloyd. Quality Health Care: A Guide to
Developing and Using Indicators. Jones and Bartlett
Publishers, 2004, 75-94.
7/25/2012
15
Population
A sample improperly pulled could result in a negative sampling bias
(red curve)…
A negatively
biased sample
Negative Outcome Positive Outcome
Sampling Bias
Source: R. Lloyd. Quality Health Care: A
Guide to Developing and Using
Indicators. Jones and Bartlett Publishers,
2004, 75-94.
Population
…or a positively biased sample (green curve).
A negatively
biased sampleA positively
biased sample
Negative Outcome Positive Outcome
Sampling Bias
Source: R. Lloyd. Quality Health Care: A
Guide to Developing and Using
Indicators. Jones and Bartlett Publishers,
2004, 75-94.
7/25/2012
16
Population
But, a properly pulled sample will be representative of the total population.
A negatively
biased sampleA positively
biased sample
Negative Outcome Positive Outcome
Sampling Bias
How do you draw your samples?
Source: R. Lloyd. Quality Health Care: A
Guide to Developing and Using
Indicators. Jones and Bartlett Publishers,
2004, 75-94.
Non-probability Sampling Methods
• Convenience sampling
• Quota sampling
• Judgment sampling
Sampling Methods
Probability Sampling Methods
• Simple random sampling
• Stratified random sampling
• Stratified proportional random sampling
• Systematic sampling
• Cluster sampling
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones
and Bartlett Publishers, 2004, 75-94.
7/25/2012
17
Sampling Options
Simple Random Sampling
Population Sample
Proportional Stratified Random Sampling
Population Sample
Medical Surgical OB Peds
Judgment Sampling
Jan March April May JuneFeb
S S M P
M M M
OB OB S
34
Judgment Sampling
• Include a wide range of conditions
• Selection criteria may change as
understanding increases
• Successive small samples instead
of one large sample
Especially useful for PDSA testing. Someone with
process knowledge selects items to be sampled.
Characteristics of a Judgment Sample:
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones
and Bartlett Publishers, 2004, 75-94.
7/25/2012
18
Judgment Sampling in Action!
It always seems
pretty calm to me
here in the
afternoon.
It is absolutely
nuts here
between 8 and
10 AM!
Well the night shift
is totally different
from the day shift!
How often and for how long do
you need to collect data?
• Frequency – the period of time in which you collect data (i.e., how often will
you dip into the process to see the variation that exists?)• Moment by moment (continuous monitoring)?
• Every hour?
• Every day? Once a week? Once a month?
• Duration – how long you need to continue collecting data• Do you collect data on an on-going basis and not end until the measure is always
at the specified target or goal?
• Do you conduct periodic audits?
• Do you just collect data at a single point in time to “check the pulse of the process”
• Do you need to pull a sample or do you take every occurrence of the data (i.e.,
collect data for the total population)
7/25/2012
19
The Frequency of Data Collection:
The Story of Flight #1549, January 15, 2009
Flight #1549, January 15, 2009
7/25/2012
20
ERROR: stackunderflow
OFFENDING COMMAND: ~
STACK:
