Embed Size (px)
Citation preview
4/17/2014
1
Improvement Science In Action
How Do We Know Change Is
Improvement?
Sandra Murray April 30, 2014
This presenter has
nothing to disclose
SESSION OBJECTIVES:
By the end of this session participants will be able to:
Differentiate between data used for improvement, accountability and research
Explain the value of viewing data over time
Explain why improvement requires a small family of measures (FOM)
Differentiate between outcome, process and balancing measures
Explain the importance of operational definitions
Begin to develop a useful set of measures to track your improvement project
4/17/2014
2
What are we trying to accomplish?
How will we know that a change is an improvement?
What change can we make that will result in improvement?
Model for Improvement
Act Plan
Study Do
Used with permission: Improvement Guide Improvement Guide, Jossey-Bass, 2009
Going
Deeper
Sources of Data
Health Care Data Guide , Provost and Murray, 2011. p.26
4/17/2014
3
Types of Data: Qualitative and Quantitative
Qualitative (Non-numeric) and Quantitative: (Numeric)
Qualitative: Typically from observations Some qualitative examples: – “the customer had difficulty using the form”
– “the appointment was not on schedule”
Measurements can be qualitative (for example, blood type)
Quantitative data are usually preferred for learning but some reasons to use qualitative data: – Quantitative data can be difficult or expensive to obtain
– The information is so dramatic that qualitative data are sufficient to meet all needs
– Observations of people may best describe the phenomena of interest
DG p. 32
How Are Data Used?
Our emphasis is on using data for improvement
– Improving health care processes difficult to do without feedback
to determine where to focus or to determine if changes tested
result in improvement
Different uses for data will drive different methods and
tools
Getting data for improvement made difficult due to
confusion between data for improvement, accountability
and research
Need to be clear about purpose for our data and be able
to explain to others
DG p. 26
4/17/2014
4
Enumerative and Analytic Studies Enumerative Analytic
Action Taken directly on the population
from which samples taken
Action taken will be on an underlying
causal system with the intent of
improving performance of a product,
process, or system in the future.
Static (fixed population) and distribution
of population known
Dynamic (ongoing process or stream of
data) with unknown distribution
Random sampling More typically judgment sampling
Data often collected, analyzed, and
displayed as an aggregate measure(s)
before the study intervention and again
after the intervention is made
Data collected and analyzed at regular
ongoing intervals (such as daily,
weekly, or monthly at slowest) rather
than solely before and after a change
Data analyzed using classic statistical
methods such as a T-test or confidence
intervals, P values.
Data analyzed using run and Shewhart
charts and other graphical displays of
data
DG p.41
Data for Improvement, Accountability and Research in Health Care (DG p. 27)
Aspect Improvement Accountability or
Judgment
Research
Aim: Improvement of car
processes, systems and
outcomes
Choice, reassurance, spur
for change
New generalizable knowledge
Methods:
Test observable No test, evaluate current
performance
Test blinded
Bias: Accept consistent bias Measure and adjust to
reduce bias
Design to eliminate bias
Sample Size: “Just enough” data, small
sequential samples
Obtain 100% of available,
relevant data
“Just in case” data
Flexibility of
Hypothesis:
Hypothesis flexible,
changes as learning
takes place
No hypothesis Fixed hypothesis
Testing Strategy: Sequential tests No tests One large test
Determining if a
Change is an Improvement:
Run charts or Shewhart
control charts
No focus on change
Shewhart control charts
Hypothesis, statistical tests (t-
test, F-test, chi square, p-
values)
Confidentiality of the Data:
Data used only by those
involved with
improvement
Data available for public
consumption
Research subjects’ identities
protected
Frequency of Use: Daily, weekly, monthly Quarterly, annually At end of project
Source: The Data Guide: Learning from Data to Improve Healthcare. Developed from Solberg, Leif I., Mosser, Gordon and McDonald, Susan. “The
Three Faces of Performance Measurement: Improvement, Accountability and Research.” Journal on Quality Improvement. March 1997, Vol.23, No.
3.
4/17/2014
5
Measurement for Accountability
Data collected for accountability designed to evaluate, compare, or judge
Tip-Offs: – Data typically not viewed over time
– Data infrequent (e.g. quarterly, semi-annual, annual)
– The use of percentiles
– Case-mix adjustment
– A measure that yields nearly all 0s or all 100%.
Accountability data often in “met, did not meet” format and not sensitive enough to help team tell if changes tested are improvements
Example: – Although it may be useful in terms of accountability to know that only 62% of
patients needing a colonoscopy waited less than 90 days, that data may not be very helpful to the improvement team that is testing changes to reduce waiting time.
– Tracking actual waiting time as the team tests changes will be more helpful than solely tracking the percentage meeting the waiting time standard.
DG p. 29
DG P. 29
4/17/2014
6
Some Guidelines for Collecting Data for
Improvement ● Be sure to have a few key measures that clarify the aim of
the improvement effort and make it tangible. These
should be regularly reported throughout the life of the
project (daily, weekly, or monthly, depending on the
length of time for the project).
DG p.32
DG p. 61
4/17/2014
7
Some Guidelines for Collecting Data for
Improvement
● Be sure to have a few key measures that clarify the aim of the improvement
effort and make it tangible. These should be regularly reported throughout
the life of the project (daily, weekly, or monthly, depending on the length
of time for the project).
● Be careful about overdoing process measures. A balance
of outcome, process, and balancing measures is
important.
DG p. 32
Outcome, Process, and Balancing
Measures
Outcome Measures: – Voice of the customer or patient. – How is the system performing? – What is the result? – Direct link to project aim Process Measures: – Voice of the workings of the system. – Are the parts/steps in the system performing as planned? – Are the key changes we are testing having an impact? Balancing Measures: – Looking at a system from different directions/dimensions. – What happened to the system as we improved the outcome and
process measures? – Could be unintended consequences – Could be competing explanation for success
DG p. 63
4/17/2014
8
Measures for Emergency Department Teams
AIM: Improve waiting time and patient satisfaction in the Emergency Department
Outcome: – O1. Total Length of Stay (LOS=wait time)
– O2. Patient Satisfaction Scores
Process:
– P1. % patient receiving discharge materials
– P2. Average Time to registration
– P3. Percent Updated in Wait Area
Balancing:
- B1. Patient volume
- B2. Staff satisfaction
- B3. “Left without being seen” (LWBS) .
Exercise: Evaluate Potential Set of Measures Increase the % of Well Child Visits (WCV) and immunization for enrolled
children aged 3-6 by improving current workflows to improve access, creating a reminder system, creating a recall system, and better engaging parents on the importance of these visits.
Measure O-P-B?
1.Days to Next Available Appointment for Well Child Visit
2. % No Shows for Well Child Visits
3 % of enrolled children aged 3-6 with at least one Well Child Visit
4. % of children who were recalled who completed their Well Child
Exam
5. # Patients in Pilot Population each Month
6. % of children aged 4-6 with DTap immunization
7. Team satisfaction with workflow and systems changes
8. % Cancelled Well Child Visits
9. Patient satisfaction with workflow and systems changes
4/17/2014
9
What is an Operational Definition
To learn from data need an agreement as to how the
data will be collected in order to maintain data collection
consistency
An operational definition is an agreement that gives
communicable meaning to a concept (such as error,
waiting time, and appropriate care) by specifying how the
concept is applied within a particular set of
circumstances.
DG. p 37
Components of Operational Definition
Developing an operational definition requires agreement on
two things:
1. A method of measurement
– Which device? (clock, wristwatch, stopwatch?)
– To what degree of precision (nearest hour, 5 minutes, minute,
second?)
2. A set of criteria for judgment
– What is “late”, “error”, “a fall”?
Page 37
4/17/2014
10
The Importance of Operational
Definitions
If data collected differently by different people, or
differently each time collected, it makes it hard to know
whether changes in the data are due to the changes
tested or from inconsistencies in data collection.
DG. P. 37
Excerpt from Form for Developing
Improvement Project Measures
DG p.40
4/17/2014
11
4/17/2014
12
Table Exercise: Developing a
Set of Measures Background: A friend has come to you and asked you to help develop measures so she can track progress of the group she is working with
Group Aim: The aim of the improvement project is for participants to lose weight. She wants to tell how group is doing—not just each individual
Develop a Family of 4 to 6 measures that could be reported each week for the project: – Outcome Measures – 1-2 measures
– Process Measures – 2 measures
– Balancing Measures – 1 or 2 measures
Use
24
Lose
Weight
Aim Exercise More
Take up sport
I actually like
Build more
activity into
everyday life
Decrease
portion
sizes of
other foods
Increase
fruits and
veggies
Primary
Drivers
Secondary
Drivers
Specific Changes
to test
Consume fewer
calories
Try volleyball class
Go swimming
once next week
Park further away
Use steps at work
2x a day
Climb steps at
home 5x a day on
purpose
Take a fruit and
veggie to work
each day
Switch to smaller
bowls, plates,
glasses
Before eating set
aside ½ food if
eating out
Outcome
Measures
Process
Measures
4/17/2014
13
Project Aim: Lose Weight
Type
Name of
Measure
Definition, how to collect data Weekly summary
statistic
Outcome
Measure
Process
Measure
Balancing
Measure
Outcome, Process, and Balancing
Measures
Outcome Measures: – Voice of the customer or patient. – How is the system performing? – What is the result? – Direct link to project aim Process Measures: – Voice of the workings of the system. – Are the parts/steps in the system performing as planned? – Are the key changes we are testing having an impact? Balancing Measures: – Looking at a system from different directions/dimensions. – What happened to the system as we improved the outcome and
process measures? – Could be unintended consequences – Could be competing explanation for success
DG p. 63
4/17/2014
14
Some Guidelines for Collecting Data for
Improvement
● Be sure to have a few key measures that clarify the aim of the improvement
effort and make it tangible. These should be regularly reported throughout
the life of the project (daily, weekly, or monthly, depending on the length of
time for the project).
● Be careful about overdoing process measures. A balance of outcome,
process, and balancing measures is important.
● Plot data visually on the key measures over time.
DG p 32
Cycle Time
Results for Units
1, 2 and 3
Unit 1
Unit 3
Unit 2
The run chart: a simple analytical tool for learning from variation in healthcare processes.
Rocco J Perla, Lloyd P Provost and Sandra K Murray. BMJ Qual Saf 2011 20: 46-51.
4/17/2014
15
Some Guidelines for Collecting Data for
Improvement
● Be sure to have a few key measures that clarify the aim of the improvement
effort and make it tangible. These should be regularly reported throughout
the life of the project (daily, weekly, or monthly, depending on the length of
time for the project).
● Be careful about overdoing process measures. A balance of outcome,
process, and balancing measures is important.
● Plot data visually on the key measures over time.
● Limit time devoted to data
• Make use of existing databases and data already collected for
developing measures.
• Whenever feasible, integrate data collection for measurement
into the daily work routine.
DG p.32
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample=1
per week
Sample=20
per week
Sample=50
per week
Sample=10
per week
Sample=5
per week
DG p.47
4/17/2014
16
Sample Size
For data used as part of an improvement effort, getting
data across a wide range of conditions (locations, days
of the week, shift, and so forth) may be more important
than the amount of data collected under a specific
condition.
In general, more data (larger sample sizes) lead to more
information and better precision of results.
Unless the data are already collected and reported,
larger sample sizes also involve more effort and cost.
Some Guidelines for Collecting Data for
Improvement
● Be sure to have a few key measures that clarify the aim of the improvement effort and
make it tangible. These should be regularly reported throughout the life of the project
(daily, weekly, or monthly, depending on the length of time for the project).
● Be careful about overdoing process measures. A balance of outcome, process, and
balancing measures is important.
● Plot data visually on the key measures over time.
● Make use of existing databases and data already collected for developing measures.
● Whenever feasible, integrate data collection for measurement into the daily work
routine.
● Be aware that the second question of the Model for
Improvement—“How will we know that a change is an
improvement?”—usually requires more than one
measure. A balanced set of three to eight measures will
ensure that this question can be answered.
Page 32
4/17/2014
17
DG p. 64
Project Team Exercise
• 20 Minutes
1. Using your project draft charter and driver diagram start
to identify potential:
– Outcome (1 or 2)
– Process (at least 2 based on your strategies –what you will
work on
– At least 1balancing measures
2. Be prepared to share
4/17/2014
18
Measure
Name
O-
P-
B
?
Driver
Addres
sed
Denominator Numerator Data Collection
Procedure
(sampling,
stratification)
Statisti
c
to Plot
Frequ
ency
Goal
1
2
3
4
5
6
7
8
Project Charter What are we trying to accomplish? Purpose (Aim): By August 2013, reduce inappropriate 30-day readmissions by 25%, for
advanced illness patients who were seen by the Palliative Care service at XXX University Hospital and subsequently discharged to a sub-acute facility (SNF) through effectively collaborating and communicating with two partner SNFs.
Theory: By improving the collaboration and communication between the PC service line at
XXX and two partner SNFs, the patient and family preferences and expectations will be known across the care continuum, which will help to align end of life resources at a sub-acute facility to the wishes of the patient and family, and as a result, there will be reduced inappropriate readmissions for patients from a sub-acute facility.
Why is it Important to work on now? •Aligns with the health system’s strategic plan through 2017. •Aligns with current (and expected) reforms to the US healthcare model. •Provides immediate benefits to both the patient and the care provider.
Goals: • 25% reduction in the percentage of patients with inappropriate (unwanted or non-
beneficial) 30-day readmissions (i.e. patients discharged to a sub-acute facility that and were readmitted to the hospital within 30 days).
• 95% of greater of patients with advanced illness will have completed MOLST form on discharge Source: Brian Galli. Used with permission.
Example
4/17/2014
19
Project Charter (Continued) What are we trying to accomplish? Expected Benefits:
• Direct Benefits (with measures):
– Reduction in inappropriate (unwanted or non-beneficial) 30-day readmissions from a sub-
acute facility.
• Indirect Benefits (without measures):
– Patients and family preferences and expectations are known, documented, and followed.
– Better alignment of end of life care (resources) at the sub-acute facility with the
preferences of the patient and family.
– For the patient: Quality of life; Patient/Family satisfaction; & Financial savings.
– For the care provider: Reduction in costs & Utilization of resources.
In Scope:
• Patients who were seen by the Palliative Care service at NSUH and subsequently discharged
to a sub-acute rehab service.
• Start of the process: Patient is seen by the Palliative Care service at XXX
• End of process: Patient discharged to sub-acute care.
• Patient discharged to either of two sub-acute facilities: AAA or BBB.
Potential Barriers/Obstacles:
• Culture of the healthcare organization.
• Relationship between hospital care team and patient and/or family.
• Relationships between hospital care team and external care providers.
• Mixture of EMR and paper-based documentation in the hospital setting.
Source: Brian Galli. Used with permission.
Example
Project Charter (continued) How do we know that a change is an improvement?
Outcome Measures (Monthly):
1. Percentage: Patients discharged to a sub-acute facility and were readmitted to the
hospital within 30 days.
Process Measures (Weekly):
2. Percentage: Patients discharged to a sub-acute facility with a MOLST form
documented in their records.
3. Count: Patients discharged to sub-acute facility without a MOLST form.
4. Percentage: Of family able to articulate understanding of patient’s prognosis
Balancing Measure (Weekly):
5. Average Length of Stay (LOS) for advanced illness patients
6. Monthly Occupied Bed Days (OBD)
Team: •Champion/Chief: Dr. LL
•Admin Director: AC
•Nurse; GF
•Social Worker/Case Manager BD
•SNF Reps – AI and LF •Patient KG
Source: Brian Galli. Used with permission.
Example
4/17/2014
20
Project Driver Diagram
XXX
Source: Brian Galli. Used with permission.
1
4
2,3 Balancing Measure (Weekly):
5. Average Length of Stay (LOS)
for advanced illness patients
6. Monthly Occupied Bed Days
(OBD)
Example
Project Charter (continued)
What changes can we make that will lead to improvement?
Change Ideas:
• Improve the collaboration between PC team and case management/social
worker departments at XXX.
• Improve the collaboration between the PC team at XXX and the clinicians/PC
teams at the partnering SNFs.
• Use team huddle to plan for MOLST and Goals of Care Meeting.
• Use MOLST form to discuss the Goals of Care with the patient and family as
well as communicate to the SNF facility that the patient is discharged to.
Source: Brian Galli. Used with permission.
Example
