What is Your Data Telling You?
Clara Ballantine, SIA OntarioFalls VLC Team CallJanuary 11, 2011
ObjectivesIn the next 30 Minutes we will: Differentiate among data for accountability,
for research and for improvement Review how to effectively display Falls data
using a run chart Analyze your run chart data “by the rules” Consider strategies to share your data
effectively with others
Aspect Improvement Comparison or Accountability
Clinical Research
Aim:
Improvement of care Comparison, choice, reassurance, spur for
change
New knowledge
Methods: Test
observability Test observable No test, evaluate
current performance Test blinded
Bias
Accept consistent bias Measure and adjust to reduce bias
Design to eliminate bias
Sample size
“Just enough” data, small sequential samples
Obtain 100% of available, relevant, data
“Just in case” data
Flexibility of hypothesis
Hypothesis flexible, changes as learning
takes place
No hypothesis Fixed hypothesis
Testing strategy
Sequential tests No tests One large test
Determining if change is
improvement
Run charts or Shewhart charts
No change focus Hypothesis tests (T-tests, F-tests, Chi-square), p-value
Confidentiality
of data
Data used only by those involved in the improvement
Data available for public consumption
Research subjects’ identities protected
Table 2.1: Data for Improvement, Accountability, Research
Data for Improvement, Accountability and Research
Source: The Data Guide: Learning from Data to Improve Healthcare. Developed from Solberg, Leif I., Mosser, Gordon and McDonald, Susan. “The Three Faces of Performance Measurement: Improvement, Accountability and Research.” Journal on Quality Improvement. March 1997, Vol.23, No. 3.
What Do We Need to Know? How much variation do we have? Is this process changing significantly over time? Have our changes resulted in improvement? Are we holding the
improvement?
Run Chart
• A line graph of data plotted over time• Data is kept in time order• Can see flow of data• Helps answer our improvement questions
Before and After Test(Change Made Between Week 7 & Week 8)
8
3
0123456789
10
Before Chance (measure point on week 4) After Change (measure point on Week 11)
Del
ay T
imes
(H
rs)
Case 1
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Weeks
De
lay
Tim
e (
Hrs
)
Made Change
Why Use a Run Chart?
I @:@ -1 : - f, I I .
'And this is the period when the cat was away. '
8
Run Chart – Falls 4.0
Baseline
1MD, 1RN trained
Staff Training complete
Team Leader
resigned
New staffstarted
Thanks to: Virginia FlintoftSHN Central Measurement Team
Source: National Nursing Home Improvement Collaborative: Pressure Ulcer Prevention and Treatment Handbook, Qualis Health
Key Point #1
• Annotate your run chart – it tells your story• Annotation helps you see the impact of your
test of change and can help you decide to “adopt, adapt or abandon” a change.
How to Annotate a Run Chart
11
1. Select “Insert” tab
2. Select “Text Box”
How to Annotate a Run Chart
12
3. Enter text in Text Box
4. Right click on Text Box and select “Format Shape “ from menu.
How to Annotate a Run Chart
13
5. Format text box – use your imagination!• fill colour• line colour and style • etc.
% Timely Reperfusion
Months
pe
rce
nt
1/07 2 3 4 5 6 7 8 9 10 11 12 1/08 2 3 4 5 65 7 8 9 10 11 1232 23 32 38 35 35 40 21 38 26 22 27 23 32 36 29 38 42 39 36 50 48 39 44
DateData
Run Chart
1/072 3 4 5 6 7 8 9 10 11 12 1/08
2 3 4 5 65 7 8 9 10 11 12
15
20
25
30
35
40
45
50
55
60
Change 1
Chg 2,3
Chg 4,5,6
Chg 7
Chg 8,9Median 35
MEDIAN
In a series of numbers, the median is physically the middle number .
It has the same number of points equal to it or above it as it has equal to it or below it.
Finding the Median: Reordering the Data504844424039393838383635353532323229272623232221
• To find the median reorder the numbers from high to low and find the number physically in the middle. If you have two numbers left in the middle, add them together and divide by two.
• Excel: place cursor in blank cell and type=MEDIAN(A2:A21) where A2 is the first cell you want to include and A21 the last)
Key Point # 2
• Calculate and show a Median line to see trends and a Goal line
• Analyze your data “by the rules”
Heart Failure Readmission Rates*Source: Residents Satisfaction on Discharge Hand-Off, Peg Bradke, St. Luke’s Hospital, Cedar
Rapids, Iowa, Hospital to Home: Optimizing the Transition January 2009
*Percent of heart failure Residents readmitted for exacerbation of their heart failure.
0
5
10
15
20
25
30
May
-05
Jul-0
5
Sep-
05
Nov
-05
Jan-
06
Mar
-06
May
-06
Jul-0
6
Sep-
06
Nov
-06
Jan-
07
Mar
-07
May
-07
Jul-0
7
Sep-
07
Nov
-07
Jan-
08
Mar
-08
May
-08
Jul-0
8
Sep-
08
Nov
-08
Per
cen
tag
e
Rate (%) Median
GoodAug 06 = Implemented use of new Residents education materials
Jan 07 = Initiated complimentary visits
Rule 1 Six or more consecutive POINTS either all above or all below the
median. Skip values on the median and continue counting points. Values on the median DO NOT make or break a shift.
Median=10Median=11
Rule 1
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Mea
sure
or C
hara
cter
istic
Median 10
Ott, Ellis, Process Quality Control, McGraw-Hill Book Company, NY, 1975
Source: National Nursing Home Improvement Collaborative: Pressure Ulcer Prevention and Treatment Handbook, Qualis Health
Rule 2Five points all going up or all going down. If the value of two or more successive points is the same count the first one then ignore the identical points when counting; like values do not make or break a trend.
Rule 2
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Mea
sure
or C
hara
cter
istic
Median 11
Olmstead, Pl, “Distribution of Sample Arrangements for Runs Up and Down, Annals of Mathematical Statistics, Vol 17, pp. 24- 33, March, 1945
Rule 3• To Determine The Number of Runs Above and Below the Median:
– A run is a series of points in a row on one side of the median. Some points fall right on the median, which makes it hard to decide which run these points belong to.
– So, an easy way to determine the number of runs is to count the number of times the data line crosses the median and add one.
– Statistically significant change signaled by too few or too many runs.
Rule 3
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
Measure
or C
hara
ceristic
Median 11.4
Data line crosses onceToo few runs: total 2 runs
Rule 3: # of RunsTable for Checking for Too Many or Too Few Runs on a Run Chart
Total number of data
points on the run chart
that do not fall on the
median
Lower limit for the number of runs(< than this number of runs is “too few”)
Upper limit for the number of runs(> than this number of runs is “too many”)
10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
21 7 16
22 7 17
23 7 17
24 8 18
25 8 18
Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart, (1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives. Annals of Mathematical Statistics. Vol. XIV, pp.66 and 87, Tables II and III
RULE 4For detecting unusually large or small numbers:
• Data that is Blatantly Obvious as a different value• Everyone studying the chart agrees that it is unusual• Remember:
– Every data set will have a high and a low - this does not mean the high or low are astronomical
Rule 4
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Measure
ment or C
hara
cte
ristic
What do “They” Want to Know?Senior leaders: How are we doing? How do we compare with others?Falls project team: What is the impact of our work? Where do we focus our efforts?Staff How is this helping our patients/residents/clients? What can we do?
Engage Senior Leaders With Data and Evidence
The "whiskers" depict the 95th% CI of the National Mean. Your mean is statisically higher if
it is above the "whiskers"; it is the same as the National Mean if it is within the "whiskers"; and it is statistically lower if it is below the "whiskers".
INTERVENTION - FALLS: 3.0 Percentage of Residents with Completed Fall Risk Assessment on Admission
0%20%40%60%80%
100%
Month
Percent
Local Team National Goal
Bring Data to Improvement Teams Frequently: Process
1MD, 1RN trained
Baseline
Staff Training complete
Team Leaderresigned
New staffstarted
Median
Bring Data to Improvement Teams Frequently: Outcome
1.0 Falls Rate per 1000 Resident Days
0
2
4
6
8
10
12
MonthFalls
/ 100
0 D
esid
ent D
ays
Actual Goal
Post Process Measure Data for All Staff to See Progress
5.0 Percentage of "At Risk" Residents with Falls Prevention/Protection Intervention Implemented
0%10%20%30%40%50%60%70%80%90%
100%
Month
Perc
enta
ge w
ith Im
plem
ente
d Fa
lls P
reve
ntio
n/Pr
otec
tion
Actual Goal
Median
Post Process Measure Data for All to See Opportunities for Improvement
3.0 Percentage of Residents with Completed Fall Risk Assessment
0%10%20%30%40%50%60%70%80%90%
100%
Month
Perce
ntage
with
Fall R
isk
Asse
ssme
nt
Actual Goal
Key Point #3
• Share your data effectively with others at all levels of your organization
• Consider your audience when choosing which data and how to display
For Help With Your Data and Measurement Questions
Virginia Flintoft Clara Ballantine
416.946.8350 613.736.9142
[email protected] [email protected]
Dannie Currie Alexandru Titeu
902.842.0716 416.946.3103
[email protected] [email protected]
Chantal Bellerose
514.340.8222 ext 4901