Www.ideasontario.ca Data is your Friend Collecting, Charting, Analyzing, and Interpreting Data to Support Quality Improvement Michael Campitelli and Ruth

www.ideasontario.ca

Data is your Friend

Collecting, Charting, Analyzing, and Interpreting Data to Support Quality Improvement

Michael Campitelli and Ruth CroxfordQI Epidemiologists, Institute for Clinical Evaluative Sciences (ICES)

Doug MitchellDirector Decision Support, Guelph General Hospital

Susan TaylorDirector, QI Program Delivery, Health Quality Ontario

www.ideasontario.ca 2

• Relationships with commercial interests:– Grants/Research Support: None

– Speakers Bureau/Honoraria: None

– Consulting Fees: None

– Other: None

Faculty:Ruth Croxford





– Other: None

Faculty:Michael Campitelli





– Other: None

Faculty:Susan Taylor


• Relationships with commercial interests:

– Grants/Research Support: None– Speakers Bureau/Honoraria: None– Consulting Fees: None– Other: None

Faculty:Doug Mitchell


Outline

• Review (45-50 minutes)• Bar charts and Pareto charts• Scatter plots• Run charts and SPC charts• Statistical testing between groups

• Break (10 minutes)

• Case Studies specific to the health care sector you identify with the most (45-50 minutes)• Primary care• Long-Term care• Acute care


GUELPH GENERAL HOSPITAL – HIP AND KNEE REPLACEMENT IMPROVEMENT INITIATIVE


Abbreviated, anonymized version of the data


HC Data Guide, p 65 (fig 2.28)

Tools to learn from variation in data


Bar Charts


Bar Charts

A B C0

10

20

30

40

50

60

70

80

90

100

% wait time 1 within target% wait time 2 within target


Bar Charts (fictional data)


Statistics are like bikinis.

What they reveal is suggestive, but what they conceal is vital.

Aaron Levenstein, Professor of Business, Baruch College


Pareto Charts


Pareto Charts(fictional data)

Reason (fictious data) FrequencyReferral missing X-rays 86Referral missing other health information 75Patient weight loss required 35Other medical 22Patient refused offered consult date 10Patient requested specific surgeon 43Surgeon schedule 25



Scatter Plots


Scatter Plots


Scatter Plots


Histograms


Bikini #1


Histograms

Surgeon % within target

Mean Median

A 33% 176 163B 24% 154 155C 66% 68 71


HC Data Guide, p 65 (fig 2.28)

Tools to learn from variation in data


Run Charts


Run Charts


Run Charts

Four (probability-based) rules to identify non-random signals of change in a run chart (Health Care Data Guide, pgs 76 – 85)• A trend

– Five or more consecutive points all going up or all going down.

• A shift– Six or more consecutive points either all above or all below

the median

• Too many or too few runs (crossings of the median)– Depends on the number of points on the graph - requires a

table

• An astronomical data point


http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/run-chart.aspx


Run Charts


The Run Chart as a Bikini


Corresponding Shewhart Chart


Statistical Process Control (SPC) Charts


Shewhart Chart Selection Guide

HC Data Guide p. 151(fig 5.1)


Types of Measures – Continuous Variables

Patient ID

Wait 2 (days) - decision to procedure

1 3122 3493 4724 3155 2926 2867 2558 2659 297

10 27211 1112 28613 12214 16215 24716 281

Month Case 1 Case 2 Case 3 Case 4 Case 5 Case 6May-11 312 349Jun-11 472 315 292Jul-11 286 255 265 297 272 11

Aug-11 286 122 162 247 281 288Sep-11 211 289 391 226 272Oct-11 121 299 243 160 240 278

Nov-11 122 129 164 110 110 138Dec-11 112 101 104 139 251 87Jan-12 70 73 79 154 178 173Feb-12 153 190 344 359 21 286Mar-12 85 272 202 206 210 182Apr-12 184 287 208 260 31 122

MonthAverage Wait 2

May-11 330.5Jun-11 359.7Jul-11 148.2

Aug-11 153.4Sep-11 277.8Oct-11 147.9

Nov-11 200.8Dec-11 155.8Jan-12 200.3Feb-12 186.3Mar-12 152.8Apr-12 152.7

May-12 105.7Jun-12 115.3Jul-12 105.3


Types of Measures – Count Data

• Requires two columns of data for each time period: the count and the number of “opportunities”

• The event being counted can occur more than once per “opportunity”

WeekNumber of flash

sterilizationsNumber of surgeries

1 42 842 47 1463 51 914 45 1065 36 886 34 1267 37 818 49 869 39 8310 46 7711 28 7812 46 10813 34 7214 44 13115 41 83

Rate of flash sterilizations (flash sterilizations per

100 surgeries)


Types of Measures – Classification Data

• Requires two columns of data for each time period: the total number of people or events that were observed, and the number of “non-conforming” events.

Month

Number within target wait time

Total number

Apr-11 0 1May-11 0 2Jun-11 0 3Jul-11 0 6Aug-11 1 12Sep-11 0 5Oct-11 1 16Nov-11 9 29Dec-11 6 21Jan-12 7 25Feb-12 15 39Mar-12 9 31Apr-12 7 22May-12 15 28Jun-12 9 31Jul-12 5 24

Percent (percent of patients seen within the target

time)


Learning from a Shewhart Chart

• Rules for detecting special cause variation.• Annotation• Setting and re-setting the baseline


SPC Chart Rules


Baseline Data


First PDSA


New Baseline


PDSA 2


Final Graph


Rare Events (fictitious data)

date of complications

time from previous event

1-Jul-11 2930-Jul-11 24

23-Aug-11 1911-Sep-11 23

4-Oct-11 281-Nov-11 43

14-Dec-11 2912-Jan-12 3314-Feb-12 3823-Mar-12 424-May-12 4013-Jun-12 546-Aug-12 40

15-Sep-12 504-Nov-12 43

17-Dec-12 5611-Feb-13 50

2-Apr-13 4921-May-13 8312-Aug-13 888-Nov-13 6916-Jan-14 62

19-Mar-14 763-Jun-14 5124-Jul-14 82

14-Oct-14 70


Rare Events (fictitious data)


Data for Judgement vs. Data for Improvement

• Measurements towards a target may hide or discourage authentic and sustainable improvement

• Targets for accountability may focus on what is easily measured rather than what has value (process rather than outcome)



(fictitious data)



(fictitious data)


Statistical Testing


Statistical testing

• Statistical testing is a common form of analysis used in clinical research and epidemiological studies

• Tests the hypothesis that the average/proportion/rate of some outcome in one group of patients is equal to the average/proportion/rate in another group of patients

• Statistical tests produce a P-value, which represents the likelihood that the observed difference in the outcome between the two groups is due to chance

• Studies often set the significance level at 0.05, meaning if there is less than 5% chance the observed results are due to chance, we deem the results `statistically significant`


Statistical testing versus QI analysis

• While used heavily in clinical research and epidemiology, statistical testing is not the analytic method of choice (e.g., the `Gold Standard`) for quality improvement

• QI involves conducting sequential tests of change over time to some existing process; therefore, it is logical that tracking outcome and process measures over time in an SPC chart would be the preferred method of analysis

• Performing statistical tests, rather than tracking measures over time, may cause us to claim improvement when none has occurred, or miss improvement when some has occurred.


A medical unit has 40 COPD discharges per month.

On July 1, they implement a self-management training program prior to discharge for all patients.

There is a statistically significant decrease (p=0.026) in COPD readmission rates after the implementation.

Pre (Jan - Jun) Post (Jul - Dec)0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

20.0%

12.5%Readmissions

P = 0.026


Here are the readmission rates plotted by month.

There is an apparent decrease happening throughout the year, perhaps due to other quality improvement initiative.

Difficult to tie decrease to the July 1 initiative

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec0

0.05

0.1

0.15

0.2

0.25

Readmissions1


Is statistical testing forbidden in QI…

• All QI projects should strive to track measures over time and use annotated run and SPC charts for analysis of their data

• Having said that, sometimes it is not feasible to collect data in any other fashion (e.g., satisfaction surveys which are burdensome and time-consuming to complete), and you are stuck with having to do a pre-post comparison

• The following website has multiple online calculators to help you perform basic statistical tests between 2 groups for averages (means), proportions, and rates:• http://www.socscistatistics.com/tests/


Thank You

• [email protected]• [email protected]

mailto:[email protected]

mailto:[email protected]


Delivered in partnership and collaboration with:

Funding provided by the Government of Ontario

Documents

Www.ideasontario.ca Data is your Friend Collecting, Charting, Analyzing, and Interpreting Data to Support Quality Improvement Michael Campitelli and Ruth