Challenges using Safety Monitoring Systems A review of Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of

Challenges using Safety Monitoring Systems

A review of Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of the Elephant

David PeavyCarter SewardJake Swinton

ReferenceOsnat Levtzion-Korach, M.D. M.H.A.; Allan Frankel, M.D.; Hanna Alcalai, B.P.T.,

M.H.A; Carol Keohane B.S.N., R.N.; John Orav, Ph.D.; Erin Graydon-Baker, M.S., R.R.T.; Janet Barnes, R.N, J.D., C.P.H.R.M.; Kathleen Gordon, M.S.N., M.S.; Anne Louise Puopulo, B.S.N, R.N.; Elena Ivanova Tomov, M.B.A.; Luke Sato, M.D.; David W. Bates, M.D., M.Sc. (September 2010). Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of the Elephant. The Joint Commission Journal on Quality and Patient Safety, Vol. 36, No. 9. Retrieved from: http://psnet.ahrq.gov/public/JCJQPS-2010-PSNet-ID-19031.pdf. Accessed February 6, 2015.

2ETM 568 - Peavy, Seward & SwintonSafety Monitoring Systems

http://psnet.ahrq.gov/public/JCJQPS-2010-PSNet-ID-19031.pdf



Introduction● The article investigates the various reporting systems and how they

correlate to one anothero Use different methods of obtaining informationo Place results using different categorieso Different personnel fill our reportso Prioritize different events

● The article reviewed the quality reporting systems/processes used at Brigham and Women’s Hospital (BWH) located in Boston, MA.o 747-bed tertiary care academic medical centero 52,000 inpatient admissions/950,000 outpatient admissions per yearo 12,000 employees; approximately 3000 are doctors


Methods● BWH uses a total of 5 Quality Reporting Systems:

o Incident Reportingo Reports to Risk Managemento Patient Complaintso Executive Walk Aroundso Malpractice Claims

● The authors wanted to:o Evaluate what type of information is received by each systemo Develop a common framework for representing the identified safety issueso Assess the correlation between types of information collectedo Evaluate the overall safety picture


Data● The data was collected from the source systems over a

22 month period between May 10th, 2004 and February 28th, 2006

● The 22 month time period had a small number of malpractice claims, so to increase this number, the time period was extended to 10 years.


Assessment Areas● Incident Reporting System

o Commercially available web- based incident reporting systemo Usually filled out by nurseso Evaluated by Department Managers

● Hospital Risk Managemento Physicians and Nurses call the team to report adverse events and

poor patient outcomeso Even split of reporting between Nurses and Physicianso Evaluated by Risk Management staff


Assessment Areas (Cont.)● Patient Complaints

o The Family and Patient Relations Department Responds to patient and family complaints, suggestions, and compliments

● Executive Walk Aroundo Executive leadership walk arounds occur semiweekly and engage with staff

about safety concerns

● Malpractice Claimso The malpractice insurer has a system used for data collection called CMAPS. o This system takes information from potential claim reports risk managers, and

formal malpractice claims


Results


(Page 404; Levtzion-Korach, M.D. et. al.)

Comparisons● The five assessment areas

did not have overlapping categorization schemes

● The experimenters developed a categorization scheme with 23 major categories to compare the 5 reporting systems.

● Examples:o Staffo Staff Training/Educationo Technical Skillso Equipment/Supplieso Medication Errorso Identification Issueso Monitoring of Patiento Clinical Judgemento Medical Recordso Test Reconciliation Issues


Analysis● Data collected by the five reporting systems was analyzed and

classified into one of the new categories by a physician reviewer trained in patient safety.

● Investigators calculated the frequencies of each of the events

● Overlap between rankings of frequency of different categories were then assessed and correlations between the detection approaches were calculated


Categorical Analysis● The leading major categories across the

reporting systems were:o Communication (11.6%)o Technical Skills (10.9%o Clinical Judgement (9%)

● However, each system had a different category that was most frequent


Statistical Correlation

● Overall, across the five systems, Cronbach’s standardized alpha was 0.22, suggesting a low level of consistency



Correlations● Correlations presented:

o Highest correlation observed was between risk management reports and malpractice claims

o A high correlation was found between patient complaints and malpractice claims

o The incident reporting system has little correlation with the other systems

o Executive walk rounds had a negative correlation with all other systems






Discussion● Some things to keep in mind about the study:

o Only a single medical center was used. Results shown may not be generalizable to other facilities

o Not all possible types of safety data were included. However, BWH allowed the authors to closely examine numerous independent data source


Discussion (cont’d)● The study allowed for the comparison and evaluation of

each reporting system, both individually, and as a whole

● Each system provides a different set of partial conclusions, and there is little correlation between individual systems


Conclusion● Studies or reports commenting on only one of the reporting

systems are incomplete

● Much better, more complete conclusions may be drawn from a complete perspective that utilizes information from multiple reporting systems

● Hospitals should utilize a more broad approach to the classification of adverse events in order to make more informed decisions


Documents

Challenges using Safety Monitoring Systems A review of Integrating Incident Data from Five Reporting Systems to Assess Patient Safety: Making Sense of