Measuring Mistakes and Mistakes in Measuringkumcce.ku.edu/sites/kumcce.ku.edu/files/docs...100% compliance documented, but only 48% adherence to guideline elements (e.g., appropriate

1

Measuring Mistakes

& Mistakes in Measuring

Robert S. Lagasse, MD Professor & Vice Chair Quality Management & Regulatory Affairs Department of Anesthesiology Yale School of Medicine New Haven, CT

64th Annual Postgraduate Symposium on Anesthesiology April 11, 2014, 13:45-12:30, InterContinental Kansas City at the Plaza, Kansas City, MO

Disclosure

l  Dr. Lagasse has had no relevant financial relationship with any commercial entity related to the content of this lecture.

l  Dr. Lagasse has no potential conflict of interest related to the content of this lecture –  ASA Representative to the Joint

Commissions Professional Technical Advisory Committee until December 2013

–  Member of Steering Committee for the CDC/CMS Surgical Care Improvement Project (SCIP)

Objectives

After attending this lecture, participants should be able to:

1.  Evaluate the methodologies used to measure perioperative morbidity and mortality

2.  Discuss the factors contributing to adverse perioperative events

3.  Predict the impact of performance measurement on anesthesiology practice

Is Performance Measurement Required?

l  Accrediting Bodies With Deeming Authority –  The Joint Commission (formerly JCAH 1965) –  American Osteopathic Association HFAP (1966) –  DNV Healthcare (2008) –  Center for Improvement in Healthcare Quality (2013) –  Accreditation Association for Ambulatory Health Care (1979) –  American Association for Accreditation of Ambulatory

Surgical Facilities (1980) l  Regulatory Agencies

–  Centers for Medicare and Medicaid Services (CMS) –  State Department of Health

l  American Board of Anesthesiology –  Maintenance of Certification in Anesthesiology (MOCA)

l  Surgical Care Improvement Project (SCIP) l  American College of Surgeons National

Surgical Quality Improvement Project (NSQIP) l  Physicians’ Quality Reporting System (PQRS) l  Specialty Societies

–  Society of Thoracic Surgeons –  Anesthesia Quality Institute

l  Others –  QIOs, insurers, advocacy groups, & ACO’s

Is Performance Measurement Required?

The Joint Commission Performance Improvement

PI.01.01.01 The organization collects data to monitor its performance

PI.02.01.01 The organization compiles and analyzes data

PI.03.01.01 The organization improves performance

LD.04.04.05 The organization has organization-wide, integrated

patient safety program

The Joint Commission, CAMH, 2010

2

Standard PI.01.01.01 Elements of Performance

1.  The leaders set priorities for data collection 2.  Hospital identifies the frequency for data

collection The hospital collects data on the following: 3.  Performance improvement priorities identified

by leaders 4.  Operative or other procedures that place

patients at risk of disability or death 5.  All significant discrepancies between

preoperative and postoperative diagnoses


Standard PI.01.01.01 Elements of Performance

The hospital collects data on the following: 6.  Adverse events related to using moderate or

deep sedation or anesthesia 7.  Use of blood and components 8.  All confirmed transfusion reactions 11.  The results of resuscitation 12.  Behavior management and treatment 14.  Significant medication errors 15.  Significant adverse drug reactions 16.  Patient perception of safety and quality


Professional Practice Evaluation

Standard MS.08.01.03 (OPPE) Ongoing professional practice evaluation information is factored into the decision to maintain existing privilege(s), to revise existing privilege(s), or to revoke an existing privilege prior to, or at the time of, renewal

Standard MS.08.01.01 (FPPE) The organized medical staff defines the circumstances requiring monitoring and evaluation of a practitioner’s professional performance


Sentinel Event Alert Database

l  Unexpected occurrence involving death or serious physical or psychological injury, or the risk thereof. Serious injury specifically includes loss of limb or function

l  The phrase, "or the risk thereof" includes any process variation for which a recurrence would carry a significant chance of a serious adverse outcome

l  Such events are called "sentinel" because they signal the need for immediate investigation and response

http://www.jointcommission.org

TJC Sentinel Event Database


TJC Sentinel Event Database


3

ORYX

l  Hospitals required to collect and transmit data to TJC for a minimum of four core measure sets or a combination of applicable core measure sets (accountability measures)

l  Data for all applicable measures must be submitted through a performance measurement system(s) that has been approved by The Joint Commission


ORYX Measure Sets

l  Acute Myocardial Infarction (AMI) l  Heart Failure (HF) l  Pneumonia (PN) l  Pregnancy and Related Conditions (PR) l  Hospital-based Inpatient Psychiatric Services

(HBIPS) l  Children's Asthma Care (CAC) l  Surgical Care Improvement Project (SCIP) l  Hospital Outpatient Measures (HOP)


l  National quality partnership of 10 organizations –  ASA, ACS, VA NSQIP, CMS, CDC, etc

l  Goal to reduce complications by 25% by 2010 l  Process Measures in 4 Target Areas

–  Surgical Wound Infection –  Cardiac Morbidity –  Thrombophlebitis –  Ventilator Associated Pneumonia

Surgical Care Improvement Project

http://www.medqic.org/scip/scip_homepage.html

l  Prevention of Surgical Infection –  Selection of antibiotics –  Timing of antibiotics –  Maintenance of normothermia –  Glucose control

l  Prevention of Perioperative MI –  Selective administration of B blockers

Process Measures for Anesthesiologists?

http://www.medqic.org/scip/scip_homepage.html

Implementation Gaps and Data Validity

l Hypothesized that documented compliance with antibiotic prophylaxis guidelines does not reflect actual adherence to guidelines

l Observational study of 100 elective pediatric surgical cases

l  100% compliance documented, but only 48% adherence to guideline elements (e.g., appropriate administration, type, timing*, weight-based dosing*, and redosing)

Hawkins at al. Am J Surg 2013; 206(4):451-456

Reporting Hospital Data for

Annual Payment Update

l  Congress provided a financial incentive for all hospitals to voluntarily report quality of care information for release to consumers

l  10 quality measures for 3 medical conditions: MI, CHF, & pneumonia

l  0.4 percentage point reduction in annual payment update from Medicare for 2006 for non-participating hospital resulted in 99% participation

Medicare Prescription Drug, Improvement & Modernization Act 2003

4

Amendments to Deficit Reduction Act

l  Beginning in 2007, hospitals reported performance measures or lost 2% of their market basket increase

l  Established procedures for making the reported performance measures available to the public

l  Measures maintained by the Secretary of Health & Human Services

Deficit Reduction Act, Sec. 5001. Hospital Quality Improvement

Hospital-Acquired Conditions (Never Events)

l  After October 1, 2008, hospitals do not receive payment for certain high cost, high volume DRGs that are ‘preventable’ through evidence-based guidelines, if not present on admission –  Catheter-associated urinary tract infections –  Pressure ulcers (decubitus ulcers) –  Foreign object retained after surgery –  Air embolism –  Blood incompatibility –  Staphylococcus aureus septicemia

Federal Register, May 3, 2007 (Part II), Dept of Health & Human Services

National Surgical Quality Improvement Project

l  NSQIP has developed and validated separate risk adjustment models for 30-day morbidity and for 30-day mortality after major surgery in eight surgical subspecialties and for all operations combined

l  Preoperative patient characteristics include demographics, symptoms, physical findings, comorbidities unrelated to the reason for surgery, preoperative laboratory values, and ASA PS

Best. J Am Coll Surg 194:257-66, 2002

NSQIP – Cost

l  Cost of data collection and analysis is approximately $38 per case in VA system

l  Expanding by approximately 100,000 cases annually and currently has over 1 million cases

l  ACS expanding to private sector at cost of $35K/site/yr + costs of 1 FTE data collector

l  VA investment followed a 1986 congressional mandate to compare outcomes to national benchmarks following accusations of poor care

Best. J Am Coll Surg 194:257-66, 2002

Khuri. Arch Surg 137:20-7, 2002

2.0

2.2

2.4

2.6

2.8

3.0

3.2

Phase 1 Phase 2 FY 96 FY 97 FY 98 FY 99 FY 00(10/1/91-12/31/93)

(10/1/95-9/30/96)

(1/1/94-8/31/95)

(10/1/96-9/30/97)

(10/1/97-9/30/98)

(10/1/98-9/30/99)

(10/1/99-9/30/00)

30-D

ay M

orta

lity

(%) MORTALITY FOR ALL SURGERY

(30 day)

1991 - 2002 27% reduction

NSQIP - ROI Khuri. Arch Surg 137:20-7, 2002

8

10

12

14

16

18

Phase 1 Phase 2 FY 96 FY 97 FY 98 FY 99 FY 00(10/1/91-12/31/93)

(10/1/95-9/30/96)

(1/1/94-8/31/95)

(10/1/96-9/30/97)

(10/1/97-9/30/98)

(10/1/98-9/30/99)

(10/1/99-9/30/00)

30-D

ay M

orbi

dity

(%)

1991 - 2002 45% reduction

MORBIDITY FOR ALL SURGERY

NSQIP - ROI

5

Physicians Quality Reporting Initiative

l  TRHCA signed on December 20, 2006 l  Established a financial incentive for eligible

professionals to participate in a voluntary quality reporting program

l  Eligible professionals who reported designated quality measures from July 1 to December 31, 2007 earned a bonus payment, subject to a cap, of 1.5% of total allowed charges for covered Medicare physician fees

Tax Relief and Health Care Act of 2006 (TRHCA)

Affordable Care Act l  2010 2% bonus l  2011 1% bonus l  2012-14 0.5% bonus l  2015 1.5% reduction l  2016 - 2.0% reduction

Patient Protection and Affordable Care Act, 2010

Physicians Quality Reporting System

PQRS Measures

l  74 PQRI quality measures in 2007 l  Medicare, Medicaid, and SCHIP Extension Act

of 2007 authorized continuation of PQRI l  318 PQRS quality measures in 2012

–  Measures, specifications & toolkit available online l  4 applicable to anesthesiologists

–  Antibiotic timing, CVC insertion protocol, and normothermia

–  MOCA has been approved as a measure

http://www.cms.hhs.gov/pqri/

PQRS Reporting Success

l  In 2010, anesthesiology had one of the highest rates of participation in PQRS, and one of the highest error rates in reporting—nearly 50 percent

l For example, an incorrect procedure code (CPT or HCPCS) was reported on approximately 46% of all 2,262,233 PQRS Quality Data Codes reported

http://www.anesthesiallc.com/publications/ealerts (May 31,2011)

Maintenance of Certification in Anesthesiology (MOCA)

l  Revised MOCA Part IV: Practice Performance Assessment and Improvement (PPAI) Program

l  A new three-part PPAI program will be required for all newly certified diplomates and non-time limited diplomates who enter the MOCA program after January 1, 2008. The three-part program includes: 1) simulation education, 2) patient safety education, and 3) case evaluation

http://www.theaba.org/anesthesiology-maintenance.shtml

MOCA Case Evaluation 4-step process

1.  Collect performance data, over extended time 2.  Compare outcomes with practice guidelines,

explicit expert consensus, or to peer data 3.  Design and implement a plan to improve

outcomes using clinical reminders, education, system/process changes, or clinical pathways

4.  Collect new data, compares the latest outcomes to the original assessment to determine the amount of improvement (goal - improve or maintain a high practice standard)


6

MOCA Case Evaluation 4-step process

l  Case evaluations may be done by a group or by an individual

l  If the group approach is used, you must be able to extract the individual candidate’s data

l  Required for non-time limited certificate holders not enrolled in MOCA who wish to participate in MOC:PQRS


Data Flow for Quality Management

Incident reported by anesthetist

Ambulatory surgery follow-up

Concurrent chart review

Report by other hospital personnel

Resident Evaluation Committee

OPPE &

MOCA

Data entered into departmental database

Departmental structured peer review

Final changes made to database & reports prepared

Clinical Research

Fact finding & preliminary review

Clinical sections structured peer review

SPC analysis & CQI

Regulation & Accreditation

Sensitivity/Specificity of Referral Sources

l  Physician self-reporting is a more sensitive means of detecting cases meeting indicator criteria and is not biased by severity of the outcome or involvement of human error

l  Concurrent chart reviewers are biased by outcomes involving death or permanent injury

l  Personnel using hospital incident reports are biased by involvement of human error

l  All sources felt potential for performance improvement was most important factor

Katz and Lagasse. Anesth Analg 90:344–50, 2000






Departmental structured peer review Clinical sections structured peer review


Resident evaluation committee

OPPE &

MOCA Clinical

Research



SPC analysis & CQI


Interrater Reliability of Peer Review

l  Stable pool of reviewers l  Multiple reviewers l  Shared expertise l  Known outcomes l  Group discussion l  Structured review process

–  System factors and human factor

Levine et al. Anesthesiology 89: 507-515, 1998

Distribution of Contributing Factors

Human Factor (7.8%)

System Factors (92.2%)

Improper technique Failure to seek appropriate data Disregard of available data Inadequate knowledge

Technical accidents Limited therapeutic standards Limited diagnostic standards Limited supervision

Lagasse et al. Anesthesiology 82: 1181-8, 1995

7

Error Distributions in Other Industries

“I should estimate that in my experience most troubles and most possibilities for improvement add up to proportions something like this: 94% belong to the system (system errors are the responsibility of management) 6% special (human errors are the worker’s responsibility)”

W. Edwards Deming






Departmental structured peer review


Resident evaluation committee


SPC analysis

& CQI

OPPE &

MOCA Clinical

Research



Clinical sections structured peer review

AQI

Sample Output Run Charts

Anesthesiology Dept., Montefiore Medical Center

Human Error Rate 1995-2006

05

1015202530354045

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

Year

Human Error

Per 10,000 Cases

Sample Output Run Charts


Perioperative Mortality Rate

02468

1012141618

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

Year

per 10,000

Sample Output Statistical Process Control

0

10

20

30

40

50

60

Asse

ssme

nt Pr

oblem

s / 10

,000 C

ases

AVG= 17

UCL

Mar 95 Jun 95 Sep 95 Dec 95 Mar 96 Jun 96 Sep 96 Dec 96 Mar 97 Jun 97 Sep 97 Dec 97 Mar 98 Jun 98 Sep 98 Dec 98 Mar 99 Jun 99 Sep 99 Dec 99 Mar 00 Jun 00

AEU Restricted

Guideline distributed


Anesthesiology - Model of Safety

IOM Quality of Healthcare in America Project 1999

l  44-98,00 Americans die each year as a result of medical errors

l  “…today, anesthesia mortality rates are about one death per 200,000–300,000 anesthetics administered, compared with two deaths per 10,000 anesthetics in the early 1980s.” 49

l  “The gains in anesthesia are very impressive and were accomplished through a variety of mechanisms (improved monitoring, practice guidelines, and other systematic approaches).”

8

Quality Management Data Perioperative Mortality

l  Definition of death during or within two post-procedure days (JCAHO 1988)

l  Two hospital networks (1992 - 1999) 347 deaths in 184,472 anesthetics (1:532) l  Human error 14:347 deaths (4.0%) l  Anesthesia-related deaths 1:13,176 (0.76 per 10,000 anesthetics)

Lagasse. Anesthesiology 97:1609-17, 2002

l  23 anesthesia-related mortality studies in general populations between 1955 & 1992

l  Perioperative mortality 1:53 to 1:5,417 (188 to 1.8 per 10,000)

l  Anesthesia-related mortality 1:1,265 to 1:85,708 (7.9 to 0.16 per 10,000)


Anesthesia Safety: Model or Myth?

Anesthesia-related Mortality (1:1,265 to 1:85,708 anesthetics)

2.07

0.12

7.91

5.87

2.2

1.9 1.98

3.38

0.16

5.11

0.62 0.38 0.54

3.4

0

3.4

0.25 0 1 2 3 4 5 6 7 8 9

1960

19

62

1964

19

66

1968

19

70

1972

19

74

1976

19

78

1980

19

82

1984

19

86

1988

19

90

1992

Year (midpoint)

Death

s (p

er 10

,000 a

nesth

etics

) 8.6

Avg 3.34


Deaths Attributed Solely to Anesthesia

l Eichhorn JH (Massachusetts 1976 - 1988) l  1,001,000 anesthetics in ASA Physical

Status I or II patients resulted in11 cases of major intraoperative accidents reported to malpractice insurance carrier, including 5 intraoperative deaths

l Anesthesia solely responsible for death in 1:200,200 anesthetics

Eichhorn. Anesthesiology 70:572–7, 1989

Deaths Attributed Solely to Anesthesia

l Eichhorn JH (Massachusetts 1976 - 1988) l  1,001,000 anesthetics in ASA Physical

Status I or II patients resulted in11 cases of major intraoperative accidents reported to malpractice insurance carrier, including 5 intraoperative deaths

l Anesthesia solely responsible for death in 1:200,200 anesthetics

Eichhorn. Anesthesiology 70:572–7, 1989

l Of the 14 anesthesia related deaths, only 4 were determined by peer review to result from ‘major contributions’ from the anesthesia provider (1 per 46,118)

l Only one of these occurred in a patient with ASA Physical Status of I or II and resulted in litigation (1 per 184, 472)

l This death did not occur intraoperatively


Quality Management Data Perioperative Mortality

9

l  Predictors of Survival following Cardiac Arrest in Patients Undergoing Non-cardiac Surgery - A Study of 518,294 Patients at a Tertiary Referral Center

l  Frequency of cardiac arrest was 4.3 per 10,000 anesthetics between 1990 and 2000

l  0.5 per 10,000 arrests due to anesthesia l  Mortality caused by arrests attributable to

anesthesia was 1 in 100,000 anesthetics

Sprung et al. Anesthesiology 99:259-69, 2003

Morbidity (e.g., Cardiac Arrest)

l  Recent studies of anesthesia-related mortality found somewhat higher rates (approximately 1:13,000 vs. 1:100,000)

l  Sprung defined cardiac arrest as an event requiring closed chest compressions or open cardiac massage

l  Unstable patients whose arrest occurred after an anesthetic induction agent was given were not considered


Morbidity (e.g., Cardiac Arrest)

l  253 deaths within 48 hours of 180,000 anesthetics (1:711 anesthetics)

l  110 involved a cardiac arrest during anesthesia care (1:1,636)

l  5 judged by peer review to be due to anesthetist error (1:36,000 anesthetics)

l  3 unstable prior to induction left only 2 meeting inclusion criteria (1:90,000)


Quality Management Data Cardiac Arrest Conclusions

l  Anesthesia may not be as safe as we believe because the methodologies used to measure perioperative morbidity and mortality are flawed

l  Lack of standardized methodology makes comparisons impossible

l  Standardization must include outcome definitions, risk adjustment, and the review process

l  Comparisons for judging clinical competence are even more difficult because most adverse events are not the result of human error

Documents

Measuring Mistakes and Mistakes in Measuringkumcce.ku.edu/sites/kumcce.ku.edu/files/docs...100% compliance documented, but only 48% adherence to guideline elements (e.g., appropriate