National Quality Standards and Statistical EvidenceNational Quality Standards Roles of Jeffrey P. Jacobs, MD: Chair, STS National Database Workforce Chair, CHSS Committee on Quality

Jeffrey P. Jacobs, M.D., FACS, FACC, FCCP

Professor of Surgery and Pediatrics, Johns Hopkins University

Co-Director, Johns Hopkins All Children’s Heart Institute

Chief, Division of Cardiovascular Surgery

Director, Andrews/Daicoff Cardiovascular ProgramSurgical Director of Heart Transplantation

Johns Hopkins All Children’s Heart Institute

Johns Hopkins All Children’s Hospital and Florida Hospital for Children

National Quality Standards and

Statistical Evidence

PCTAP

November 2, 2017

National Quality StandardsRoles of Jeffrey P. Jacobs, MD:

Chair, STS National Database Workforce

Chair, CHSS Committee on Quality Improvement and Outcomes

Working Group Leader, Heart/Heart Surgery Working Group for U.S. News America's Best Children's Hospitals rankings

Editor-in-Chief, Cardiology in the Young

Co-Chair, World Congress of Pediatric Cardiology and Cardiac Surgery 2021

Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric and Congenital Cardiac Care - Volume 1: Outcomes Analysis. Springer-Verlag London. Pages 1 – 515. ISBN: 978-

1-4471-6586-6 (Print). 978-1-4471-6587-3 (Online). Published in 2014.

Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric and Congenital Cardiac Care - Volume 2: Quality Improvement and Patient Safety. Springer-Verlag London.

2015, Pages 1 – 456. ISBN: 978-1-4471-6565-1 (Print). 978-1-4471-6566-8 (Online). Published in 2014.

Definition of Quality

how good or bad something is

a characteristic or feature that someone or something has : something that can be noticed as a part of a person or thing

a high level of value or excellence

[http://www.merriam-webster.com/dictionary/quality].

Accessed November 10, 2015

Donabedian’s Triad

Donabedian A. Evaluating the quality of medical care.

Milbank Mem Fund Q. 1966;44(Suppl):166–206.

Michael Porter

Michael E. Porter, Ph.D. Perspective. What Is Value in

Health Care? N Engl J Med 2010; 363:2477-2481

value defined as the health outcomes achieved per dollar spent

Congenital Heart DiseaseMeaningful

Multi-institutional Outcomes Analysis

Accomplishments

1) Common Language = Nomenclature

2) Mechanism of Data Collection (Database - Registry)

3) Mechanism of Evaluating Case Complexity

4) Mechanism to Verify Data Validity and Accuracy

5) Collaboration Between Subspecialties

6) Longitudinal Follow-Up and Linked Databases

7) Quality Improvement

Barach P, Jacobs JP, Lipshultz SE, Laussen P. (Eds.). Pediatric

and Congenital Cardiac Care - Volume 1: Outcomes Analysis.

Springer-Verlag London. Pages 1 – 515. ISBN: 978-1-4471-6586-

6 (Print). 978-1-4471-6587-3 (Online). Published in 2014.



Accomplishments












The validity of coding of lesions seen in the congenitally malformed heart via the International Classification of Diseases (ICD) is poor

1. Cronk CE, Malloy ME, Pelech AN, et al. Completeness of state administrative databases for

surveillance of congenital heart disease. Birth Defects Res A Clin Mol Teratol 2003;67:597-603.

2. Frohnert BK, Lussky RC, Alms MA, Mendelsohn NJ, Symonik DM, Falken MC. Validity of hospital

discharge data for identifying infants with cardiac defects. J Perinatol 2005;25:737-42.

3. Strickland MJ, Riehle-Colarusso TJ, Jacobs JP, Reller MD, Mahle WT, Botto LD, Tolbert PE, Jacobs

ML, Lacour-Gayet FG, Tchervenkov CI, Mavroudis C, Correa A. The importance of nomenclature

for congenital cardiac disease: implications for research and evaluation. In: 2008 Cardiology

in the Young Supplement: Databases and The Assessment of Complications associated with The

Treatment of Patients with Congenital Cardiac Disease, Prepared by: The Multi-Societal Database

Committee for Pediatric and Congenital Heart Disease, Jeffrey P. Jacobs, MD (editor). Cardiology in

the Young, Volume 18, Issue S2 (Suppl. 2), pp 92–100, December 9, 2008.

4. Pasquali SK, Peterson ED, Jacobs JP, He X, Li JS, Jacobs ML, Gaynor JW, Hirsch JC, Shah SS,

Mayer JE. Differential case ascertainment in clinical registry versus administrative data and

impact on outcomes assessment for pediatric cardiac operations. Ann Thorac Surg. 2013

Jan;95(1):197-203. doi: 10.1016/j.athoracsur.2012.08.074. Epub 2012 Nov 7. PMID: 23141907.

International Paediatric and Congenital Cardiac Code

(IPCCC)

and

Eleventh Iteration of the International Classification of

Diseases

(ICD-11)

www.ipccc.net

http://www.ipccc.net/



Accomplishments












The Report of the 2015 STS Congenital Heart Surgery

Practice Survey

undertaken by the Society of Thoracic Surgeons Workforce on Congenital Heart Surgery

125 centers in the United States of America perform pediatric and congenital heart surgery

8 centers in Canada perform pediatric and congenital heart surgery

Morales DL, Khan MS, Turek JW, Biniwale R, Tchervenkov CI, Rush M, Jacobs JP, Tweddell

JS, Jacobs ML. Report of the 2015 Society of Thoracic Surgeons Congenital Heart

Surgery Practice Survey. Ann Thorac Surg. 2017 Feb;103(2):622-628. doi:

10.1016/j.athoracsur.2016.05.108. Epub 2016 Aug 20. PMID: 27553498.

Jacobs JP, Jacobs ML, Mavroudis C, Tchervenkov CI, Pasquali SK. Executive Summary: The Society of Thoracic

Surgeons Congenital Heart Surgery Database – Twenty-sixth Harvest – (January 1, 2013 – December 31, 2016).

The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical

Center, Durham, North Carolina, United States, Spring 2017 Harvest.

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Participating Centers 18 21 34 47 58 68 79 93 101 105 111 113 117 116

1821

34

47

58

68

79

93

101105

111113

117 116

0

20

40

60

80

100

120

140

Growth in the STS Congenital Heart Surgery DatabaseParticipating Centers Per Harvest





2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Operations 16,461 28,351 37,093 45,635 61,014 72,002 91,639 103,664 114,041 130,823 136,617 143,842 153,558 157,357

16,461

28,351

37,093

45,635

61,014

72,002

91,639

103,664

114,041

130,823136,617

143,842

153,558157,357

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

Growth in the STS Congenital Heart Surgery DatabaseOperations per averaged 4 year data collection cycle





2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Cumulative Operations 9,747 16,537 26,404 39,988 58,181 79,399 98,406 119,266 148,110 179,697 213,416 257,932 292,828 331,672 394,980 435,373

9,74716,537

26,40439,988

58,181

79,399

98,406

119,266

148,110

179,697

213,416

257,932

292,828

331,672

394,980

435,373

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

Growth in the STS Congenital Heart Surgery DatabaseCumulative operations over time

STS Database Penetrance in USA

The STS Congenital Heart Surgery Database (STS-CHSD) is the largest

clinical database in the world for congenital and pediatric cardiac

surgery.

The Report of the 2010 STS Congenital Heart Surgery Practice and

Manpower Survey, undertaken by the STS Workforce on Congenital Heart

Surgery, documented that 125 hospitals in the United States of America

and 8 hospitals in Canada perform pediatric and congenital heart

surgery.

The STS-CHSD contains data from 120 of the 125 hospitals (96%

penetrance by hospital) in the United States of America and 3 of the 8

centers in Canada.

STS Database Penetrance in USA

The STS Congenital Heart Surgery Database (STS-CHSD) is the largest

clinical database in the world for congenital and pediatric cardiac

surgery.

The Report of the 2010 STS Congenital Heart Surgery Practice and

Manpower Survey, undertaken by the STS Workforce on Congenital Heart

Surgery, documented that 125 hospitals in the United States of America

and 8 hospitals in Canada perform pediatric and congenital heart

surgery.

The STS-CHSD contains data from 120 of the 125 hospitals (96%

penetrance by hospital) in the United States of America and 3 of the 8

centers in Canada.

REPRESENTATIVE



Accomplishments












Adjustment for Case Mix

“Differences in medical outcomes may result from

disease severity, treatment effectiveness, or

chance.

Because most outcome studies are observational….

risk adjustment is necessary to account for case mix”

Shahian DM, Blackstone EH, Edwards FH, Grover FL,

Grunkemeier GL, Naftel DC, Nashef SA, Nugent WC, Peterson

ED. STS workforce on evidence-based surgery. Cardiac

surgery risk models: a position article. Ann Thorac Surg.

2004;78(5):1868–77

Risk stratification

Risk stratification is a method of

analysis in which the data are

divided into relatively

homogeneous groups (called

strata).

Risk stratification

The Aristotle Basic Complexity Levels

– (ABC Levels)

– 2002

The Risk Adjustment for Congenital Heart Surgery Categories

– (RACHS-1)

– 2006

The STS-EACTS Mortality Categories

– (STAT Mortality Categories)

– 2010

Two traditional methodologies for

Complexity Adjustment

1) Risk Adjustment in Congenital Heart Surgery-

1 (RACHS-1 )

2) Aristotle Complexity Score

– Aristotle Basic Complexity Score (ABC

Score)

– Aristotle Comprehensive Complexity ScoreJacobs JP, Jacobs ML, Lacour-Gayet FG, Jenkins KJ, Gauvreau K, Bacha

EA, Maruszewski B, Clarke DR, Tchervenkov CI, Gaynor JW, Spray, TL,

Stellin G, O'Brien SM, Elliott MJ, Mavroudis C. Stratification of

Complexity Improves Utility and Accuracy of Outcomes Analysis in a

Multi-institutional Congenital Heart Surgery Database – Application of

the RACHS-1 and Aristotle Systems in the STS Congenital Heart

Surgery Database. Pediatric Cardiology, 2009, DOI 10.1007/s00246-009-

9496-0.

0

5

10

15

20

25

% Mortality

% Mortality 0.6 1.4 4.1 8.7 20.2

1 2 3 4 5 & 6RACHS-1

Category

STS 2006 Congenital Heart Surgery Database

45,635 cases

Jacobs JP, Jacobs ML, Lacour-Gayet FG, Jenkins KJ, Gauvreau K, Bacha EA,

Maruszewski B, Clarke DR, Tchervenkov CI, Gaynor JW, Spray, TL, Stellin G, O'Brien

SM, Elliott MJ, Mavroudis C. Stratification of Complexity Improves Utility and

Accuracy of Outcomes Analysis in a Multi-institutional Congenital Heart Surgery

Database – Application of the RACHS-1 and Aristotle Systems in the STS

Congenital Heart Surgery Database. Pediatric Cardiology, 2009, DOI

10.1007/s00246-009-9496-0.

0

2

4

6

8

10

% Mortality

% Mortality 1.6 2.6 4.1 9.9

1 2 3 4

Aristotle Basic

Level

STS 2006 Congenital Heart Surgery Database

45,635 cases

Jacobs JP, Jacobs ML, Lacour-Gayet FG, Jenkins KJ, Gauvreau K, Bacha EA,

Maruszewski B, Clarke DR, Tchervenkov CI, Gaynor JW, Spray, TL, Stellin G, O'Brien

SM, Elliott MJ, Mavroudis C. Stratification of Complexity Improves Utility and

Accuracy of Outcomes Analysis in a Multi-institutional Congenital Heart Surgery

Database – Application of the RACHS-1 and Aristotle Systems in the STS

Congenital Heart Surgery Database. Pediatric Cardiology, 2009, DOI

10.1007/s00246-009-9496-0.

From Subjective Probability to Objective Data

STAT Mortality Score

The Society of Thoracic Surgeons - European Association for Cardio-Thoracic

Surgery Congenital Heart Surgery Mortality Score

and

STAT Mortality Categories

The Society of Thoracic Surgeons - European Association for Cardio-Thoracic

Surgery Congenital Heart Surgery Mortality Categories

O'Brien SM, Clarke DR, Jacobs JP, Jacobs ML, Lacour-Gayet FG, Pizarro

CP, Welke KF, Maruszewski B, Tobota Z, Miller WJ, Hamilton L , Peterson

ED, Mavroudis C, Edwards FH. An empirically based tool for analyzing

mortality associated with congenital heart surgery. The Journal of

Thoracic and Cardiovascular Surgery, 2009 Nov;138(5), November 2009.


STAT Mortality Score and Categories

were developed based on analysis of

77,294 operations entered in the STS Congenital

Heart Surgery Databases and the EACTS Congenital

Heart Surgery Database

EACTS = 33,360 operations

STS = 43,934 operations







Procedure-specific mortality rate estimates were calculated using a Bayesian model that adjusted for small denominators.

Operations were sorted by increasing

risk and grouped into 5 categories that

were designed to • minimize within-category variation

and

• maximize between-category variation







0

5

10

15

20

% Mortality

% Mortality 0.78 2.1 3.4 8.5 19.9

1 2 3 4 5STAT Category

Combined ECHSA/EACTS and STS Congenital Heart Surgery Databases:

111,494 index cardiac operations

Jacobs JP, Jacobs ML, Maruszewski B, Lacour-Gayet FG, Tchervenkov CI, Tobota Z, Stellin G, Kurosawa H,

Murakami A, Gaynor JW, Pasquali SK, Clarke DR, Austin EH 3rd, Mavroudis C. Initial application in the EACTS

and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity

adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg. 2012 Nov;42(5):775-80. doi:

10.1093/ejcts/ezs026. Epub 2012 Jun 14. PMID: 22700597.

STS Congenital Heart Surgery

Database Mortality Risk ModelVariable

Age a

Primary procedure b

Weight (neonates and infants)

Prior cardiothoracic operation

Any non-cardiac congenital anatomic abnormality (except ‘Other noncardiac congenital abnormality’ with code value = 990)

Any chromosomal abnormality or syndrome (except ‘Other chromosomal abnormality’ with code value = 310 and except ‘Other syndromic abnormality’ with code value = 510)

Prematurity (neonates and infants)

Preoperative Factors

Preoperative/Preprocedural mechanical circulatory support (IABP, VAD, ECMO, or CPS) c

Shock, Persistent at time of surgery

Mechanical ventilation to treat cardiorespiratory failure

Renal failure requiring dialysis and/or Renal dysfunction

Preoperative neurological deficit

Any other preoperative factor (except ‘Other preoperative factors’ with code value = 777) d

a Modeled as a piecewise linear function with separate intercepts and slopes for each STS-defined age group (neonate, infant, child, adult). b The model adjusts for each combination of primary procedure and age group. Coefficients obtained via shrinkage estimation with The Society of Thoracic Surgeons–European Association for Cardio-Thoracic Surgery (STS-EACTS [STAT]) Mortality Category as an auxiliary variable. c CPS = cardiopulmonary support; ECMO =extracorporeal membrane oxygenation; IABP = intraaortic balloon pump; VAD = ventricular assist device

d Any other preoperative factor is defined as any of the other specified preoperative factors contained in the list of preoperative factors in the data collection form of the STS Congenital Heart Surgery Database, exclusive of 777 = ‘Other preoperative factors’.

• All index cardiac operations in the STS-CHSD

(January 1, 2010–December 31, 2013) were

eligible for inclusion.

• Isolated PDA closures in patients <2.5kg were

excluded, as were centers with >10%

missing data and patients with missing data

for key variables.


Database Mortality Risk Model

52,224 operations

from 86 centers were

included



Model CovariatesDevelopment Sample C-Stat

Validation Sample C-Stat

1 STAT Levels C = 0.772 C = 0.7872 STAT Levels +

age and weightC = 0.818 C = 0.817

3 STAT Levels + age and weight +patient factors

C = 0.862 C = 0.852

4 Primary procedure + age and weight

C = 0.846 C = 0.831

(Final Model) Primary procedure + age and weight +patient factors

C = 0.875 C = 0.858








C = 0.862 C = 0.852


C = 0.846 C = 0.831


C = 0.875 C = 0.858








C = 0.862 C = 0.852


C = 0.846 C = 0.831


C = 0.875 C = 0.858








C = 0.862 C = 0.852


C = 0.846 C = 0.831


C = 0.875 C = 0.858








C = 0.862 C = 0.852


C = 0.846 C = 0.831


C = 0.875 C = 0.858








C = 0.862 C = 0.852


C = 0.846 C = 0.831


C = 0.875 C = 0.858



39

Fig 1. Distribution of hospital-specific observed-to-expected (O/E)

ratios for operative mortality with 95% confidence intervals (gray

lines).



40



lines).



41



lines).



42



lines).



43



lines).



44

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality

Number (%) Number (%) Number (%) Number (%)

80% Confidence Intervals

86 (100%) 19 (22%) 52 (60%) 15 (17%)

90% Confidence Intervals 86 (100%) 13 (15%) 63 (73%) 10 (12%)



What about Confidence

Intervals

45

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality



86 (100%) 19 (22%) 52 (60%) 15 (17%)





Intervals

46

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality



86 (100%) 19 (22%) 52 (60%) 15 (17%)





Intervals

47

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality



86 (100%) 19 (22%) 52 (60%) 15 (17%)





Intervals

48

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality



86 (100%) 19 (22%) 52 (60%) 15 (17%)





Intervals

49

Total

Programs

Programs with

higher-than

expected

mortality

Programs with

same-as

expected

mortality

Programs with

lower-than

expected mortality



86 (100%) 19 (22%) 52 (60%) 15 (17%)





Intervals

Endorsed by National Quality Forum:

• NQF is a multistakeholder, nonprofit, membership-based organization that aims to improve the quality of health care through the preferential use of only the most valid performance measures

• An NQF endorsement is the gold standard for health care quality measures, and NQF endorsed measures are recognized by the national health care community as “best in class,” evidence-based, and valid.

• https://www.qualityforum.org/Home.aspx



https://www.qualityforum.org/Home.aspx

STS Congenital Heart Surgery Database Participants

January 1, 2013 to December 31, 2016

One Star Programs = 18

Two Star Programs = 74

Three Star Programs = 11

No Star Rating = 13

Congenital Public Reporting Numbers

Data Update Participants %

Round 12014 Fall Harvest

January 2015 25 23%

Round 22015SpringHarvest

August 2015 38 33%

3/24/201668/113 60.2%

4/23/2017 74 / 117 63.2%

Current Numbers (9/29/2017)78 / 117

66.6%

Participation in Public Reporting

%

Enrolled

Unique STS consents / US & Canada

participants

(as of Friday, September 29, 2017)

Adult

Cardiac 59.9% 658 / 1,098

Congenital 66.6% 78 / 117

Thoracic 18.1% 52 / 287

Basic Principles

1. Variation in outcomes exist

Basic Principles


2. Patients and their families have the right to

know the outcomes of the treatments that

they will receive.

Basic Principles


2. Patients and their families have the right to

know the outcomes of the treatments that

they will receive.

3. It is our professional responsibility to share

this information with them in a format that they

can understand.

Basic Principles

The solution to risk aversive behavior is proper risk adjustment.

The solution to fear of stifling innovation is proper risk adjustment.

Our tools for public reporting are not perfect, but they are the BEST available (and these tools will improve)…..

Adjusted Mortality Rates (AMR’s)

The adjusted mortality rate (AMR) is an estimate

(based on a statistical model) of what the hospital’s

mortality rate would be if its observed

performance was extrapolated to the overall STS

case mix (specifically, the mix of age, weight,

procedure types, and other model specific variables

including prior cardiothoracic operations, non-

cardiac congenital anatomic abnormalities,

chromosomal abnormalities, syndromes, and

preoperative risk factors).

Adjusted Mortality Rates (AMR’s)

AMR is calculated by the following

formula:

AMR of hospital = O/E ratio of hospital x

overall observed STS mortality rate*.

If O/E x STS mortality rate is greater

than 100%, the AMR is set to 100%.

Ongoing Refinement in Risk Model

1. Coefficients are updated every 6 months

2. Specific coefficients being developed for:

– Non-cardiac Congenital Abnormalities

– Chromosomal Abnormalities

– Syndromes

3. Multi-domain composite

– Mortality

– Major morbidity

– Postoperative Length of Stay



Accomplishments












STS CHSD Data Verification

10% of sites audited each year

Analysis of general variables – data completeness rate of 99.94% and

– overall data agreement rate of 98.05%

Analysis of mortality variables– data completeness rate of 100% and

– overall data agreement rate of 99.09%



Accomplishments












Jacobs JP. (Editor). 2008 Cardiology in the Young

Supplement: Databases and The Assessment of

Complications associated with The Treatment of Patients

with Congenital Cardiac Disease, Prepared by: The Multi-

Societal Database Committee for Pediatric and Congenital

Heart Disease, Cardiology in the Young, Volume 18, Supplement

S2, pages 1 –530, December 9, 2008.

Collaboration Between Subspecialties

Accomplishments

1) STS Congenital Heart Surgery Database

2) IMPACT Database of the American College of

Cardiology (Interventional Cardiology)

3) MAP-IT: Multicenter Pediatric and Adult Congenital EP

Common Language = Nomenclature

4) Pediatric Cardiac Critical Care Consortium (PC4)

5) Congenital Cardiac Anesthesia Society Database

(CCAS)

“Science tells us what we can do;

Guidelines what we should do; &

Registries what we are actually doing.”

Outcomes Analysis

Patient SafetyQuality

Improvement

Documents

National Quality Standards and Statistical EvidenceNational Quality Standards Roles of Jeffrey P. Jacobs, MD: Chair, STS National Database Workforce Chair, CHSS Committee on Quality