Jonathan H Chen MD, PhD Muthuraman Alagappan MD

Mary K Goldstein MD, MS Steven M Asch MD, MPH Russ B Altman MD, PhD

General Medical Disciplines, Department of Medicine,

Stanford University

ClickWell Care, Hospitalist Group

Undesirable Variability Patients receive only ~55% of recommended care1

E.g., ICU care in last 6 months varies >5x by state2

Practice Uncertainty

RCTrials cost >$15M to answer individual questions

~11% of guideline recs have high quality evidence3

1. McGlynn E, Asch S, Adams J, et al. Quality of health care delivered to adults in the United States. N. Engl. J. Med. 2003 2. http://www.darthmouthatlas.org 3. Tricoci P, Allen JM, Kramer JM, et al. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 2009

2

Meaningful Use

E.g., Order sets and alerts

Features of Success1

Computer-based

Clinical workflow

Time and location of decision-making

Recommendations not assessments

Open Challenges

Top-down distribution constraints

Mine clinical data from the bottom-up?2

3

1. Kawamoto K, Houlihan C a, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems. BMJ 2005

2. Sittig DF, Wright A, Osheroff J a, et al. Grand challenges in clinical decision support. J. Biomed. Inform. 2008

Recommender Systems

Collaborative Filtering

Automated Decision Support

Distributing crowd sourced info

Predict real clinical practices?

Collaborative Filter

Doctor Collective Patient

Medical Record

Linden G, Smith B, York J. Amazon.com recommendations: item-to-item collaborative filtering. Smith B, ed. IEEE Internet Comput. 2003

4

5 years Inpatient Data (2009-2013)

>74,000 Patients >11,000 Unique Orders >16,000,000 Order Instances

>7,800 Medications >36,000,000 Lab Results

>1,600 Labs >1,100,000 Diagnosis Codes

>1,100 Imaging >2,200,000 Team Assignments

>1,000 Nursing / Procedures

Commerce Customer (Basket) Purchases

Medicine Patient (Encounter) Orders 5

Item Date Category Description

12/25/2010 12:07 ECG ECG 12-LEAD

12/25/2010 12:13 Imaging XR CHEST 1V 12/25/2010 12:13 Lab BLOOD GASES, ARTERIAL

12/25/2010 12:13 Med (Intravenous) Sodium Chloride (Intravenous) 12/25/2010 12:13 Microbiology BLOOD CULTURES

12/25/2010 12:13 PoCare Test POC ISTAT TROPONIN I

12/25/2010 12:13 Lab Result ALBUMIN, SER/PLAS(ALB) (Low)

12/25/2010 12:13 Lab Result WBC(WBC) (High)

12/25/2010 12:13 Lab Result SODIUM, SER/PLAS (Low) 12/25/2010 12:13 Lab Result UREA NITROGEN, SER/PLAS (High)

12/25/2010 12:13 Lab Result NEUT, ABS (High)

12/25/2010 12:13 Lab Result WBC, URINE (Abnormal)

12/25/2010 12:13 Lab Result BACTERIA(UBAC) (Abnormal)

12/25/2010 12:13 Lab Result PROCALCITONIN(PROCTL) (High)

12/25/2010 12:28 PoCare Test POC ISTAT,VENOUS BLOOD GASES AND LACTATE (CG4)

12/25/2010 12:28 Med (Intravenous) Methylprednisolone (Intravenous)

12/25/2010 12:35 Med (Inhalation) Levalbuterol (Inhalation)

12/25/2010 12:54 Med (Intravenous) Levofloxacin (Intravenous)

12/25/2010 12:55 Med (Intravenous) Piperacillin/Tazobactam (Intravenous)

12/25/2010 12:55 Med (Intravenous) Vancomycin (Intravenous)

12/25/2010 13:36 PROBLEM_LIST Pneumonia, organism unspecified

12/25/2010 13:36 ADMIT_DX Shortness of breath

12/25/2010 13:36 Admission ADMIT/PLACE PATIENT

12/25/2010 15:03 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER

12/25/2010 15:04 Microbiology URINE CULTURE 12/25/2010 20:37 Imaging US VEIN LOWER EXTREMITY BILAT R/O DVT

12/25/2010 20:37 Isolation DROPLET ISOLATION

12/25/2010 20:37 Lab LACTIC ACID

12/25/2010 20:37 Lab RESP DFA PANEL-REFLEX PCR

12/25/2010 20:37 Med (Inhalation) Albuterol/Ipratropium (Inhalation)

12/25/2010 20:37 Med (Intravenous) Pantoprazole (Intravenous)

12/25/2010 20:37 Med (Oral) Acyclovir (Oral) 12/25/2010 20:37 Med (Oral) Gabapentin (Oral)

12/25/2010 20:37 Med (Subcutaneous) Heparin (Subcutaneous)

12/25/2010 20:37 Microbiology FUNGAL CULTURE

12/25/2010 20:37 Microbiology RESPIRATORY CULTURE + Gram Stain

12/25/2010 20:37 Respiratory Care RESP - OXYGEN ADMINISTRATION

12/25/2010 22:29 Med (Intravenous) DEXMEDETOMIDINE IV INFUSION

12/25/2010 22:30 Respiratory Care RESP - NON INVASIVE POS. PRESS VENT (BIPAP/CPAP)

12/25/2010 23:58 Lab HEPARIN ACTIVITY LEVEL

12/25/2010 23:58 Med (Intravenous) Heparin (Intravenous)

B+ B- Total

A+ NABt NA-NABt NA

A- NB-NABt N-NA-NB+NABt N-NA

Total NB N-NB N

6

Episode Mining Apriori Algorithm (Co)-Occurrence Counts

NA = Patients with event A

NABt = Patients with event A followed by event B, within time t

N = Total number of patients

Item Description Prevalence PPV OR P

POC Troponin I 16% 71% 14.4 2E-148

ECG 12-Lead 52% 93% 12.7 8E-80

Nitroglycerin (Sublingual) 1% 9% 11.2 2E-25

Consult Cardiology 5% 28% 9.6 8E-64

D – Dimer 1% 10% 9.4 5E-24

Aspirin (Oral) 24% 68% 7.2 2E-85

CK, MB 16% 51% 6.1 2E-68

Troponin I 24% 62% 5.6 3E-67

Clopidogrel (Oral) 6% 21% 4.7 2E-27

Cardiac Catheterization 3% 10% 4.4 7E-14

Heparin Activity Level 6% 19% 3.8 2E-20

Lipid Panel 9% 24% 3.7 5E-24

NT – proBNP 10% 26% 3.4 6E-23

Nitroglycerin (Topical) 2% 6% 3.2 8E-07

Dobutamine Stress Echo 1% 3% 3.0 6E-04

… … … … …

Top orders following an admission diagnosis of “Chest Pain” Co-occurrences within 24 hours, ranked by Odds Ratio (OR) 7

Item Date Category Description

12/25/2010 12:07 ECG ECG 12-LEAD

12/25/2010 12:13 Imaging XR CHEST 1V 12/25/2010 12:13 Lab BLOOD GASES, ARTERIAL

12/25/2010 12:13 Med (Intravenous) Sodium Chloride (Intravenous) 12/25/2010 12:13 Microbiology BLOOD CULTURES

12/25/2010 12:13 PoCare Test POC ISTAT TROPONIN I

12/25/2010 12:28 PoCare Test POC ISTAT,VENOUS BLOOD GASES AND LACTATE (CG4)

12/25/2010 0:00 Lab Result ALBUMIN, SER/PLAS(ALB) (Low)

12/25/2010 0:00 Lab Result WBC(WBC) (High) 12/25/2010 0:00 Lab Result SODIUM, SER/PLAS (Low)

12/25/2010 0:00 Lab Result UREA NITROGEN, SER/PLAS (High)

12/25/2010 0:00 Lab Result NEUT, ABS (High)

12/25/2010 0:00 Lab Result WBC, URINE (Abnormal)

12/25/2010 0:00 Lab Result BACTERIA(UBAC) (Abnormal)

12/25/2010 0:00 Lab Result PROCALCITONIN(PROCTL) (High)

12/25/2010 12:28 Med (Intravenous) Methylprednisolone (Intravenous)

12/25/2010 12:35 Med (Inhalation) Levalbuterol (Inhalation)

12/25/2010 12:54 Med (Intravenous) Levofloxacin (Intravenous)

12/25/2010 12:55 Med (Intravenous) Piperacillin/Tazobactam (Intravenous)

12/25/2010 12:55 Med (Intravenous) Vancomycin (Intravenous)

12/25/2010 13:36 PROBLEM_LIST Pneumonia, organism unspecified

12/25/2010 13:36 ADMIT_DX Shortness of breath

12/25/2010 13:36 Admission ADMIT/PLACE PATIENT

12/25/2010 15:03 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER

12/25/2010 15:04 Microbiology URINE CULTURE 12/25/2010 20:37 Imaging US VEIN LOWER EXTREMITY BILAT R/O DVT

12/25/2010 20:37 Isolation DROPLET ISOLATION

12/25/2010 20:37 Lab LACTIC ACID

12/25/2010 20:37 Lab RESP DFA PANEL-REFLEX PCR

12/25/2010 20:37 Med (Inhalation) Albuterol/Ipratropium (Inhalation)

12/25/2010 20:37 Med (Intravenous) Pantoprazole (Intravenous)

12/25/2010 20:37 Med (Oral) Acyclovir (Oral) 12/25/2010 20:37 Med (Oral) Gabapentin (Oral)

12/25/2010 20:37 Med (Subcutaneous) Heparin (Subcutaneous)

12/25/2010 20:37 Microbiology FUNGAL CULTURE

12/25/2010 20:37 Microbiology RESPIRATORY CULTURE + Gram Stain

12/25/2010 20:37 Respiratory Care RESP - OXYGEN ADMINISTRATION

12/25/2010 22:29 Med (Intravenous) DEXMEDETOMIDINE IV INFUSION

12/25/2010 22:30 Respiratory Care RESP - NON INVASIVE POS. PRESS VENT (BIPAP/CPAP)

12/25/2010 23:58 Lab HEPARIN ACTIVITY LEVEL

12/25/2010 23:58 Med (Intravenous) Heparin (Intravenous)

Query

Set

Valid

ation S

et

Separate validation set Identify real order set use Use initial data as query Predict subsequent orders Compare against next

actual 24 hour orders

Recommender

Test

Set

8

Rank Category Description 1 Lab NT - PROBNP

2 Med (Inhalation) Albuterol/Ipratropium (Inhalation)

3 Med (Intravenous) Furosemide (Intravenous)

4 Med (Oral) Aspirin (Oral) 5 Diet DIET NPO 6 Lab CK, MB (MASS)

7 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER

8 Med (Subcutaneous) Heparin (Subcutaneous)

9 Med (Subcutaneous) Enoxaparin (Subcutaneous)

10 Med (Oral) Prednisone (Oral)

Precision Recall (Sensitivity) Weighted Recall

12.0%

7.8%

8.2%

27.7%

21.3%

15.9%

P-value (Chi2)

Relative Risk (RR)

Odds Ratio (OR)

Pos Pred Value (PPV)

Prevalence

Order Sets

18.9%

19.6%

18.4%

53.2%

42.9%

35.1%

27.3%

28.5%

28.6%

20.9%

9.7%

20.3%

𝑇𝑃𝑖𝑘𝑖=1

𝑇𝑃𝑖𝑘𝑖=1 + 𝐹𝑃𝑖

𝑘𝑖=1

𝑇𝑃𝑖𝑘𝑖=1

𝑇𝑃𝑖𝑘𝑖=1 + 𝐹𝑁𝑖

𝑘𝑖=1

1

𝑁𝑖𝑇𝑃𝑖

𝑘𝑖=1

1

𝑁𝑖𝑇𝑃𝑖

𝑘𝑖=1 +

1

𝑁𝑖𝐹𝑁𝑖

𝑘𝑖=1

9

“…invariably, simple models and a lot of data trump more elaborate models based on less data.”

“…use available large-scale data rather than hoping for annotated data that isn’t available.”

10

2009 Top Items

Overlap at Rank Rank

Fraction Overlap 2012 Top Items

Levofloxacin (IV) 0 1 0.00 Azithromycin (IV)

Blood Culture (2x Aerobic) 1 2 0.50 Levofloxacin (IV)

Blood Culture ((An)Aerobic) 1 3 0.33 Vancomycin (IV)

Respiratory Virus Panel 1 4 0.25 Piperacillin-Tazobactam (IV)

Respiratory Isolation 1 5 0.20 Ceftriaxone (IV)

Oseltamivir (Oral) 1 6 0.17 Azithromycin (Oral)

Vancomycin (IV) 2 7 0.29 Albuterol-Ipratropium (Inh)

Respiratory Culture 2 8 0.25 Sodium Chloride (Inhalation)

Albuterol-Ipratropium (Inh) 4 9 0.44 Blood Culture (2x Aerobic)

CBC w/ Diff 5 10 0.50 Blood Culture ((An)Aerobic)

Influenza A PCR 5 11 0.45 Ipratropium (Inhalation)

… … … … … 11

Predicting 2013 Admission Orders with Different Training Data Samples

41%

46%

51%

56%

Re

call

2013 (12 Months)

2012 Base

2009 (12 Months)

21%

23%

25%

27%

0 10 20 30 40 50

Pre

ciso

in

Training Duration (Months)

2013 (12 Months)

2012 Base

2009 (12 Months)

12

“Prediction is difficult, especially about the future.”

Evolving clinical knowledge and practices

Recent data more useful than simply more data

Predicting 2013 Admission Orders with Decaying Weight to Historical Data

50.5%

51.0%

51.5%

52.0%

52.5%

Re

call

Non-Decaying Model 12 MonthsDecaying Models 48 MonthsNon-Decaying Model 48 Months

26.0%

26.5%

27.0%

0 50 100 150 200

Pre

cisi

on

Decay Window (Months)

Non-Decaying Model 12 MonthsDecaying Models 48 MonthsNon-Decaying Model 48 Months

13

Ni: Item count during i-th interval z: Total number of intervals w: Decay window

𝑁 = 𝑁𝑖 1 −1

𝑤

𝑧−𝑖𝑧

𝑖=1

Decaying estimate for N

“Half-life” ~ 4 months

NIH Big Data 2 Knowledge K01ES026837 (2015-2020) Mentored Career Development Award in Biomedical Big Data Science Russ B. Altman, Mary K. Goldstein, Steven M. Asch Nigam Shah, Lester Mackey, Mike Baiocchi (Vinnie Liu: Writing sample)

Veteran Affairs, Advanced Fellowship in Medical Informatics Doug Owens and Ruth C. Cronkite

Stanford Translational Research and Applied Medicine Grant Dean Felsher and Joanna E. Liliental

Stanford Learning Healthcare Systems Innovation Fund Stanford Clinical and Translational Science Award (CTSA) to Spectrum (UL1

TR001085), NIH National Center for Advancing Translational Sciences (NCATS)

Prof. Russ B Altman and Research Group NIH/National Institute of General Medical Sciences PharmGKB resource:

R24GM61374, LM05652, GM102365

STRIDE Data Group ▪ Tanya Podchiyska, Lee Ann Yasukawa, Gomathi Krishnan, Susan Weber ▪ National Center for Research Resources and the National Center for Advancing

Translational Sciences, National Institutes of Health: UL1 RR025744

Speaker discloses no conflicts of interests Content does not necessarily represent the views of the NIH, VA, or Stanford Health Care

Students Muthu Alagappan Gustavo Chavez

14

Property Death ICU Life Support Discharge / Length of Stay Readmission Evaluation

Period 30 days 2 days 4 days 1 week 2 days 4 days 1 week 2 week 30 days

Outcome Rate

2.2% 1.0% 1.6% 2.3% 22.2% 58.6% 79.2% 90.1% 8.3%

ROC AUC (c-stat) 0.91 0.87 0.83 0.84 0.69 0.75 0.80 0.82 0.68

ROC AUC 95% CI

0.88- 0.93

0.80-0.95

0.76-0.90

0.79- 0.89

0.66-0.72

0.72-0.77

0.78-0.83

0.79-0.85

0.64- 0.73

Related Prediction

Models

APACHE, MPM, SAPS

CURB-65, PSI, SCAP, REA-ICU

Tu et al 1995, ISS, NISS, CSI

Amarasingham et al 2010,

LACE, CMS

ROC AUC for

Related 0.75-0.90 0.69-0.81

0.69 (Only Tu et al reports as ROC AUC)

0.56-0.72

15

Based upon separate validation patients’ first 24 hours of

clinical items, score predicted outcomes by estimated PPV

Frequency Conditional Baseline Ratio Rank Description Confidence Prevalence Lift P

1 Comfort Care Measures 0.11 0.02 5.18 3E-89

2 Lactic Acid (High) 0.46 0.11 4.15 3E-296

3 Calcium Chloride Infusion 0.06 0.01 3.94 9E-32

4 Norepinephrine IV Infusion 0.14 0.04 3.69 4E-71

5 Consult Palliative Care 0.15 0.04 3.63 3E-71

6 Lung Protective Ventilation 0.07 0.02 3.61 7E-35

7 pH Venous (Low) 0.22 0.06 3.50 9E-103

8 Osmolality, Serum (High) 0.06 0.02 3.43 3E-27

9 INR (High Panic) 0.05 0.02 3.30 3E-21

10 Cont Renal Replacement 0.10 0.03 3.24 4E-40

16

Inverted query results for top “recommendations” for items preceding patient death ranked by freqRatio(B|A)week

Chemo (V58.11)

Pneumonia (486)

Joint Pain (719.4)

0%

5%

10%

15%

20%

25%

30%

0 0.2 0.4 0.6 0.8 1

Ave

rag

e W

eig

hte

d R

eca

ll

Rank Biased Overlap (RBO)

R = 0.63

R² = 0.39

P<10-5

Order Prediction

Accuracy vs. Practice

Stability

240 261

178 214

228 532

187 245

458 311

191 283

307 365

196 202

662 442

388 938

351 845

239 267

257

0.33 0.42 0.44 0.44 0.47 0.48 0.48 0.51 0.51 0.53 0.54 0.54 0.54 0.55 0.57 0.57 0.57 0.58 0.59 0.59 0.59 0.60 0.66 0.68 0.68

0 250 500 750 1000

Symptoms involving digestive system (787)Back disorders (724)

Cardiac dysrhythmias (427)Abdominal pain, unspecified site (789.00)

Osteoarthrosis, unspecified (715.9)Abdomen and pelvis symptoms (789)

Osteoarthrosis, lower leg (715.36)Chest pain, unspecified (786.50)

Abdominal pain (789.0)Chest pain (786.5)

Gastrointestinal hemorrhage (578)Fever, unspecified (780.60)

Fever and temperature dysregulation (780.6)Osteoarthrosis, localized (715.3)

Altered mental status (780.97)Other general symptoms (780.9)

Osteoarthrosis (715)Dyspnea (786.0)

Specific procedure complications (996)General symptoms (780)

Shortness of breath (786.05)Respiratory and chest symptoms (786)

Encounter for antineoplastic chemotherapy (V58.11)Encounter for procedures and aftercare (V58)

Encounter for chemotherapy/immunotherapy (V58.1)

[RBO] Number of Admissions in 2012 Admission Diagnosis (ICD9)

Top 25 Admission Diagnoses (2012) with Rank Biased Overlap (RBO) of Related Orders (2009 vs. 2012)

17

Use Rate Size Description

35.1% 51 Anesthesia - Post-Anesthesia Care (Inpatient) 27.2% 161 Medicine - General Admit 23.5% 51 General - Pre-Admission / Pre-Operative

18.4% 17 Insulin - Subcutaneous

15.7% 28 General - Transfusion

9.3% 13 General - Discharge

7.6% 150 Surgery - General Admit 6.9% 9 Emergency - Admit 6.1% 224 Intensive Care - General Admit 5.9% 147 Orthopedics - Total Joint Replacement

… … …

The most commonly used order sets

revolve around non-specific, provider-

focused care processes

18

Metric (First 24 hours of each hospitalization) Mean

Std Dev Median

Interquartile Range

A: Order Sets Used 3.0 (1.4) 3 (2, 4)

B: Orders Entered (including non-Order Set) 32.7 (15.7) 30 (22, 41)

C: Orders Entered from Order Sets 13.3 (7.9) 12 (8, 18)

D: Orders Available from Used Order Sets 129.0 (47.5) 130 (102, 153)

E: Order Set Precision = (C / D) 11% (7.2%) 9.5% (6.3%, 13.8%)

F: Order Set Recall = (C / B) 43% (20%) 42% (28%, 58%)

Metric 2009 2010 2011 2012 2013

Patients 13,493 18,459 19,070 19,327 19,523

Age (mean) 58.4 58.1 58.6 58.5 58.6

Age (std dev) 18.5 18.8 18.7 18.7 18.6

Female 52% 52% 52% 51% 51%

White 60% 63% 62% 60% 58%

Hispanic/Latino 12% 13% 13% 13% 14%

Asian 11% 11% 11% 12% 12%

Black 5% 5% 5% 5% 5%

Providers 1,709 1,892 1,917 1,798 1,821

New Providers … 41% 33% 29% 34%

Departing Providers 34% 32% 33% 33% …

Item Description Prevalence Sodium Chloride (IV) 90%

CBC w/ Diff 79% Metabolic Panel, Basic 77% Potassium Chloride (IV) 68%

Glucose (IV) 60% Docusate (Oral) 58%

Prothrombin Time (PT/INR) 58% Physical Therapy Eval 55%

Metabolic Panel, Comp. 55% Diet NPO 53%

CBC 53% Magnesium, Serum 52%

ECG 12-Lead 52% Occupational Therapy Eval 50% Partial Thromb. Time (PTT) 47%

Xray Chest 1 View 45% … …

20 Top orders by baseline prevalence across all hospitalized patients

Item Description Prevalence PPV OR P ECG 12-LEAD 52% 93% 12.7 8.E-80

CBC w/ Diff 79% 90% 2.5 5.E-10 Metabolic Panel, Comp. 55% 76% 2.6 2.E-19

POC Troponin I 19% 75% 14.4 5.E-147 Prothrombin Time (PT/INR) 58% 70% 1.7 3.E-07

Aspirin (Oral) 24% 68% 7.2 2.E-85 Metabolic Panel, Basic 77% 65% 0.5 8.E-09

Troponin I 24% 62% 5.6 3.E-67 Partial Thromb. Time (PTT) 47% 53% 1.3 1.E-02

Xray Chest 1 View 45% 52% 1.3 5.E-03 CK, MB 16% 51% 6.1 2.E-68

Xray Chest 2 View 29% 49% 2.4 5.E-18 Sodium Chloride (IV) 90% 44% 0.1 7.E-127 Magnesium, Serum 52% 40% 0.6 1.E-07

… … … … …

21

Top orders following an admission diagnosis of “Chest Pain” Co-occurrences within 24 hours, by Positive Predictive Value (PPV)

Cognitive Overload

75 RCTs + 11 systematic reviews published daily1

“The complexity of modern medicine exceeds the

capacity of the unaided expert mind.” – David Eddy

1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Med. 2010

22