Generating Problem-Oriented Summary

from Electronic Medical Records

IBM Watson HealthChing-Huei Tsou

April 14, 2016

Watson’s post-Jeopardy challenge: Healthcare

Our first domain of exploration is medical decision support

because of its mature, complex and meaningful

problem solving nature

After Watson’s win on Jeopardy!, people (outside of computer

science community) assumed that anything that could be

phrased as a question could be correctly answered by Watson:

Watson, “Given my medical record

<insert hundreds of pages of structured and unstructured

data here>

, what’s wrong with me?”

Filing System

Summarization Multi-Step ReasoningClinical Knowledge QA EMR Deep QA & Search

*Not to scale

From the generatedProblem oriented summary, physician noticed the patient’s “Creatinine level is high”

What has been done to treat his diabetic nephropathy?What else can I try?

Clinical Decision

Support SystemEMR

What are the causes of creatinine elevation?

What are the most likely causes for

this patient?

Toward a Clinical Decision Support System

EMR Analysis Medical QA Reasoning EMR QA

Problem-Oriented Summarization

Electronic Medical Records

Unstructured DataClinical Notes

Semi-structured Data, e.g.Diagnoses,

medications, lab test results

100s of notes for a

typical patient;

1,000s for older patients

with inpatient notes

Promises of Electronic Medical/Health Record

Why a Dr. went into medicine

Not why a Dr. went into medicine Prevent medical errors

Reduce health care costs

Increase administrative efficiency

Decrease paperwork

Expand access to affordable care

Today’s EMR is Broken

Digitizing medical records does not reduce physicians’ cognitive load

Today’s EMR is largely billing-oriented

Billing compliance regulations require that notes stand on their own, which

may promote duplication of text

Detailed coding = bill = better quality?

General Coding Specific Coding

428.0 CHF NOS428.9 HF NOS

428.21 Ac Systolic HF428.23 Ac-Chr Systolic HF428.31 Ac Diastolic HF428.33 Ac-Chr Diastolic HF428.41 Ac Comb S&D HF428.43 Ac-Chr Comb S&D HF

$29,716 $53,670

How EMR Summarization Can Help a Physician?

Consider a physician who is about to see a patient in an outpatient setting

Perhaps this is the first encounter for the physician with the patient, or

It has been a while since the physician has seen this patient

Before seeing the patient, the physician may want to know

What are the patient’s current problems?

When was a problem last discussed / addressed?

How a problem is being managed?

Current medications?

Related lab test results?

Most questions are problem-oriented

Problem-Oriented Medical Record (POMR) Summary

Problems List

Medications

Lab tests

“treated by”

“measured by”

“discussed in”

Procedures

Vitals

Clinical Notes & timeline

POMR, as originally defined

by Dr. Lawrence Weed in the

1960s is the official record

keeping method in most US

hospitals

Problem list is also a

mandatory section in the

CCD (continuity of care),

part of HL7’s CDA (clinical

document architecture)

standard

The key to success?

An accurate problem list

The Problem List Challenge

Unfortunately, manually maintained problem lists are not

accurate

Our assessment of existing problem lists based on a gold standard indicates the challenge

Entered Problem List Accuracy:Recall (Sensitivity) = 0.55 Precision (Positive Predictive Rate) = 0.28

Ground-TruthProblem

Problems on the entered list

Resolved Problem

Acute Problem

Problems added

for billing purpose

Patient’s pre-

existing problems

No time to

update the list

FN

FPTP

Rule-out diagnoses

Automated Problem List Generation

Problem List: A list of current and active diagnoses as

well as past diagnoses relevant to the current care of

the patient

CMS (center of medicare and medicaid services)

Meaningful Use Stage 1

Problem List Definition

Problem-List Ground-Truth Annotation GuidelinesWhat to include:1. Chronic disease like diabetes, hypertension, hyperlipidemia etc.2. History of cardiovascular events such as CVA, MI, DVT, PE.3. Non-injury related musculoskeletal conditions like degenerative disc disease, osteoarthritis, osteoporosis, and rickets4. History of drug or alcohol ABUSE5. All psychiatric diagnoses6. Obesity and obesity related problems like sleep apnea, fatty liver disease etc.7. Resolved problems of high importance such as recurrent PNA, anemia, etc.8. Complications from other disease processes, such as diabetic neuropathy, CKD from hypertension etc9. malignant Neoplasms (or history of) regardless of patient status and any benign neoplasms that need to be monitored

What should NOT be included:1. all injuries2. resolved problems of either low importance, or those which have been corrected by surgery(bronchitis, pneumonia, cholecystitis with cholecystectomy,

hernia that has been surgically corrected, appendicitis with appendectomy, etc).3. Most dermatologic conditions including warts, transient skin rashes of low importance that are resolved. Only exception to this is Acne (regardless of severity)

is included.4. Signs or symptoms of disease; chest pain, headache, abdominal pain, epistaxis, hematuria, etc. Usually these will have some corresponding diagnosis. If not

then it isn’t included. Only exception is Lumbago, which because of its usual chronicity IS included. 5. Severity of disease, as these tend to wax and wane in many chronic problems.6. Cause of death or anything from an autopsy report

Where to take information from:1. Any clinical note, operative report, telephone encounter, etc, where a specific diagnoses is discussed.2. Do not make inferences. Ie, if a note says fasting glucose of 156, unless it explicitly says this patient has diabetes, leave the diagnosis off 3. Words like probable or suspected before a given diagnoses are situation dependent. Sometimes a later note will confirm or refute that diagnosis.

Tips:1. Remember that notes have places for allergies, past surgeries, procedures, etc. so leaving things off of a problem list doesn’t mean the information isn’t

available.2. Try to make the diagnosis as concise as possible, abbreviations are acceptable.3. If you’re unsure then include it and it can always be removed during adjudication

Guidelines are subject to

explanation and extensive domain

knowledge is required

EMRA Problem List Generation

Candidate Generation Scoring & Ranking

Find everything that looks

like a disorder from the

clinical notes

Look for contextual

information and

supporting evidences

Clinical Note

(Watson) Annotated Clinical Note / Entity Linking

Parsing

Sections

Paragraphs

Part-of-speech

Entity-Linking

Recognition

Disambiguation

Negation Detection

Context-aware Computing

Given the context, we have no problem reading the

sentences above, even though the characters H and A

(and B and13) are identical

Context in EMR

Word

Sentence

Section

Note

Medication & Labs

Similar Pattern in Other EMRs

Hypertension

Hypertension: Yes

Assessment and Plan

Hypertension: Yes

Mentioned in several other notes

Taking HTN drugs, elevated BP

Other patients with similar

pattern has been diagnosed

with hypertension

19

EMRA Problem List Accuracy:

Recall (Sensitivity) = 0.84

Precision (Positive Predictive Rate) = 0.52Recall Oriented F2 = 0.75

Entered Problem List Accuracy:

Recall (Sensitivity) = 0.55

Precision (Positive Predictive Rate) = 0.28

GroupingCandidate Generation

Feature Generation

Info

rmat

ion

Ex

trac

tio

nTe

xt

Segm

en-

tati

on

Scoring / WeightingEMR

Clinical Factors

Extraction

CUI Confidence

Note Section

Notes

Structured Data

(Medications, Orders, Lab,

etc)

CUIs of unique Disorders (100s)

Candidate Problems (10s)

CUIs of unique Medications (10s), Orders, Lab, etc.

Merging and

Clustering Closely Related

Problems

Term Frequency

Rel

atio

nsh

ip

LSA / DSRD

CUI Path

LSA / DSRD

CUI PathM

eds

Lab

s

Score

1.0

0 0.4Confidence

Score

1.0

0 10

Term Frequency

Score

1.0

0 0.3LSA Score

Score

1.0

0 A may treat B

Path Pattern

Score

1.0

0 PMH

Note Section

Note Type

EMRA Problem List Generation

EMRA Problem List Generation

20

EMRA Problem List Accuracy:

Recall (Sensitivity) = 0.70

Precision (Positive Predictive Rate) = 0.73Precision Oriented F1 = 0.72

Entered Problem List Accuracy:

Recall (Sensitivity) = 0.55

Precision (Positive Predictive Rate) = 0.28

GroupingCandidate Generation

Feature Generation

Info

rmat

ion

Ex

trac

tio

nTe

xt

Segm

en-

tati

on

Scoring / WeightingEMR

Clinical Factors

Extraction

CUI Confidence

Note Section

Notes

Structured Data

(Medications, Orders, Lab,

etc)

CUIs of unique Disorders (100s)

Candidate Problems (10s)

CUIs of unique Medications (10s), Orders, Lab, etc.

Merging and

Clustering Closely Related

Problems

Term Frequency

Rel

atio

nsh

ip

LSA / DSRD

CUI Path

LSA / DSRD

CUI PathM

eds

Lab

s

Score

1.0

0 0.4Confidence

Score

1.0

0 10

Term Frequency

Score

1.0

0 0.3LSA Score

Score

1.0

0 A may treat B

Path Pattern

Score

1.0

0 PMH

Note Section

Note Type

EMRA in Action

EMR Summarization

Watson generates and groups Problems by clinical relevance

Watson groups medications by clinical relevance

Each panel contains answers to a pre-defined question

Context-aware User Interface

Labs show elevated glucose and A1C among

the others…

When a problem is selectedCurrent and related meds

are highlighted

Relevant notes are highlighted

Is the patient's diabetes well-controlled?

What was patient's last HbA1c? When was it taken?

Patient's hemoglobin A1c is red indicating it is not within normal range.

Patient’s HbA1c has been high except for a single reading in 2013, so

patient's diabetes has NOT been well-controlled.

A1C went down, why?

A1C went up, why?

A1C went down; why?

A1C went up in most recent test despite being on Victoza (liraglutide);

why?

Endocrinology note on 03/06/2013

Endocrinology note on 07/17/2013

EMRA makes it easy

to find and bring up

relevant notes

Is the patient's diabetes well-controlled?

Semantic Find

Acute problems are normally not considered as problems, and don’t show

up in the Summarization UI

Patient come in complaining of hearing problem

has patient experienced this before?

Was patient started on any treatment?

Quality Assessment

Quality Assessment

“I’d consider Watson extremely useful if it can

find one important problem that is missed by

physician”

Neil Mehta. M.D., Internist, Cleveland Clinic

Quality Assessment

6 Cleveland Clinic physicians

reviewed 15 EMRs to

generated their own problem

lists, and then compared and

rated the problem lists

each physician reviewed 5 EMRs, and each EMR is reviewed by 2 physicians

Watson generated lists were given after physicians completed their own list. Physicians were asked to rate the Watson generated problems one by one and as a whole

for each problem, is it correct? Is it on your list? If correct, how important is it?

as a whole, rate each list from 1-10 (Likert scale)

Very Important

Ground Truth

Physician

Watson

Important

Somewhat Important

Not at all Important

Quality Assessment

Manually Maintained

Physician Generated

WatsonGenerated

Average Rating

Current System: 5.8

Watson: 7.4

Physician: 8.4 *The differences are statistical Significant (p=0.02)

Quality Assessment

Simple linear regression indicates the most important factor to

higher Watson rating is “Percentage of very important

problems that are missed by physician and found by Watson”

In average Watson found 1.2 very important or important

problem missed by physician per EMR (avg. 6 problems)

Specialty Specific and/or

Personalized

Type of False-Positive Problems

Transient problem

51%Correct

21%

Redundant Problem

11%

Certainty error5%

System error4%

Noise4%

Negation error3%

Human error1%

Error analysis showed most of the false-positives are “transient problems”

Transient problems are true findings or disorders of the patient that are less important to the medical care

Minor / self-limited problem

waxing and waning, e.g. seasonal

Resolved

The definition is somewhat subjective

a resolved problem to one physician

may be a significant past medical

history to another physician

CMS (center of medicare and medicaid services) Meaningful Use

Stage 1

Problem List: A list of current and active diagnoses as well as past diagnoses relevant to the current care of the patient

Problem List Definition

Every known findings / risk factors

/ disorders of the patient

• “ideal” problem list for a nephrologist

• The blue list contains too many irrelevant problems

• “ideal” problem list from an internist

• The green list is too

specific and not comprehensive

The Problem List Challenge

Cardiovascular

Digestive

Bo

dy S

yste

m

Endocrine

Respiratory

Genitourinary

The Problem List Challenge

Cardiovascular

Digestive

Bo

dy S

yste

m

Endocrine

Respiratory

Genitourinary

Active Learning (Personalized)

Active Learning (Sample Complexity)

0.5

0.6

0.7

0.8

0 50 100 150 200 250 300

F 2M

easu

re

Number of Training EMRs

Current Research Direction

LearningSupervised

(batch learning)Supervised

(active learning)

FeaturesKnowledge-based features O(100) selected using ADT

tree / boosting

Features O(1,000) extracted and selected by DNN (e.g.

auto-encoder)

Temporal Aspect

Modeled implicitlyExplicitly clustering

multivariate time series

Today Work in Progress