Temporal Reasoning and Planning in Medicine

Visualization and Exploration of

Time-Oriented Medical Data

Yuval Shahar, M.D., Ph.D.

The Need for Visualization of Information

• To be effective, care providers and other decision makers need to be able to visualize both clinical data and their multiple levels of abstraction

• Larkin and Simon [1987]: the benefit of visual representations is mainly due to– reduction of logical computation through the use of

direct perceptual inference

– reduction of necessary search for information through the use of efficient graphical representations.

The Granularity Issue

• Not all events are measured the same way

• Example: Birthday is accurate to the day, but age is accurate to the year

• The granularity may even change over time

• Example: Age is measured first in months, then in years

Standard Time Quanta

• Year• Month• Week• Day• Hour• Minute• Second• Smaller...

Note, not all quanta can easily be expressed in terms of other quanta.

For example, how many weeks are there in a year? How many minutes are there in a month?

How do you deal with mixed granularities?

The Granularity Denial Approach

• Pretend there is no granularity problem

• Arbitrarily choose some time quantum and use that for all measurements

• Works best when the most logical quantum is large, say 1 day (or larger)

• If all you have is a day, how do you choose the right second?

Complications of Ignoring Granularity

• Suppose we know some event occurred on March 15th, 1999

• Assume that the chosen granularity is seconds

• Do we record 3/15/1999, 12:0:0?

• Or, 3/15/1999, 0:0:0?

• Or, something else?

Granularity: an Object-Oriented Approach

• Each event class is assigned a relevant granularity

• Some classes have multiple valid granularities

• A given timeline can specify only a small number of granularity levels

Object-Oriented Example

H R-H ou r-

-M in u te -

B P-H ou r-

-M in u te -

S ta tu s-D ay-

IC U S tay-W eek -

H R-D ay--H ou r-

B P-D ay--H ou r-

O n H osp . F loor-W eek -

H osp ita liza tion-M on th -

The Interval-Uncertainty Approach

• Measure each event with the most relevant granularity

• When viewing data at a finer granularity, introduce uncertainty

• Uncertainty can include up to 6 degrees of freedom (start, duration, stop)

Uncertainty Examples

• Nov. 3, 1999 can be represented as:

11/3/1999, 0:0:0 — 11/3/1999, 23:59:59

• Feb. 6, 1986 at 13:37 can be truncated:

2/6/1986

• Or rounded:

2/7/1986

More Uncertainty Examples

Start

Maximum Duration

Minimum Duration

Start Stop

Not all possible values need be stored:

Timelines

A timeline is a tuple <E, M>, where E is a finite set of events containing at least the special null event, and M is a measure function M:ER+. The measure function M assigns a temporal offset to each event in E.

Cousins, S.B., and Kahn, M.G. The visual display of temporal information. Artificial Intelligence in Medicine 3(6) (1991) 341–357

Timeline Operators

• New: creates a new timeline containing only the null event.

• Add: adds an event e to an existing timeline, may increase length of timeline as a side-effect.

• Slice: remove events from one or both ends of a timeline, moves the null event as needed.

Timeline Operators (cont.)

• Filter: remove all events not satisfying some predicate P; the null event cannot be removed

• Overlay: merges two timelines. If common events do not coincide, they are copied

Timeline Operator Examples

Input Operation Output

Slice(e1, e2)e1 e1e2 e

2Filter(“b-ness”)a1

a2

b1

b1

b2

b2

Overlay (a, a)

a b ca b c

a b c

b ca b c c’

Overlay (b, b)

Grounded Timelines

• A grounded timeline is one that can be directly mapped to a calendar (e.g., a Julian calendar)

• Slice and Filter are ‘safe’.

• Overlay and New may cause a timeline to become ungrounded

Time Line Browser

• Prototype used to display diabetes patient data over time

• Created the formal definition of, and operators for, a timeline

• Provides GUI for manipulating timeline operators

.

Sample Queries

• For the previous week, display the patient’s logbook and personal calendar

• Summarize the patient’s blood sugar at breakfast, lunch, dinner and nighttime over the past month

Logbook and Calendar

Monday Tuesday Wednesday Thursday Friday Saturday Sunday Monday

Mild Illness Hospitalized

X-ray

Work Vacation in Florida

12 13 14 15 16 17

Work

Blood Sugar Summary

B L D N

Note that multiple slices have been overlaid to produce this result

For Breakfast, Lunch, Dinner, Night:

Knowledge-Based Visualization andExploration of Time-Oriented Medical Data:

Desiderata

• Interactive composition of (temporal-abstraction) queries

• Visualization of query results

• Exploration of multiple levels of temporal abstractions

• The semantics of the query, visualization and exploration operators should be domain independent, but should use the terms and relations specific to each (e.g., medical) domain

Knowledge-Based Visualization andExploration of Time-Oriented Data:

The KNAVE-I and KNAVE-II Projects(Shahar and Cheng, 1999, 2000; Shahar et al., 2003, in press)

• KNAVE = Knowledge-Based Navigation of Abstractions for Visualization and Explanation

• Interactive queries regarding both raw data and multiple levels of time-oriented abstractions derivable from these data

• Visualization and manipulation of query results• Dynamic exploration of the results using the domain’s temporal-

abstraction ontology• The semantics of all operators do not depend on any specific

domain, but the interface uses each domain’s ontology to compute and display specific terms and explore their relations

• KNAVE accesses the data through a temporal-abstraction mediator, such as IDAN

The KNAVE-II Browsing and Exploration Interface[Shahar et al., AIM 2006]

Overall pattern

Raw clinical data

Intermediate abstractions

Medical knowledge

browser

Concept search

Moving Data Panels Around

Global Temporal-Granule Zoom (I)

Global Temporal-Granule Zoom (II)

Global Calendar-Based Zoom

Global Content-Based Zoom (I)

Global Content-Based Zoom (II)

Local Time-Sensitive Zoom

Exploration Operators• Motion across semantic links in the domain’s knowledge base by

using the semantic explorer; in particular, relations such as:

- part-of - is-a - abstracted-from - subcontext • Motion across abstraction types: state, gradient, rate, pattern

• Application of aggregation operators such as mean and distribution

• Dynamic change of temporal-granularity (e.g., days, months)

• Explanation by context-sensitive display of relevant knowledge

• “What- if” queries allow hypothetical assertion or retraction of data and examination of resultant patterns

Semantic Exploration of Temporal Abstractions

Explanation: A Classification Function

Explanation: A Persistence Function

Evaluation of the KNAVE-II Intelligent Visualization and Exploration Module

[Martins et al., AIM 2008]

Site: Palo Alto Veterans Administration Health Care System Eight clinicians with varying medical/computer use backgrounds

– A second study used 6 additional clinicians and more difficult queries

Each user was given a brief demonstration of the interface The evaluation used an online database of more than 1000 bone-marrow

transplantation patients followed for 2 to 4 years Each user was asked to answer 10 queries common in oncology

protocols, about individual patients, at increasing difficulty levels A cross-over study design compared the KNAVE-II module versus two

existing methods (in the 2nd study, users chose which): – Paper charts – An electronic spreadsheet (ESS)

Measures: – Quantitative: time to answer and accuracy of responses– Qualitative: the Standard Usability Score (SUS) and comparative ranking

The KNAVE-II Evaluation Results(Martins et al. AIM 2008)

• Direct Ranking comparison: KNAVE-II ranked first in preference by all users • Detailed Usability Scores: The Standard Usability Scale (SUS) mean scores: KNAVE-II 69,

ESS 48, Paper 46 (P=0.006) (more than 50 is user friendly)• Time to answer:

– Users were significantly faster using KNAVE-II as the level of difficulty increased, up to a mean of 93 seconds difference versus paper, and 27 seconds versus the ESS, for the hardest query (p = 0.0006)

– The second evaluation, using more difficult queries and more advanced features of KNAVE-II, emphasized the differences even further: The comparison with the ESS showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); on the average, study participants answered each of the two hardest queries 277 seconds faster using KNAVE than the ESS

• Correctness: – Using KNAVE-II significantly enhanced correctness versus using paper, especially as

level of difficulty increased, even in the initial study (P=0.01) (99% accuracy with K-II versus only 78% paper accuracy, 1st study; 92% with K-II vs. 57% for ESS, 2nd study)

– The correctness scores for KNAVE-II versus ESS in the second study, which used more difficult queries, are significantly higher for all queries (p<0.0001)

• VISualizatIon and exploration of Time-Oriented raw data and abstracted concepts for multiple subject RecordS

– Graphical queries enable end users to define the constraints for selecting the relevant population to further explore

– Knowledge-based interpretation of the data

– Visual display and interactive exploration of multiple records

– Aggregation of multiple records and creation of associations amongst subject-related [temporal] patterns

The VISITORS System(Klimov and Shahar, AMIA 2005; Klimov et al., AIM 2010)

Subject and Time Interval Queries[Klimov et al., JIIS 2010]

• Three types of queries:– Select subjects (Who had this pattern?)– Select Time Intervals (When did this pattern occur?)– Get Data (What were the data for these subjects?)

• Selection constraints include:– Demographical constraints (non-temporal): ID, age,

smoking, sex, political group, …– Time and value knowledge-based constraints: measured

parameters, interventions, temporal-abstraction concepts• Pair-wise constraints between concepts• both absolute and relative (following a reference event) time lines

– Statistical constraints: filter the subjects’ data on the basis of a specific statistical function

VISITORS: Multiple-Patient, Multiple-ConceptIntelligent Browsing and Exploration

Multiple-patients raw data

Knowledge browser

Patient groups

Distribution of derived patterns over time

Concept search

Temporal Association Charts

Data of each patient are connected by line

[Klimov et al., MIM 2009]

support and confidence of association rules indicated by width

and hue

Using Temporal Abstractions for Temporal Data Mining

• Meaningful associations typically exist among clinically meaningful, interval-based, abstract concepts (e.g., 3-5M Moderate Anemia precedes 2M Deteriorating Renal Function), rather than among time-stamped raw data such as Hemoglobin and Creatinine values

• Temporal Abstraction can be used to create time-oriented, interval-based, abstract concepts and patterns

– By using domain knowledge; e.g., the knowledge-based temporal abstraction method [Shahar, AIJ, 1997]

– By using automated temporal discretization methods [Verduijn et al., AIM, 2007; Moskovitch et al., IDAMAP, 2009]

• Interval-based abstract concepts can be then be mined to discover time-intervals related patterns (TIRPs)

• Example: The KarmaLego algorithm [Moskovitch and Shahar, IDAMAP, 2009; AMIA. 2009]; used in several domains:, such as

– Analysis of diabetes-patients data

– Prediction of getting off an ICU ventilator

– Classification of Hepatitis type from the course of the disease

Temporal Interval Related Patterns-An Example

A Temporal Interval Related Pattern (TIRP) is a conjunction of temporal relations among symbolic time intervals

{A1 o B, A1 o D, A1 m C1, A1 b C2, A1 b A2, B o D, B c C1, B b C2, B b A, C1 b C2, C1 b A, C2 o A}

0.26

0.18

0.22

0.28

0.25

0.23

0.33

0.42

0.29

Exploration of the Diabetes TIRPs Tree:Exploration of the Diabetes TIRPs Tree: An ExampleAn Example

[Moskovitch and Shahar, AMIA 2009][Moskovitch and Shahar, AMIA 2009]

Visualization of TIRPs: The KarmaLegoV Tool

• Enables browsing of a KarmaLego TIRP enumeration tree; includes several options:

– Presenting the next level, i.e., the next time-interval related to the current TIRP, and its temporal relation

– Sorting by vertical support (% of patients who have the pattern), mean horizontal support (number of instances of TIRP per patient), and interestingness measures

– Visualizing the current [mean] TIRP and its instances

– Visualizing the distributions of external static (non-temporal) properties, such as age and gender, or a classification outcome (e.g., recovery or not), for the patient class in which the TIRP was found

The KarmaLegoV Tool: An Example (I)

The KarmaLegoV Tool: An Example (II)

Knowledge-Based Visualization & Exploration

of Time-Oriented Data: Conclusions

• Interactive query, visualization, and exploration requires runtime access to the domain’s temporal-abstraction ontology

• The visualization and exploration semantics can be specific to the temporal-abstraction task, but need not be specific to the domain

• Typical examples: Computation, visualization and exploration of multiple time-oriented records, their aggregations, and their inter- and intra- temporal relations– the VISITORS system [Klimov and Shahar 2005]– The KarmaLego framework [Moskovitch and Shahar 2009]