Karin Schurink

Peter Lucas

Marc Bonten

Stefan Visscher

pneumonia

symptomssigns

mechanicalventilation

colonisation

antibiotic therapy

noyes

Incorporating Evaluation Incorporating Evaluation into the Design of a into the Design of a Decision-Support SystemDecision-Support System

UMC Utrecht

Radboud University Nijmegen

Ventilator-Associated Pneumonia

VAP

Pneumonie acquise sous ventilation mécanique

Beademings-gerelateerde Longontsteking

ContentsContents

1. Problem: incorporating evaluation into the design of a Decision-Support System (DSS)

2. Approach:• A DSS for Ventilator-Associated Pneumonia

(VAP): its underlying Bayesian-network and decision-theoretic model

• Clinical setting: advising regarding the diagnosis and treatment of VAP in the ICU

• Design of DSS guided by evaluation considerations

IntroductionIntroduction

• Ventilator-Associated Pneumonia (VAP)

• Nosocomial infection on Intensive Care Units (ICU)

• Gold Standard = infectious-disease specialists

• Decision-Support System in uncertainty helps diagnosing the patient

finding optimal treatment

General: Bayesian Network; 2 parts1. qualitative = structure and relations between nodes (vars)

2. quantitative = conditional (estimated) probabilities example: P(pneumonia | body temperature 38.5 °C)

A DSS for VAP:A DSS for VAP: its underlying Bayesian-network and decision-theoretic model its underlying Bayesian-network and decision-theoretic model (1)(1)

Bayesian Network for VAP; 2 parts1. diagnostic = clinical signs/ symptoms, duration of stay,

mechanical ventilation

2. therapeutic = most effective combination of antibiotic treatment

Decision-theoretic model

Providing utilities for combinations of antibiotics,

taking into account• side effects• financial costs • antimicrobial spectrum

A DSS for VAP:A DSS for VAP: its underlying Bayesian-network and decision-theoretic model its underlying Bayesian-network and decision-theoretic model (2)(2)

utility : a quantitative measure of the strength of the expert’s preferencein decision making

A DSS for VAP:A DSS for VAP: its underlying Bayesian-network and decision-theoretic model its underlying Bayesian-network and decision-theoretic model (3)(3)

Global structure of the Bayesian Network

colonisation

hospitalisation

aspiration

pneumonia

mechanicalventilation

symptomssigns

immunologicalstatus

sideeffects

antimicrobialtherapy

susceptibility

coverage

previousantibiotic use

The problem:The problem: incorporating evaluation into the design of a DSSincorporating evaluation into the design of a DSS

Has the DSS been properly evaluated?

• Diagnostic performance• Usability + effect

Diagnostic performance of the DSSDiagnostic performance of the DSS

AUC 0.795

Dataset: 17.700 rows/ patient days in the ICU

157 VAP days

experts vs.

DSS

Design for an evaluation study Design for an evaluation study of the DSS for VAPof the DSS for VAP

Known systematic effects/ biases in an evaluation study:

1. volunteer effect

2. Hawthorne effect

3. checklist effect

Resulting DSS for VAP Resulting DSS for VAP (1)(1)

Components

Clinical database

User interface

start

Give your professional opinion: VAP (yes/ no) treatment + motivation

Patient’s symptoms arepre-selected and presented to

the user

endManagement advice:1. p(VAP)2. antimicrobial treatment

random

Give your professional opinion: VAP (yes/ no) treatment + motivation

end

Resulting DSS for VAP Resulting DSS for VAP (2)(2)

all users (volunteer/ Hawthorne)

(checklist)

Conclusions & future workConclusions & future work

• Medical Decision-Support Systems are meant to assist

clinicians in the difficult process of medical management• Known biases in evaluation studies were discussed • More attention should be given to evaluation issues:

only then it is possible to perform a reliable evaluation of a DSS

In future we intent to perform the described evaluation study did the clinician revise his/ her judgement, taking into account the

system’s advice? did the system’s advice influence the clinician’s diagnosis?

pneumonia

symptomssigns

mechanicalventilation

colonisation

antibiotic therapy

noyes

Contact:

Stefan Visscher (S.Visscher@azu.nl)