ULJKW 'DQLHO 5XELQon-demand.gputechconf.com › gtc › 2017 › presentation › s... · Z W l l Á Á Á X À r ( } µ v ] } v X } P l } v l } v v z À z î ì í ñ l l> } v P

Copyright © Daniel Rubin 2016 1

DEEP LEARNING IN MEDICALIMAGING: OPPORTUNITIESAND NEW DEVELOPMENTS

Daniel L. RubinDarvin Yi

Associate Professor of Radiology, Biomedical Data Science, of Medicine (Biomedical Informatics), and by courtesy,

of Ophthalmology

Director of Biomedical Informatics, Stanford Cancer Institute

Stanford University

AcknowledgementsPostdoctoral Scholars

– Assaf Hoogi

Funding Support– NCI QIN grants U01CA142555, 1U01CA190214,

1U01CA187947– NVIDIA Academic Hardware Grant Program

– No commercial conflicts

– Xuerong Xiao

– Alfiia Galimzianova

– John Lambert

– Zeshan Hussain

– Jocelyn Barker

Graduate Students Faculty Collaborators

– Sandy Napel

– Chris Beaulieu

– Curt Langlotz

– Christopher Re

Alumni trainees

– Mehmet Ertosun

– Darvin Yi

– Imon Banerjee

Outline

Motivations and potential promises of deep learning in medical imaging

Unique aspects of medical imaging Key challenges and strategies

Outline



Challenge #1: Variation in practice

There are large variations and disparities in care (Institute of Medicine, 2001)

“Errors and variations in interpretation now represent the weakest aspect of clinical imaging*”

*Robinson PJ. Radiology’s Achilles’ heel: error and variation in the interpretation of the Röentgen image. British Journal of Radiology. 1997 Jan 1;70(839):1085–98.

Barlow WE, Chi C, Carney PA, Taplin SH, D'Orsi C, Cutter G, et al. Accuracy of Screening Mammography Interpretation by Characteristics of Radiologists. J. Natl. Cancer Inst. 2004 Jan 15;96(24):1840–50.

Variable Performance of Radiologists


Challenge #2: People (and disease they have) differ…

Biological diversity◦ Heterogeneous genomic

and molecular aspectsof disease

Phenotypic diversity ◦ Variable appearance of

disease in patients nd of lesions on images

Clinical diversity◦ Patients have different

response to treatment

Ideally we would “profile” disease for personalized medicine

Goal of Machine Learning in Medicine

Goal of Machine Learning in Medicine

Fundamental Theorem of Biomedical Informatics:*

*After Charles Friedman

Outline



What we care about in consumer images is usually not subtle“Is there a cat in this image?

NO

YESNO

NO

NO

NO

NO

NONONO

NO

NO

NO

NO

Benign

BenignCancer

Cancer

1) What we care about in medical images can be subtle“Is there a cancer in this image?”


2) Multiple images need to be considered

Multiple imaging modalities Multiple views within a modality Multiple time points

Deep learning methods need to integrate the multiplicity of images, possibly even with non-image data

Multiple imaging modalities

MRI T1 CE MRI DWI

FusionFDG PET

Multiple views within a modality

Multiple time points

Disease is dynamic: it changes over time

The task requires evaluation across many images

Common interpretation task:“Is disease responding to treatment?”

BL

FU1

FU1

FU3

3) Different goals in medical imaging besides object recognition Detection

◦ Is there a lesion in the image, and where is it? Segmentation

◦ Draw an outline around the lesion Classification

◦ What is the diagnosis? Decision making

◦ Response assessment: Is the patient responding to treatment?

◦ Prediction: Will this drug work? Will the disease progress? Will disease recur?

Outline




Key challenges Building successful models with limited

training set size Creating toolkits to do classification or

segmentation within a few hours of first receiving the data with the target labels

Example applications

1. Lesion detection2. Lesion segmentation3. Disease classification4. Visualization of what deep networks

learn5. Toolkit for deep learning




Detection of image abnormalities

AKA “where’s Waldo?”

IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1310-1315, 2015

Detection of breast masses with deep learning

Digital Database for Screening Mammography (DDSM)

2420 mass ROIs 80%/10%/10%

training/test evalu-ation sets

256x256 patches, labeled as “mass” or “non-mass”

Data augmentation: cropping, translation, rotation, flipping and scaling of image tiles

Probability classification map of location (fully connected CNN)

Performance:

Examples

MammogramRescaled

MammogramLocalization

Image

IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1310-1315, 2015





General Fully Connected Networks

http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf

Ophthalmology - Exudates

http://www.glycosmedia.com/wp-content/uploads/2015/01/0010024_hover.jpg http://www.glycosmedia.com/wp-content/uploads/2015/01/0010026_hover.jpg

Ophthalmology - Exudates

Data: e-ophtha http://www.adcis.net/en/Download-Third-Party/E-Ophtha.html

Ophthalmology - Exudates (34 Images)




Ophthalmology - Microaneurysms

https://image.slidesharecdn.com/dmcomplications-110210091023-phpapp02/95/dm-complications-15-728.jpg?cb=1297338033

Ophthalmology - Microaneurysms

Increase Contrast via Adaptive Histogram Equalization per Color Channel

Most Important Color Channel: Green

Ophthalmology - Microaneurysms (76 Images)




Bone Tumors

Stanford Bones Jones Dataset 10,000s of Bone Examples

◦ All Different Anatomical Locations

Ground Truth Segmentations from Radiologist (Chris B.)◦ <1000 Examples

Bone Tumors

Bone Tumors

Bone Tumors




Copyright © Daniel Rubin and Darvin Yi 2017

Mammograms - Data

CBIS-DDSM Dataset Non-Automatic Binary Classification

◦ Malignant◦ Benign

1,081 Patients, 1,992 Images

83% Test Accuracy Radiologist Seed

Pixel Constant Bounding

Box


Mammograms - Network

Dia

gn

osis

CC

MLO

Dia

gn

osis

Mammograms - Results

Urology - Data

Live Endoscope Bladder Confocal Microscopy

~100 Patients ~400 Movies ~100,000 Frames Movie-Level Labels Binary Classification: Benign (73%) vs.

Malignant Achieved Human Level Performance of 85%

Urology - Classes

Urology - Live Classification

Normal

Urology - Live Classification

High Grade


NeRDD - Data Stanford Radiology 1.5+ Million Cases

◦ 400,000 Chest X-Rays◦ 70,000 Head CT’s◦ 50,000 Abdominal CT’s◦ etc...

Full Text Reports 3-ary Labels

◦ 0: Normals◦ 1: Non-normals◦ 2: Emergent Cases

CXR Subset - Data Stanford Radiology 1.5+ Million Cases

◦ 400,000 Chest X-Rays◦ 70,000 Head CT’s◦ 50,000 Abdominal CT’s◦ etc...

Full Text Reports 3-ary Labels

◦ 0: Normals◦ 1: Non-normals◦ 2: Emergent Cases

“Where’s Waldo” Metaphor

“Where’s Waldo” Metaphor

CXR - Abnormality: cardiomegaly, effusion

CXR - Abnormality: left lower lobe opacity


CXR - Abnormality: widened mediastinum

CXR - Simple Preprocessing

Simple Preprocessing to Optimize SNR





Saliency Maps + Deep Dream

Visualizing what deep networks learn

We combine the iterative visualization method of Google's DeepDream algorithm with saliency maps

Generates an image that shows an exaggerated image with features of the queried class.

Can show clinically relevant features, such as “spiculation”

Directed Dream


Glasses vs. No-Glasses



Benign vs. Malignant









FirstAid

Toolkit for deep learning Medical images pose unique challenges

◦ Large size (order of 10 megapixels)◦ Areas of medical interest is often less than 1% of

the total image ◦ Limited data set size (usually < 10,000)

There is usually rich semantic information associated with medical images that could be exploited.

https://github.com/yidarvin/FirstAid

Current Features Tasks:

◦ N-ary Classification◦ N-ary Segmentation

Networks:◦ AlexNet◦ VGG Net (11,13,16,19)◦ Inception Network (v1 and v3)◦ ResNet (151 Layer)


Filepath Philosophy

Filepath Philosophy

Summary Deep learning in medical imaging can help

physicians improve care Unique aspects of medical imaging

◦ What we care about can be subtle◦ Multiple images, modalities, views, time points◦ Different goals besides object recognition

Important challenges◦ Limited training set size◦ Need toolkits for rapid development

Many emerging applications◦ Lesion detection, segmentation, classification

Thank you.

Contact info:[email protected]

Documents

ULJKW 'DQLHO 5XELQon-demand.gputechconf.com › gtc › 2017 › presentation › s... · Z W l l Á Á Á X À r ( } µ v ] } v X } P l } v l } v v z À z î ì í ñ l l> } v P