Clinical-grade Computational...

Clinical-grade Computational Pathology:

Hype or Hope for Cancer Care

ECP 2019, Nice

Thomas J. FuchsMemorial Sloan Kettering Cancer Center

Weill Cornell Graduate School of Medical Sciences

Computational Pathology and Medical Machine Learning Lab

Department of Pathology

fuchst@mskcc.org thomasfuchslab.org

Paige.AI@ThomasFuchsAI

Fuchs Lab @ MSKCC + Weill Cornell

2

MSKCC is the largest

and oldest private cancer center in the world

The Warren Alpert Center for Digital and

Computational Pathology at MSK est 2017

Disclosure:TF is Founder and CSO of Paige.AI

Pathology Today

Pathology workload for one week at Memorial

Sloan Kettering

1,000,000 new glass slides per year @ MSKCC

1,000,000 new glass slides per year @ MSKCC

1,000,000 new glass slides per year @ MSKCC

15

0 0

00

pixel

25

CT

MRI

Lab

Sono

Derm

…

CTC

TissuePathologySurgical PathologyHematopathologyDermatopathology

Molecular Pathology

Diagnosis

Testing

Sequencing

Pharma

Studies

Insurance

…

Follow-up

Screening& Detection

Treatment& Research

Clinical Workflow

The whole edifice of medicine rests on the pathologist’s diagnosis

Pathologist under stress

Copyright © 2018 PAIGE.ai, Inc.

All rights reserved

Pathology is headed for a crisis

Almost half of cancers are “rare cancers”Source: World Health OrganizationCredit: Michaeleen Doucleff/NPR

Source: World Health Organization

0

2

4

6

US WesternEurope

Americas(non US)

Asia

Pathologists per 100,000 People

Source: Metter et al, JAMA 2019

Source: Metter et al, JAMA 2019

Copyright © 2018 PAIGE.ai, Inc.

All rights reserved

Pathologist quality decreases with workload

The percentage of pathologists experiencing adverse events increases significantly with a workload greater than 39 hours per week.

Source: Australian Government Department of Health, “Impact of Workload of anatomic Pathologists on Quality and Safety” 2011

Computational Pathology

Computer Vis ion Tasks in Pathology

Nuclei Detection and ClassificationSub-cellular level

Segmentation

Structure EstimationMorphology

CIFAR-10(32*32)*60K= 61.44 million pixels

1 Whole Slide= 100,000 x 60,000 = 6 billion pixels

All 60,000 CIFAR images fit into this box

Dataset Sizes: Computer Vision vs. Computational Pathology

All of ImageNet 482 x 415 * 14,197,122 = 2.8 trillion pixels

n=1

n=474

474 Whole Slides100,000 x 60,000 *474= 2.8 trillion pixels

Dataset Sizes: Computer Vision vs. Computational Pathology

Why is Computational Pathologyso chal lenging?

Diagnosis

Fuchs Lab Projects 2019

Diagnosis Prognosis

Fuchs Lab Projects 2019

Diagnosis Prognosis

Large-Scale Machine Learning Framework

Fuchs Lab Projects 2019

The State-of-the-Art

Dataset sizes in Computational Pathology over time

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019

First Computational Pathology Paper

[Fuchs et al. 2008]

1 slide (Tissue Microarray)

1

Google

[Liu et al. 2017]

509 slides

Camelyon

Challenge

400 slidesGLASS

challenge

200 slides

20200

400 500

Dataset sizes in published articles over time

Equivalent to

State-of-the-art vs. Reality in clinical practice

State-of-the art datasets in pathology:• tiny (~400 slides)• very well curated

Like training your autonomous car only on an empty parking lot.It has never seen rain, snow or a dirt road.

State-of-the-art vs. Reality in clinical practice

State-of-the art datasets in pathology:• tiny (~400 slides)• very well curated

Like training your autonomous car only on an empty parking lot.It has never seen rain, snow or a dirt road.

Clinical reality:• messy• diverse• surprising

How can we ever hope to

train clinical-grade models?

What would it take to go

beyond the State-of-the-Art

Machine Learning:New ways to learn from data at petabyte scale

Data:Real-world, clinically relevant datasets at scale

Domain Experts:Pathologists and computer scientist working in tandem

Computation:HPC for efficient deep learning at scale

What does it take to go beyond:

0

50000

100000

150000

200000

250000

Clinical Slide Scanning @ Memorial Sloan Kettering

Nu

mb

er o

f D

igit

ized

Wh

ole

Slid

es

2015 2016 2017

Nu

mb

er o

f D

igit

ized

Wh

ole

Slid

es

2015 2016 2017

0

200000

400000

600000

800000

1000000

1200000

2018

Clinical Slide Scanning @ Memorial Sloan Kettering

Projection with current ramp-up to40,000 slides / month

~ 1 petabyte of compressed image data

Dataset sizes in Computational Pathology over time

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019

First Computational Pathology Paper

[Fuchs et al. 2008]

1 slide (Tissue Microarray)

1

Google

[Liu et al. 2017]

509 slides

Camelyon

Challenge

400 slidesGLASS

challenge

200 slides

20200

400 500

Dataset sizes in published articles over time

Dataset sizes in Computational Pathology over time

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019

First Computational Pathology Paper

[Fuchs et al. 2008]

1 slide (Tissue Microarray)

1

Google

[Liu et al. 2017]

509 slides

Camelyon

Challenge

400 slidesGLASS

challenge

200 slides

20200

400 500

Dataset sizes in published articles over time

AperioScanner

cBioPortal

ConsultationPortal

...

AperioViewer

HamamatsuViewer

PhilipsViewer

cBio PortalViewer

ConsultationViewer

...

PhilipsScanner

HamamatsuScanner

ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....

AperioScanner

cBioPortal

ConsultationPortal

... PhilipsScanner

HamamatsuScanner

slides.mskcc.org

H i g h P e r f o r m a n c e C o m p u t i n g f o r P a t h o l o g y

Awarded “Center of Excellence for GPU Computing” fromfor our work in Pathology and csBio.

320 GPUs in totalPascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology

MSKCC’s HPC Cluster

Deep Learning Cluster for Computational Pathology @MSKCC

Deep Learning at Scale

DGX-1 Cluster for Computational Pathology

Beyond Manual Image Annotation

56

Strongly Supervised LearningPixel-level Annotation

“Classical” supervised model.

Weakly Supervised LearningImage-level Annotation

Multiple Instance learningDictionary Learning, etc.

Binary label for the whole slide from the pathology report.

0 | 1

57

Cancerous lesions can be tinyProstate Cancer Biopsies

Multiple Instance Learning

Multiple Instance Learning vs. Fully Supervised Learning

Global Test Set> 800 Institutions

Paige Prostate Cancer System

Potential Impact on Clinical Practice: Detection / Quantification

Potential Impact on Clinical Practice: Prioritization / Triaging

Are we done yet?

One AI to rule them all?

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

Frozen H&E IHC Fluorescent Single Cell FNA

Bone & Soft Tissue

Breast

Dermatopathology

Gastrointestinal

Genitourinary

Gynecologic

Head and Neck

Neuropathology

Thoracic

Hematopathology

Cytology

MSKCC & Paige Teams

MSKCC Col laborators

David Klimstra Meera Hameed Victor Reuter Malcolm Pike

Joe Sirintrapun Hikmat Al-Ahmadie Edi Brogi Jinru Shia

Klaus Busam

Oscar Lin

Jung Hun Oh HariniVeeraraghavan

Adity Apte John L. HummJoseph O. Deasy

Thank you for your attention!

Questions welcomed!

Thomas J. Fuchs

fuchst@mskcc.org

thomasfuchslab.org

Open ML and CS positions in Manhattan

@ThomasFuchsAI