Colon Gland Segmentation with

Deep Convolutional Neural Networks and

Total Variation Segmentation

Philipp Kainz1,2, Michael Pfeiffer2, and Martin Urschler3,4

1 Institute of Biophysics, Medical University of Graz, Graz, Austria 2 Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland 3 Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria 4 Ludwig Boltzmann Institute for Clinical Forensic Imaging, Graz, Austria

philipp.kainz@medunigraz.at

Oct. 5, 2015 Page 1

Team vision4GlaS

Colon Gland Segmentation Algorithm Overview

Oct. 5, 2015 Page 2 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

Color Deconvolution Hematoxylin-Eosin

Contrast Limited Adaptive

Histogram Equalization CLAHE

Preprocessing

Combine Model

Predictions Total Variation Segmentation (TVseg)

Active Contour Energy Minimization Postprocessing

Learn

Glandular Objects OBJECT NET

Learn

Separator Structures SEPARATOR NET

Learning & Prediction Pixel

Probabilities

Image Preprocessing: Tissue Decomposition

Oct. 5, 2015 Page 3 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

Color Deconvolution [Ruifrok and Johnston, 2001]

source image (RGB)

Contrast Limited Adaptive

Histogram Equalization [Zuiderveld, 1994]

red channel, CLAHE

deconvolved image (RGB)

red channel

green channel

blue channel

Learning Glandular Objects

C0: background benign

C1: gland benign

C2: background malignant

C3: gland malignant

4-Class Classification

OBJECT NET predicts the label of the

center pixel in a

101x101 px patch

Deep Convolutional Neural Network [LeCun et al., 2010]

C3: 128@16x16

S1: 80@46x46

C2: 96@40x40

S2: 96@20x20 Input: 101x101

C1: 80@91x91

S3: 128@8x8

C4: 160@6x6

max-pooling

2x2

max-pooling

2x2 convolutions

11x11

convolutions

7x7

convolutions

3x3

max-pooling

2x2 convolutions

5x5

Output: 4

F6: 512

F5: 1024

full connection

Oct. 5, 2015 Page 4 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

Predicting Glandular Objects

Oct. 5, 2015 Page 5 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

background benign gland benign

background malignant gland malignant annotation image

input image

Learning Separator Structures

Oct. 5, 2015 Page 6 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

Binary Classification

SEPARATOR NET predicts the label of the

center pixel in a

101x101 px patch

max-pooling

2x2

Output: 2

F6: 512

F5: 1024

full connection

C3: 128@17x17

S1: 64@47x47

C2: 96@41x41

S2: 96@21x21 Input: 101x101

C1: 64@93x93

S3: 128@9x9

C4: 160@7x7

max-pooling

2x2 convolutions

9x9

convolutions

7x7

convolutions

3x3

max-pooling

2x2 convolutions

5x5

Predicting Separator Structures

Oct. 5, 2015 Page 7 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

separator structures annotation image

input image non-separator structures

Combining Model Predictions

Oct. 5, 2015 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al. Page 8

background benign

gland benign

background malignant

gland malignant

gland

background

separator

structures

background probability map

foreground probability map

Total Variation Segmentation

Based on a convex geodesic active contour model, a figure-ground

segmentation u is computed:

Oct. 5, 2015 Page 9 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al.

input I(x)

segmentation u

Globally optimal solution giving minimal contour length foreground probability map

background probability map

Segmentation Results: Test Set A (off-site)

Oct. 5, 2015 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al. Page 10

Segmentation Results: Test Set A (off-site)

Oct. 5, 2015 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al. Page 11

Thanks for your attention!

Philipp Kainz1,2, Michael Pfeiffer2, and Martin Urschler3,4

1 Institute of Biophysics, Medical University of Graz, Graz, Austria 2 Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland 3 Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria 4 Ludwig Boltzmann Institute for Clinical Forensic Imaging, Graz, Austria

philipp.kainz@medunigraz.at

Oct. 5, 2015 Page 12

Team vision4GlaS

References

[Ruifrok and Johnston, 2001]

A. C. Ruifrok and D. A. Johnston. Quantification of histochemical staining by color deconvolution. Anal Quant

Cytol Histol, 23(4):291–299, August 2001.

[Zuiderveld, 1994]

K. Zuiderveld. Contrast limited adaptive histogram equalization. In Graphics gems IV, pages 474–485.

Academic Press Professional, Inc., 1994.

[LeCun et al., 2010]

Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In ISCAS, pages

253–256, May 2010.

[Hammernik et al., 2015]

K. Hammernik, T. Ebner, D. Stern, M. Urschler, and T. Pock. Vertebrae segmentation in 3D CT images based

on a variational framework. In Recent Advances in Computational Methods and Clinical Applications for Spine

Imaging, pages 227–233. Springer, 2015.

[Bresson et al., 2007]

X. Bresson, S. Esedoglu, P. Vandergheynst, J.-P. Thiran, and S. Osher. Fast global minimization of the active

contour/snake model. J Math Imaging Vis, 28(2):151–167, 2007.

[Chambolle and Pock, 2011]

A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging.

J Math Imaging Vis, 40(1):120–145, 2011.

Oct. 5, 2015 Colon Gland Segmentation with Deep Convolutional Neural Networks and Total Variation Segmentation, P. Kainz et al. Page 13