EFFICIENT EXPLORATION OF REGION HIERARCHIES FOR SEMANTIC

SEGMENTATION

Míriam Bellver Bueno Xavier Giró i Nieto Carles Ventura

1

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

2

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

3

Motivation

Recognition Tasks

Object Detection

Content-based Image Retrieval

Medical Imaging

4

Motivation

Recognition Tasks

Object Detection

Content-based Image Retrieval

Medical Imaging

5

Image Segmentation

MotivationSemantic Segmentation

Segmentation Prediction

ImageObject Candidates Final segmentation

6

MotivationSemantic Segmentation

Segmentation Prediction

ImageObject Candidates Final segmentation

7

MotivationEfficient Semantic Segmentation

Segmentation Prediction

Only a few regions

The minimum computational time for the calculation of each region

ImageObject Candidates Final segmentation

8

MotivationEfficient Semantic Segmentation

Segmentation Prediction

Only a few regions

The minimum computational time for the calculation of each region

ImageObject Candidates Final segmentation

But what regions??

9

MotivationEfficient Semantic Segmentation

Segmentation Prediction

ImageObject Candidates

Semantic Segmentation

UCM hierarchical partitions

CPMC object candidates

MULTI-SCALE INFORMATION

10Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts.

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

11

Related Work

12

Object Detection and Recognition

Sliding Windows

Partition

e.g. Viola Jones

Hierarchical

Flat

e.g UCM

e.g CPMC

e.g Watershed Partition

Object Proposals

Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.

Ultrametric Contour Map (UCM)

13

Original Image Ultrametric Contour Map Dendrogram

Arbelaez, P. (2006, June). Boundary extraction in natural images using ultrametric contour maps

root node

leavescosts

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

14

Pipeline: Database

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

15

Pipeline: Object Candidates

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

CPMC UCM

16

Pipeline: Features

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

SIFT-based features [O2P]

[O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012).17

Pipeline: Assessment

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

18

AssessmentAverage Accuracy per Category (AAC) / Intersection over Union (IoU)

19

Pipeline

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

CPMC UCMSIFT-based features

[O2P]

[O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012) 20

Object candidates

Class-agnostic exploration

Class-dependent exploration

Comparison

Based on a ranked list (CPMC) Based on a partition tree (UCM)

Contributions

21

Class-agnostic tree exploration

22

COSTS

INDEXES1

2

Class-agnostic tree exploration

23

Class-agnostic tree exploration: Indexes

24

521

525

easier to generate than its sibling…

more homogeneous

Merging Sequence527

3

7

6

11

Ranked List of Object Candidates

7 8 5 3 10

1 2

Max Queue

7 8 5

4 5

10

8 9

9

3

1

1

6 4 2 11

Class-agnostic tree exploration: Indexes

25

Class-agnostic tree exploration: Indexes

26

521

525

527

Class-agnostic tree exploration: Costs

27

521525

527

511

495

MAX

MIN

SECOND DERIVATIVE

0.1

0.3

0.7

Class-agnostic tree exploration

28

Class-dependent tree exploration

Class-agnostic tree exploration

...

OBJECTS

Class-dependent tree exploration

table

chair

plane

sofa

0.15

0.05

0.60

0.05

table

chair

plane

sofa

0.05

0.00

0.05

0.05

cow 0.01

...

ANALYSED IN QUEUE

29

confidence values confidence values

cow 0.01

...

Class-dependent tree exploration

table

chair

plane

sofa

0.05

0.05

0.70

0.00

table

chair

plane

sofa

0.05

0.00

0.05

0.05

ANALYSED IN QUEUE

30

confidence values confidence values

cow 0.01

...

cow 0.01

...

Class-dependent tree exploration

table

chair

plane

sofa

0.05

0.05

0.65

0.00

table

chair

plane

sofa

0.05

0.00

0.35

0.05

IN QUEUE IN QUEUE

31

confidence values confidence values

cow 0.01

...

cow 0.01

...

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

32

Results: SIFT-based features [O2P]

33

UCM Class-agnostic explorationUCM Class-dependent explorationCPMC

Pipeline

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC (IoU)

Ground Truth

Train

CPMC UCM

Deep learning features [SDS]

SIFT-based features [O2P]

[SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 34

Results: Deep Learning feat. (SDS)

35

UCM Class-agnostic explorationUCM Class-dependent explorationCPMC

Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping Kuo, W., Hariharan, B., & Malik, J. (2015). DeepBox: Learning Objectness with Convolutional Networks

CPMC ≈ x 8 UCM Computation Time

Outline

● Motivation● Related Work● Methodology● Results● Conclusions and Future Work

36

Conclusions1. Better results when using a few regions of UCM compared to

CPMC.

37

CPMCUCM

Conclusions2. The class-dependent exploration of UCM regions is the best configuration for a budget of a few regions.

38

Class-dependent tree exploration

...

Conclusions3. SDS descriptors extracted from a CNN obtain better results than O2P.

39

Deep learning features [SDS]hand-crafted features [O2P]

Future Work● Class-dependent tree exploration using two classifiers

● Compare performance using different object candidates, such as MCG.

40

Is there a face on this node?

Is this a face?

1

2

X. Giró, 2012, Part-based object retrieval with binary partition trees.Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014, June). Multiscale combinatorial grouping

41

42

Related Work

43

Object Detection and Recognition

Sliding Windows

Segmentation

e.g. Viola Jones

Hierarchical Segmentation

Flat segmentation

e.g UCM

Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.

e.g CPMC

Related Work: Tree explorationRegions generated from a hierarchical partition taking advantage of its multi-scale information in order to guide an efficient exploration throughout the

tree.

X. Giró, 2012, Part-based object retrieval with binary partition trees.

X. Giró, 2012, Part-based object retrieval with binary partition trees. 44

Constrained Parametric Min-Cuts (CPMC)

Carreira, J., & Sminchisescu, C. (2012). Cpmc: Automatic object segmentation using constrained parametric min-cuts. 45

Motivation

Local Feature Descriptors: SIFT HOG

Learned Descriptors

From hand-crafted to learned features

~1995 to ~2005 ~2005 to ~2010 ~2010 to ~2015

Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]

hand-crafted descriptors

46

Features: SDS features

[SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 47

Features: Second Order Pooling (O2P)

Average Pooling Max Pooling

2nd orderSIFT

O2PSIFT Second Order SIFT Pooling

Carreira, J., Caseiro, R., Batista, J., & Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. 48

Results: Deep Learning feat. (O2P)

49

50

Class-agnostic tree exploration: CostsBig difference of

cost between node and its father

Small difference of cost between node

and one of its children

51

SECOND DERIVATIVE

Class-agnostic tree exploration

Second derivatives costs

associated to consecutive nodes yield good results

52

Class-agnostic tree exploration: CostsBig difference of

cost between node and its father

Small difference of cost between node

and one of its children

53

54

Class-agnostic tree exploration: Indexes

Objects can be found in regions associated to indexes that differ from the indexes of their adjacent regions

150 149

147

costeasier to generate than its sibling…

more homogeneous

indexes

55

521

Database

56

Class-agnostic tree exploration

Contours of UCM

Merging Sequence

INDEXES of the merging sequence

COSTS values of the contours

Input image

Based on the structure of the UCM partition, defined by these two files:

1

2

57

58

Class-dependent tree explorationGuide a top-down efficient exploration throughout the tree based on the

classifier’s decision.

X. Giró, 2012, Part-based object retrieval with binary partition trees.

Motivation

59

60

61

62

Related Work

63

Object Detection and Recognition

Sliding Windows

Segmentation

e.g. Viola Jones

Hierarchical Segmentation

Flat segmentation

e.g UCM

Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.

e.g CPMC

Related Work

64

Object Detection and Recognition

Sliding Windows

Segmentation

e.g. Viola Jones

Hierarchical Segmentation

Flat segmentation

e.g UCM

Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features.

e.g CPMC

Pipeline: Features

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

Deep learning features [SDS]

SIFT-based features [O2P]

[O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012)[SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015].

65

Pipeline

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC (IoU)

Ground Truth

Train

CPMC UCM

Deep learning features [SDS]

SIFT-based features [O2P]

[O2P] Carreira, Caseiro, Batista, Sminchisescu, “Semantic Segmentation with Second-Order Pooling” (ECCV 2012)[SDS] Hariharan, Arbeláze, Girshick, Malik, “Simulatenous Detection and Segmentation” (ECCV 2014) - More details on [Eduard Fontdevila BSc 2015]. 66

67

Motivation

Recognition Tasks

Object Detection

Content-based Image Retrieval

Medical Imaging

68

Motivation

Recognition Tasks

Object Detection

Content-based Image Retrieval

Medical Imaging

69

Image Segmentation

MotivationGoal: Guide a top-down exploration of a hierarchical partition by answering the following question:

● Does this region contain the object we are seeking?

● If so, does this region represent the object we are seeking?

70

Motivation

71

Motivation

Recognition Tasks

Object Detection

72

73

Acknowledgments

Technical support

Albert Gil Josep Pujal

74