Contour Detection and Hierarchical Image Segmentation

– Some Experiments

CS395T – Visual Recognition

Presented by Elad Liebman

Overview

• Understanding the method better: – The importance of thresholding the ouput – Understanding the inputs better – Understanding the UCM

• Pushing for boundaries: – Types of difficult inputs – Difficult input examples

Warm-up: choosing the right threshold

𝑘 = 0.4

Warm-up: choosing the right threshold

𝑘 = 0.1

Warm-up: choosing the right threshold

𝑘 = 0.8

So what goes into the OWT?

• From Contours to Regions: An Empirical Evaluation. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. CVPR 2009.

• Contour Detection and Hierarchical Image Segmentation. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2011

The Features

• Difference in feature channels on the two halves of a disc of radius 𝜎 and orientation 𝜃.

• Feature channels in our case: – Color gradients – Brightness gradients – Texture gradients

• Comparison between the two disc halves using 𝜒2 distance.

The basic signals

• Color/brightness channels based on Lab color space.

• Repeatedly generated at different scales (different 𝜎 radii values - 𝜎

2,𝜎, 2𝜎).

• All in all we get 13 different inputs.

Illustration n. 1: Color gradient 𝑎 (red-green scale)

𝜃 =𝜋2

𝜃 = −𝜋4

𝜃 =

𝜋2

max𝜃

{𝑐𝑐𝑎}

Illustration n. 2: Color gradient 𝑎, different 𝜎 value

max𝜃

{𝑐𝑐𝑎}

𝜃 = 0

𝜃 =𝜋4

𝜃 = −𝜋4

Illustration 3: Brightness gradient (light-dark scale)

max𝜃

{𝑐𝑐𝑎}

𝜃 =𝜋2

𝜃 = −𝜋4

𝜃 =

𝜋2

Illustration 4: Texton Map

Multiscale 𝑃𝑃 and Spectral/Global 𝑃𝑃 • The features are linearly combined to create a

unified signal, the multiscale Pb input. • They are then “globalized” spectrally. • An affinity matrix 𝑤 is constructed with 𝑊𝑖𝑖

being the maximal value of 𝑚𝑃𝑃 along the line connecting 𝑖 and 𝑗.

• Generalized eigenvectors are then extracted and combined (after processing) with the local 𝑃𝑃 data to create the final input for the OWT.

𝑚𝑃𝑃 vs. 𝑐𝑃𝑃

(In practice we sample 8 orientations…)

𝑐𝑃𝑃 = max𝜃

{ }

𝜃 = [0,𝜋8

,𝜋4

,3𝜋8

, … ,7𝜋8

]

(In practice we sample 8 orientations…)

𝑐𝑃𝑃 =

Playing with the orientations a bit…

Playing with the orientations a bit…

Using less than 8 orientations

2 orientations

Using less than 8 orientations

4 orientations (in this case it’s enough)

The Ultrametric Contour Map

• The basic idea is to organize and merge sub-regions hierarchically and iteratively.

• At each point we merge two regions with the weakest boundary between them.

• The result is a dendrogram of nested regions.

Example 1

Example 2

Potentially Difficult Input

• We would like to test the OWT-UCM method against problematic input.

• See where it might break.

warming up - blurring

Potentially Difficult Input II

• Blurring isn’t very interesting – doesn’t reflect the kind of challenges the algorithm is likely to face.

• What realistic problems exist? – Difficult patches – Complex, contour-rich images – Not enough color/texture information

Difficult Input – abstract art

𝑘 = 0.4

Difficult Input – abstract art

𝑘 = 0.1

Difficult(er) Input – abstract(er?) art

𝑘 = 0.4

Difficult(er) Input – abstract(er?) art

𝑘 = 0.1

Another Difficult Example • Occlusion • Similarity of objects in image:

– Similar textures – Similar colors – Contour lines blend

Another Difficult Example

𝑘 = 0.4

Another Difficult Example

𝑘 = 0.1

Another Difficult Example

𝑘 = 0.2

Sneaking a look under the hood

• Possibly we need to weight elements differently in certain cases…

CGA1 CGB1

Textons

BG1

Summary • Studied some of the technical aspects of the

method: – Threshold selection – Understanding the input data – Illustration of UCM in action

• Tested the method against difficult input: – Problematic contours – Complex images – Cases where features aren’t informative enough – Many similar items occluding one another

References • From Contours to Regions: An Empirical Evaluation. P.

Arbelaez, M. Maire, C. Fowlkes, and J. Malik. CVPR 2009. • Contour Detection and Hierarchical Image Segmentation.

P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2011

• Using Contours to Detect and Localize Junctions in Natural Images. M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik. CVPR 2008.

• What are Textons? S. Zhu, C. Guo, Y. Want Z. Xu, International Journal of Computer Vision 2005.

• Class notes for 16-721: Learning-Based Methods in Vision, taught by Prof. Alexei Efros, CMU.

• Class notes for CS 143: Introduction to Computer Vision, taught by Prof. James Hayes, Brown.