The goals of segmentation Segments as primitives for...

1

1Many slides are curtesy of S. Lazebnik , B. Girod and S. Seitz. Also thanx to Eric Weizmann, Zvi Solomon, Seri khoury , Jad Silbak, Avner Gidron, David Cohen

Segmentation

• Segmentation is the process of dividing an

image into “coherent regions”.

The goals of segmentation• The goal of segmentation is to simplify and/or change the

representation of an image into components that are more meaningful and easier to analyze.

• Image segmentation is the process of assigning a label to every pixel such that same-labeled pixels share certain characteristics.

“superpixels”

• Segmentation can be treated at different levels of details.

Object segments

Segments as primitives for recognition

J. Tighe and S. Lazebnik, ECCV 2010

2

• Interactive segmentation for graphics Approaches to segmentation

• Gray-level Histogram thresholding

• Segmentation as clustering

• Segmentation as graph partitioning

Global Thresholding –Segmentation into two components

oldV

F(V)

255

255

Threshold value

newV

T

0 100 2000

500

1000

1500

Histogram

Segmentation using Thresholding

50 75

Original

Threshold = 50 Threshold = 75

3

Segmentation using Thresholding

• How to choose the optimal threshold?

• Can a global threshold suffice?

Slide by B. Girod

Error Probability for Thresholding

• Assuming there are two components in the image:

– Foreground

– Background

• What is the probability of error given a threshold T?

T

Slide by B. Girod

Error Probability for Thresholding

• The optimal threshold T with respect to MAP estimation (Maximum A Posteriori) is T where � � � � ���|��

• Goal: Find the optimal T given an image.

T

Slide by B. Girod

Optimal Thresholding using Otsu’s Method.

Slide by B. Girod

4

Locally Adaptive Thresholding

• Slide a window over the image

• For each window decide whether to apply thresholding:

– Thresholding should not be performed in uniform windows.

– Can use variance or other criterion for “uniformity”.

• In non-uniform windows: apply Otsu’s thresholding based on local histogram.

• In uniform windows: classify the entire window as Foreground or Background based on mean value.

Slide by B. Girod

Locally Adaptive Thresholding

Slide by B. Girod

Edge Based Segmentation Edge Based Segmentation

We are looking for boundaries, not edges.

Problem with edges in presence of noise and

sampling artifacts (e.g. medical images).

5

Snakes – Active Contours

1. The snake is placed near the image contour of interest.

2. During an iterative process, the snake is attracted towards the target contour by various forces that control the shape and location of the snake within the image.

Snakes: Active Contour Models (1987)Michael Kass, Andrew Witkin, and Demetri Terzopoulos

Snakes

Works with noisy and discontinuous boundary edges

Snakes

• Representation of the contours

• Defining the energy functions

– Internal

– External

• Minimizing the energy function

(x0,y0, x1,y1,…., xn,yn)

Snakes

Usually, the total energy(cost function) of snake is

a combination of internal and external energies

exin EEE +=

Internal energy encouragessmoothness of shape

External energy encouragesalignment of curve withimage structures (edges).

(x0,y0, x1,y1,…., xn,yn)Minimize E over all possible locations of

6

External Energy

• Measure how well the curve matches the image data.

• “Attract” the curve toward different image features (edges, lines, etc)

• Think of external energy from image as gravitational pull towards areas of high contrast.

External Energy

For every image pixel v=(x,y) define its External Energy as:

22 |),(||)(|)( yxIIEex ∇−=∇−= vv

∫=1

0

))(( dssEE exex ν continuous case

]}1,0[s|)s({ ∈= νC

External energy term for the snake whose points are s :

}ni0|{ i <≤= νC

discrete case

External Energy

}ni0|{ i <≤= νC

discrete case

Snake moves to decrease external energy:

Internal Energy

The Internal Energy (= Smoothess) at contour point v(s) is defined as:

sd

d

ds

dsssEin 2

2

)()())((

22

ννβαν +=

Elasticity/stretching Stiffness/bending

The Internal Energy (smoothness) of the whole snake is :

∫=1

0

inin ds))s((EE ν ]}1,0[s|)s({ ∈= νC

7

Internal Energy

5v

4v3v

2v

1v6v

7v

8v

10v

9v

elastic energy(elasticity)

i1ivds

dν

ν−≈ +

bending energy(stiffness)

1ii1i1iii1i2

2

2)()(ds

d−+−+ ν+ν−ν=ν−ν−ν−ν≈

ν

Internal Energy

Elasticity Stiffness

∑−

=−++ +−+−=

1

0

2

11

2

1 |2|||n

i

iiiiiinE νννβννα

large α

small α large β small β

Snakes

Minimize exin EEE +=

(v0, v1,…., vn)over all possible locations of snake

vi = (xi,yi)

Gradient descent, iterative local minimization,Veterbi (dynamic programming)

Snakes

8

Tracking Using Snakes

(Original Terzopolous video)

Intelligent Scissors

Eric N. Mortensen and William A. Barrett, "Intelligent Scissors for Image Compostion", Proc. SIGGRAPH 95, 1995, 191-198.

Finds a path from seed to mouse that follows object

boundary as closely as possible?

Intelligent Scissors

Basic Idea

• Define edge score for each pixel

– edge pixels have very low cost

• Find lowest cost path from seed to mouse

seed

mouse

Uses Dijkstra’s shortest path algorithm

Intelligent Scissors

http://fotoflexer.com/app/

Demo

9

Color Segmentation:

Segmentation as clustering

Source: K. Grauman

K-means

• How to choose the representative colors?

– This is a clustering problem!

Objective

• Each point should be as close as possible to a cluster center

• Minimize sum squared distance of each point to closest center

This image

cannot

This image cannot

This

image cannot curren

R

G

R

G

Slide by Steve Seitz

This image cannot

This image cannot curren

This image cannot curren

Break it down into sub problems

• Suppose I tell you the cluster centers ci

– Q: how to determine which points to associate with each ci?

− A: for each point p, choose closest ci

Suppose I tell you the points in each cluster

• Q: how to determine the cluster centers?

• A: choose ci to be the mean of all points in the cluster

Slide by Steve Seitz

K-means clustering: algorithm

1. Randomly initialize the cluster centers, c1, ..., cK

2. Given cluster centers, determine points in each cluster

• For each point p, find the closest ci. Put p into cluster i

3. Given points in each cluster, solve for ci

• Set ci to be the mean of points in cluster i

4. If ci have changed, repeat Step 2

Text by Steve Seitz

10

K-Means - Example

י אריק ויצמן"נכתב ע

Choose K seed positions randomlyי אריק ויצמן"נכתב ע

K-Means - Example

K = 2

י אריק ויצמן"נכתב ע

K-Means - Example

Associate elements to nearest seed.י אריק ויצמן"נכתב ע

K-Means - Example

Update seed location to COM of all associated elements.

11

י אריק ויצמן"נכתב ע

K-Means - Example

Repeatי אריק ויצמן"נכתב ע

K-Means - Example

י אריק ויצמן"נכתב ע

K-Means - Example

י אריק ויצמן"נכתב ע

K-Means - Example

12

י אריק ויצמן"נכתב ע

K-Means - Example

Repeat until no change in element assignments to seeds.

K-Means – Image Segmentation

י אריק ויצמן"נכתב ע

Input Image

WhiteMatter

GrayMatter

Background CSF

K-Means – Image Segmentation

י אריק ויצמן"נכתב ע

Iteration:

126.6209

164.4521

209.5154 229.4411

1

Seed Locations: 13.2753

168.9954

204.1537 247.5998

2

2.5485

146.8334

204.1537 247.5998

3

1.4523

131.0522

197.6932 247.5998

4

1.0536

122.6038

195.0548 247.5998

5

1.0536

116.3612

192.7618 247.5998

6

0.7324

109.3117

190.8013 247.5998

7

Image Intensity-based clusters Color-based clusters

Segmentation as clustering

• K-means clustering based on intensity or color is essentially vector quantization of the image attributes

Slide by Svetlana Lazebnik

13

Segmentation as clustering

• K-means clustering based on texture

Segmentation as clustering

Clusters don’t have to be spatially coherent

Segmentation as clustering

Source: K. Grauman

Segmentation as clustering

• Clustering based on (r,g,b,x,y) values enforces more spatial coherence

Slide by Svetlana Lazebnik

14

Over segmentation – Super pixels using K-means

• SLIC (Simple Linear Iterative Clustering) super pixel based on k-means clustering (r,g,b,x,y)

• Initial centers – regular kxk windows.

• Spatial domain and color domain can be weighted differently.

• Iterate k-means until convergence.

• Taking the role of pixels in many applications.

Achanta, et. Al (2010).

Slic superpixels (EPFL-REPORT-149300).

K-Means for segmentation

• Pros

– Very simple method

– Converges to a local minimum of the error function

• Cons

– Memory-intensive

– Need to pick K

– Sensitive to initialization

– Sensitive to outliers

– Only finds “spherical” clusters

Slide by Svetlana Lazebnik

Mean shift segmentation

• The mean shift algorithm seeks modes or local

maxima of density in the feature space

imageFeature space

(L*u*v* color values)

Non-parametric density mode estimation

56

Non-parametric

Density Estimation

Non-parametric

MODE Estimation

(Mean Shift)

Data

Discrete PDF Representation

PDF Analysis

• Non-parametric – no assumption about PDF form (e.g. normal distribution)

• Density Gradient is estimated instead of Density itself.

15

Search

window

Center of

mass

Mean Shift

vector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Search

window

Center of

mass

Mean Shift

vector

Slide by Y. Ukrainitz & B. Sarel

Mean shift

Search

window

Center of

mass

Mean Shift

vector

Slide by Y. Ukrainitz & B. Sarel

Mean shiftSearch

window

Center of

mass

Mean Shift

vector

Slide by Y. Ukrainitz & B. Sarel

Mean shift

16

Search

window

Center of

mass

Mean Shift

vector

Slide by Y. Ukrainitz & B. Sarel

Mean shiftSearch

window

Center of

mass

Mean Shift

vector

Slide by Y. Ukrainitz & B. Sarel

Mean shift

Search

window

Center of

mass

Slide by Y. Ukrainitz & B. Sarel

Mean shift Mean shift clustering

• Cluster: all data points in the attraction basin

of a mode

• Attraction basin: the region for which all

trajectories lead to the same mode

Slide by Y. Ukrainitz & B. Sarel

17

Mean shift clustering/segmentation

• Find features (color, gradients, texture, etc)

• Initialize windows at individual feature points

• Perform mean shift for each window until convergence

• Merge windows that end up near the same “peak” or mode

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Mean shift segmentation results

Mean shift segmentation results Mean shift segmentation results

18

Mean shift

• Pros:– Does not assume shape on clusters

– One parameter choice (window size)

– Generic technique

– Find multiple modes

• Cons:– Selection of window size

– Does not scale well with dimension of feature space

Kristen Grauman