LECTURE 08 : SEGMENTATION

Venu Madhav GovinduDepartment of Electrical EngineeringIndian Institute of Science, Bengaluru

In this lecture we shall look at image segmentation

Segmentation

Image Segmentation

• Find groups of pixels (meaningful)

• What the basis of this grouping ?

• Context dependent

• Ambiguous with multiple interpretations

• General principles based on image primitives alone

• Consider a few well-known examples

• There are many more methods in the literature

Segmentation

Active Contours

• Snakes : Energy minimisation on 2-D spline (controls + imagedata terms)

• Eint =∫α(s)||fs(s)||2 + β(s)||fss(s)||2ds

• s is arc-length parametrisation

• f(s) = (x(s), y(s))

• α and β are first and second-order weighting functions

• This is internal energy alone

Segmentation

Active Contours

• Snakes : Minimise net sum of various energy terms

• Eimage = wlineEline + wedgeEedge + wtermEterm

• Line term : attract to ridges

• Edge term : strong gradients

• Term : line termination

• Most practical systems use edge terms only

Segmentation

Active Contours

• Eedge =∑

i−||∇I(f(i))||2

• Interactive applications use user-constraints

• Attract to anchor points

• Espring = ki||f(i)− d(i)||2

• Also repulsive forces can be used

• Snakes have tendency to shrink• Use external boundary as initialisation• Use expansionary force (move along normal)

Segmentation

Active Contours

• Solve sparse linear system (block diagonal due to difference eqns.)

• Alternate between

• linear system solution• linearisation of non-linear constraints such as edge energy

Segmentation

Dynamic Snakes

• If object is to be tracked, use previous estimate

• Can ‘predict’ based on model

• xi = Axi−1 + wi

• Kalman filtering

• Limitation : Gaussian error assumption is restrictive

• Need to remove it for clutter etc.

Segmentation

CONDENSATION

• Remove restrictive Gaussian error model

• Propagate general multi-modal distribution

• Particle Filtering

• Collection of weighted samples• Update according to linear model• General multiple samples according to updated model• Perturb according to diffusion

• Point samples represent and propagate conditional estimates ofmulti-modal density

Segmentation

CONditional propagATION or CONDENSATION

• Active Contours cannot easily change topology as curve evolves

• Alternative to this is level set

• Curve is defined by zero-crossing of characteristic function basedon image data

• Track curve by modifying the underlying embedding fuctioninstead of f(s)

Segmentation

Watershed Algorithm

• Single threshold does not work

• Works on gray-scale images

• Flood landscape at all local minima

• Label ridges wherever evolving components meet

• Also use steerable filters for local detail

• Suffers from over-segmentation

Segmentation

Mean Shift

• k-means and mixture of Gaussians - parametric model for pdf

• Mean shift implicitly models pdf

• Uses smooth continuous non-parametric model

• Efficiently finds peaks in high-dim data

• Peaks without explicitly computing complete function

• Modes are inverse of watershed

Segmentation

Mean Shift

• Estimate density function given samples

f(x) =∑i

k(x− xi) =∑i

k

(||x− xi||2

h2

)Parzen window method

Segmentation

Mean Shift

• Given f(x) can find local minima

• Too expensive to evaluate f(x) in h-d space

• Solution : find mode of pdf directly

• Start at a random point

∇f(x) =∑i

(xi − x)G(x− xi)

=∑i

(xi − x)g

(||x− xi||2

h2

)g(r) = −k

′(r)

Segmentation

⇒ ∇f(x) =

[∑i

G(x− xi)

]m(x)

where

m(x) =

∑i xiG(x− xi)∑iG(x− xi)

− x

m(x) is the mean-shift

yk+1 = yk +m(yk) =

∑i xiG(yk − xi)∑iG(yk − xi)

Mean Shift

Figure : Colour image segmentation using mean shift

• K(xj) = k(||xr||2h2r

)k(||xs||2h2s

)• Spatial location : xs = (x, y)

• Colour values : xr

• xj contains both spatial location and colour value

• M specifies size below which clusters are discarded

Normalized Cuts

Segmentation using Affinities

• Consider graph of similarity relations G = (V,E)

• In our case V can be image pixels

• E : edges between neighbouring pixels (say)

• A cut

• Deletes some edges• Separates vertices into two sets• Results in two segments

Normalized Cuts

cut(A,B) =∑

i∈A,j∈Bwij

Properties of a Cut

• Cut between two sets of vertices A and B

• A ∪B = V and A ∩B = ∅• Cost of cut is the sum of weights of deleted edges

• Can result in degenerate solutions (minimisation)

• Cut isolated vertex

Normalized Cuts

Modify to better reflect desired propertiesReplace

cut(A,B) =∑

i∈A,j∈Bwij

with

Ncut(A,B) =cut(A,B)

assoc(A, V )+

cut(A,B)

assoc(B, V )

Normalized Cuts

Ncut(A,B) =cut(A,B)

assoc(A, V )+

cut(A,B)

assoc(B, V )

where

assoc(A,A) =∑

i∈A,j∈Awij

assoc(A, V ) = assoc(A,A) + cut(A,B)

• assoc(A,A) : association within cluster A

• assoc(A, V ) : sum of all weights with vertices in A

• Normalization for relative sizes of A and B

Normalized Cuts

Ncut(A,B) =cut(A,B)

assoc(A, V )+

cut(A,B)

assoc(B, V )

Hardness of Cut

• Solving for optimal cut is NP-complete

• Can be solved by relaxing labels

• Define indicator x

• xi = +1 ⇐⇒ i ∈ A• xi = −1 ⇐⇒ i ∈ B

Normalized Cuts

• Define d = W1

• Define D = diag(d)

• Let y = ((1 + x)− b(1− x))/2 s.t. y.d = 0

• Normalized cuts problem can be redefined

miny

yT (D−W)y

yTDy

Normalized Cuts

miny

yT (D−W)y

yTDy

• Cost function solving by minimising Rayleigh quotient

• Equivalent to solving (D−W)y = λDy

• Equivalent to (I−N)z = λz

• N = D−12WD−

12 (Normalized affinity matrix)

• z = D12y

• Laplacian matrix L = I−N

• Spectral Clustering

Normalized Cuts

Issues and Details

• Multiple segmentation carried out recursively

• Can also use multiple eigenvectors to classify

• Weights can capture spatial and image similarity relationships

wij = exp

(−||Fi − Fj ||2

2σ2F

− ||xi − xj ||2

2σ2s

)

• Original method is slow

• Many variations and modifications to speed up

• Modern methods : Discrete optimization on MRF’s

• Neighbourhood need not be local

• L is sparse

Normalized Cuts