Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
1
1Many slides are curtesy of S. Lazebnik , B. Girod and S. Seitz. Also thanx to Eric Weizmann, Zvi Solomon, Seri khoury , Jad Silbak, Avner Gidron, David Cohen
Segmentation
• Segmentation is the process of dividing an
image into “coherent regions”.
The goals of segmentation• The goal of segmentation is to simplify and/or change the
representation of an image into components that are more meaningful and easier to analyze.
• Image segmentation is the process of assigning a label to every pixel such that same-labeled pixels share certain characteristics.
“superpixels”
• Segmentation can be treated at different levels of details.
Object segments
Segments as primitives for recognition
J. Tighe and S. Lazebnik, ECCV 2010
2
• Interactive segmentation for graphics Approaches to segmentation
• Gray-level Histogram thresholding
• Segmentation as clustering
• Segmentation as graph partitioning
Global Thresholding –Segmentation into two components
oldV
F(V)
255
255
Threshold value
newV
T
0 100 2000
500
1000
1500
Histogram
Segmentation using Thresholding
50 75
Original
Threshold = 50 Threshold = 75
3
Segmentation using Thresholding
• How to choose the optimal threshold?
• Can a global threshold suffice?
Slide by B. Girod
Error Probability for Thresholding
• Assuming there are two components in the image:
– Foreground
– Background
• What is the probability of error given a threshold T?
T
Slide by B. Girod
Error Probability for Thresholding
• The optimal threshold T with respect to MAP estimation (Maximum A Posteriori) is T where � � � � ���|��
• Goal: Find the optimal T given an image.
T
Slide by B. Girod
Optimal Thresholding using Otsu’s Method.
Slide by B. Girod
4
Locally Adaptive Thresholding
• Slide a window over the image
• For each window decide whether to apply thresholding:
– Thresholding should not be performed in uniform windows.
– Can use variance or other criterion for “uniformity”.
• In non-uniform windows: apply Otsu’s thresholding based on local histogram.
• In uniform windows: classify the entire window as Foreground or Background based on mean value.
Slide by B. Girod
Locally Adaptive Thresholding
Slide by B. Girod
Edge Based Segmentation Edge Based Segmentation
We are looking for boundaries, not edges.
Problem with edges in presence of noise and
sampling artifacts (e.g. medical images).
5
Snakes – Active Contours
1. The snake is placed near the image contour of interest.
2. During an iterative process, the snake is attracted towards the target contour by various forces that control the shape and location of the snake within the image.
Snakes: Active Contour Models (1987)Michael Kass, Andrew Witkin, and Demetri Terzopoulos
Snakes
Works with noisy and discontinuous boundary edges
Snakes
• Representation of the contours
• Defining the energy functions
– Internal
– External
• Minimizing the energy function
(x0,y0, x1,y1,…., xn,yn)
Snakes
Usually, the total energy(cost function) of snake is
a combination of internal and external energies
exin EEE +=
Internal energy encouragessmoothness of shape
External energy encouragesalignment of curve withimage structures (edges).
(x0,y0, x1,y1,…., xn,yn)Minimize E over all possible locations of
6
External Energy
• Measure how well the curve matches the image data.
• “Attract” the curve toward different image features (edges, lines, etc)
• Think of external energy from image as gravitational pull towards areas of high contrast.
External Energy
For every image pixel v=(x,y) define its External Energy as:
22 |),(||)(|)( yxIIEex ∇−=∇−= vv
∫=1
0
))(( dssEE exex ν continuous case
]}1,0[s|)s({ ∈= νC
External energy term for the snake whose points are s :
}ni0|{ i <≤= νC
discrete case
External Energy
}ni0|{ i <≤= νC
discrete case
Snake moves to decrease external energy:
Internal Energy
The Internal Energy (= Smoothess) at contour point v(s) is defined as:
sd
d
ds
dsssEin 2
2
)()())((
22
ννβαν +=
Elasticity/stretching Stiffness/bending
The Internal Energy (smoothness) of the whole snake is :
∫=1
0
inin ds))s((EE ν ]}1,0[s|)s({ ∈= νC
7
Internal Energy
5v
4v3v
2v
1v6v
7v
8v
10v
9v
elastic energy(elasticity)
i1ivds
dν
ν−≈ +
bending energy(stiffness)
1ii1i1iii1i2
2
2)()(ds
d−+−+ ν+ν−ν=ν−ν−ν−ν≈
ν
Internal Energy
Elasticity Stiffness
∑−
=−++ +−+−=
1
0
2
11
2
1 |2|||n
i
iiiiiinE νννβννα
large α
small α large β small β
Snakes
Minimize exin EEE +=
(v0, v1,…., vn)over all possible locations of snake
vi = (xi,yi)
Gradient descent, iterative local minimization,Veterbi (dynamic programming)
Snakes
8
Tracking Using Snakes
(Original Terzopolous video)
Intelligent Scissors
Eric N. Mortensen and William A. Barrett, "Intelligent Scissors for Image Compostion", Proc. SIGGRAPH 95, 1995, 191-198.
Finds a path from seed to mouse that follows object
boundary as closely as possible?
Intelligent Scissors
Basic Idea
• Define edge score for each pixel
– edge pixels have very low cost
• Find lowest cost path from seed to mouse
seed
mouse
Uses Dijkstra’s shortest path algorithm
Intelligent Scissors
http://fotoflexer.com/app/
Demo
9
Color Segmentation:
Segmentation as clustering
Source: K. Grauman
K-means
• How to choose the representative colors?
– This is a clustering problem!
Objective
• Each point should be as close as possible to a cluster center
• Minimize sum squared distance of each point to closest center
This image
cannot
This image cannot
This
image cannot curren
R
G
R
G
Slide by Steve Seitz
This image cannot
This image cannot curren
This image cannot curren
Break it down into sub problems
• Suppose I tell you the cluster centers ci
– Q: how to determine which points to associate with each ci?
− A: for each point p, choose closest ci
Suppose I tell you the points in each cluster
• Q: how to determine the cluster centers?
• A: choose ci to be the mean of all points in the cluster
Slide by Steve Seitz
K-means clustering: algorithm
1. Randomly initialize the cluster centers, c1, ..., cK
2. Given cluster centers, determine points in each cluster
• For each point p, find the closest ci. Put p into cluster i
3. Given points in each cluster, solve for ci
• Set ci to be the mean of points in cluster i
4. If ci have changed, repeat Step 2
Text by Steve Seitz
10
K-Means - Example
י אריק ויצמן"נכתב ע
Choose K seed positions randomlyי אריק ויצמן"נכתב ע
K-Means - Example
K = 2
י אריק ויצמן"נכתב ע
K-Means - Example
Associate elements to nearest seed.י אריק ויצמן"נכתב ע
K-Means - Example
Update seed location to COM of all associated elements.
11
י אריק ויצמן"נכתב ע
K-Means - Example
Repeatי אריק ויצמן"נכתב ע
K-Means - Example
י אריק ויצמן"נכתב ע
K-Means - Example
י אריק ויצמן"נכתב ע
K-Means - Example
12
י אריק ויצמן"נכתב ע
K-Means - Example
Repeat until no change in element assignments to seeds.
K-Means – Image Segmentation
י אריק ויצמן"נכתב ע
Input Image
WhiteMatter
GrayMatter
Background CSF
K-Means – Image Segmentation
י אריק ויצמן"נכתב ע
Iteration:
126.6209
164.4521
209.5154 229.4411
1
Seed Locations: 13.2753
168.9954
204.1537 247.5998
2
2.5485
146.8334
204.1537 247.5998
3
1.4523
131.0522
197.6932 247.5998
4
1.0536
122.6038
195.0548 247.5998
5
1.0536
116.3612
192.7618 247.5998
6
0.7324
109.3117
190.8013 247.5998
7
Image Intensity-based clusters Color-based clusters
Segmentation as clustering
• K-means clustering based on intensity or color is essentially vector quantization of the image attributes
Slide by Svetlana Lazebnik
13
Segmentation as clustering
• K-means clustering based on texture
Segmentation as clustering
Clusters don’t have to be spatially coherent
Segmentation as clustering
Source: K. Grauman
Segmentation as clustering
• Clustering based on (r,g,b,x,y) values enforces more spatial coherence
Slide by Svetlana Lazebnik
14
Over segmentation – Super pixels using K-means
• SLIC (Simple Linear Iterative Clustering) super pixel based on k-means clustering (r,g,b,x,y)
• Initial centers – regular kxk windows.
• Spatial domain and color domain can be weighted differently.
• Iterate k-means until convergence.
• Taking the role of pixels in many applications.
Achanta, et. Al (2010).
Slic superpixels (EPFL-REPORT-149300).
K-Means for segmentation
• Pros
– Very simple method
– Converges to a local minimum of the error function
• Cons
– Memory-intensive
– Need to pick K
– Sensitive to initialization
– Sensitive to outliers
– Only finds “spherical” clusters
Slide by Svetlana Lazebnik
Mean shift segmentation
• The mean shift algorithm seeks modes or local
maxima of density in the feature space
imageFeature space
(L*u*v* color values)
Non-parametric density mode estimation
56
Non-parametric
Density Estimation
Non-parametric
MODE Estimation
(Mean Shift)
Data
Discrete PDF Representation
PDF Analysis
• Non-parametric – no assumption about PDF form (e.g. normal distribution)
• Density Gradient is estimated instead of Density itself.
15
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
Search
window
Center of
mass
Mean Shift
vector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
Search
window
Center of
mass
Mean Shift
vector
Slide by Y. Ukrainitz & B. Sarel
Mean shiftSearch
window
Center of
mass
Mean Shift
vector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
16
Search
window
Center of
mass
Mean Shift
vector
Slide by Y. Ukrainitz & B. Sarel
Mean shiftSearch
window
Center of
mass
Mean Shift
vector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
Search
window
Center of
mass
Slide by Y. Ukrainitz & B. Sarel
Mean shift Mean shift clustering
• Cluster: all data points in the attraction basin
of a mode
• Attraction basin: the region for which all
trajectories lead to the same mode
Slide by Y. Ukrainitz & B. Sarel
17
Mean shift clustering/segmentation
• Find features (color, gradients, texture, etc)
• Initialize windows at individual feature points
• Perform mean shift for each window until convergence
• Merge windows that end up near the same “peak” or mode
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
Mean shift segmentation results
Mean shift segmentation results Mean shift segmentation results
18
Mean shift
• Pros:– Does not assume shape on clusters
– One parameter choice (window size)
– Generic technique
– Find multiple modes
• Cons:– Selection of window size
– Does not scale well with dimension of feature space
Kristen Grauman