Segmentation and Segmentation and ClusteringClustering

Déjà Vu?Déjà Vu?

• Q: Haven’t we already seen this with Q: Haven’t we already seen this with snakes?snakes?

• A: no, that was about boundaries, A: no, that was about boundaries, this is about interiorsthis is about interiors– Want regions of similar contentsWant regions of similar contents

Segmentation via ClusteringSegmentation via Clustering

• Segmentation:Segmentation:Divide imageDivide imageinto regionsinto regionsof similar of similar contentscontents

• Clustering:Clustering:Aggregate pixelsAggregate pixelsinto regionsinto regionsof similar of similar contentscontents

But Wait!But Wait!

• We speak of segmenting foregroundWe speak of segmenting foregroundfrom backgroundfrom background

• Segmenting out skin colorsSegmenting out skin colors

• Segmenting out the moving personSegmenting out the moving person

• How do these relate to “similar How do these relate to “similar regions”?regions”?

Segmentation and ClusteringSegmentation and Clustering

• Defining regionsDefining regions– Should they be compact? Smooth Should they be compact? Smooth

boundary?boundary?

• Defining similarityDefining similarity– Color, texture, motion, …Color, texture, motion, …

• Defining similarity of regionsDefining similarity of regions– Minimum distance, mean, maximumMinimum distance, mean, maximum

Segmentation and Clustering Segmentation and Clustering ApplicationsApplications

SemanticsSemantics

““IntelligentIntelligentscissors”scissors”

FindingFindingskin-coloredskin-colored

regionsregions

Foreground /Foreground /backgroundbackground

segmentationsegmentation

Finding theFinding themoving objectsmoving objects

Finding theFinding thecars in acars in a

video sequencevideo sequence

ThemesThemes

• Energy minimizationEnergy minimization– Snakes: discretize, greedy minimizationSnakes: discretize, greedy minimization– Snakes, shape from shading: differential Snakes, shape from shading: differential

eqneqn– Stereo with smoothness: min-cost graph Stereo with smoothness: min-cost graph

cutscuts– Today: Today: kk-means clustering-means clustering

• StatisticsStatistics

• TemplatesTemplates

Hot topic in vision research: combining theseHot topic in vision research: combining these

Segmentation and Clustering Segmentation and Clustering ApplicationsApplications

““IntelligentIntelligentscissors”scissors”

FindingFindingskin-coloredskin-colored

regionsregions

Foreground /Foreground /backgroundbackground

segmentationsegmentation

Finding theFinding themoving objectsmoving objects

Finding theFinding thecars in acars in a

video sequencevideo sequence

StatisticsStatistics TemplatesTemplates

Clustering Based on ColorClustering Based on Color

• Let’s make a few concrete choices:Let’s make a few concrete choices:– Arbitrary regionsArbitrary regions– Similarity based on color onlySimilarity based on color only– Similarity of regions =Similarity of regions =

distance between mean colorsdistance between mean colors

Simple Agglomerative Simple Agglomerative ClusteringClustering

• Start with each pixel in its own Start with each pixel in its own clustercluster

• Iterate:Iterate:– Find pair of clusters with smallestFind pair of clusters with smallest

inter-cluster distanceinter-cluster distance– MergeMerge

• Stopping thresholdStopping threshold

Simple Divisive ClusteringSimple Divisive Clustering

• Start with whole image in one clusterStart with whole image in one cluster

• Iterate:Iterate:– Find cluster with largest intra-cluster Find cluster with largest intra-cluster

variationvariation– Split into two pieces that yield largest Split into two pieces that yield largest

inter-cluster distanceinter-cluster distance

• Stopping thresholdStopping threshold

Difficulties with Simple Difficulties with Simple ClusteringClustering

• Many possibilities at each iterationMany possibilities at each iteration

• Computing distance between clusters or Computing distance between clusters or optimal split expensiveoptimal split expensive

• Heuristics to speed this up:Heuristics to speed this up:– For agglomerative clustering, approximate For agglomerative clustering, approximate

each cluster by average for distance each cluster by average for distance computationscomputations

– For divisive clustering, use summary For divisive clustering, use summary (histogram) of a region to compute split(histogram) of a region to compute split

kk-means Clustering-means Clustering

1.1. Pick number of clusters Pick number of clusters kk

2.2. Randomly scatter Randomly scatter kk “cluster centers” in color “cluster centers” in color spacespace

3.3. Repeat:Repeat:a.a. Assign each data point to its closest cluster Assign each data point to its closest cluster

centercenterb.b. Move each cluster center to the mean of the Move each cluster center to the mean of the

points assigned to itpoints assigned to it

• Instead of merging or splitting, start Instead of merging or splitting, start out with the clusters and move them out with the clusters and move them aroundaround

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

kk-means Clustering-means Clustering

k-means Clusteringk-means Clustering

• This process always converges (to This process always converges (to something)something)– Not necessarily globally-best assignmentNot necessarily globally-best assignment

• Informal proof: look at energy minimizationInformal proof: look at energy minimization

– Reclassifying points reduces (or maintains) energyReclassifying points reduces (or maintains) energy– Recomputing centers reduces (or maintains) energyRecomputing centers reduces (or maintains) energy– Can’t reduce energy foreverCan’t reduce energy forever

points clusters

2

i jijji assignedxxE

points clusters

2

i jijji assignedxxE

Results of ClusteringResults of Clustering

Original ImageOriginal Image kk-means, -means, kk=5=5 kk-means, -means, kk=11=11

Results of ClusteringResults of Clustering

Sample clusters withSample clusters with k k-means clustering-means clusteringbased on colorbased on color

Other Distance MeasuresOther Distance Measures

• Suppose we want to have compact Suppose we want to have compact regionsregions

• New feature space: 5DNew feature space: 5D(2 spatial coordinates, 3 color (2 spatial coordinates, 3 color components)components)

• Points close in this space are close Points close in this space are close both in color and in actual proximityboth in color and in actual proximity

Results of ClusteringResults of Clustering

Sample clusters with Sample clusters with kk-means clustering-means clusteringbased on color and distancebased on color and distance

Other Distance MeasuresOther Distance Measures

• Problem with simple Euclidean distance:Problem with simple Euclidean distance:what if coordinates range from 0-1000 but what if coordinates range from 0-1000 but colors only range from 0-255?colors only range from 0-255?– Depending on how things are scaled, gives Depending on how things are scaled, gives

different weight to different kinds of datadifferent weight to different kinds of data

• Weighted Euclidean distance: adjust Weighted Euclidean distance: adjust weights to emphasize different dimensionsweights to emphasize different dimensions

22)( iii yxcyx 22)( iii yxcyx

Mahalanobis DistanceMahalanobis Distance

• Automatically assign weights based Automatically assign weights based on actual variation in the dataon actual variation in the data

where C is covariance matrix of all where C is covariance matrix of all pointspoints

• Gives each dimension “equal” weightGives each dimension “equal” weight

• Also accounts for correlations Also accounts for correlations between different dimensionsbetween different dimensions

)(1T2yxyxyx

C )(1T2yxyxyx

C