Grouping and Segmentation

Grouping and Segmentation

• Previously: model-based grouping (e.g., straight line, figure 8…)

• Now: general ``bottom-up” image organization

(Like detecting-specific-brightness-patterns vs “interesting” patterns/interest points/corners)

Grouping

• Grouping is the process of associating similar image features together

Grouping

• Grouping is the process of associating similar image features together

• The Gestalt School:

– Proximity: nearby image elements tend to be grouped.– Similarity: similar image elements ... – Common fate: image elements with similar motion ...– Common region: image elements in same closed region...– Parallelism: parallel curves or other parallel image elements ...– Closure: image elements forming closed curves ...– Symmetry: symmetrically positioned image elements...– Good continuation: image elements that join up nicely ...– Familiar Conguration: image elements giving a familiar object...

Good Continuation

Good continuation

Common Form: (includes color and texture)

Connectivity

Symmetry

Symmetry

Convexity (stronger than symmetry?)

Good continuation also stronger than symmetry?

Closureclosed curves are much easier to pick out than open ones

Grouping and Segmentation

• Segmentation is the process of dividing an image into regions of “related content”

Courtesy Uni Bonn

Grouping and Segmentation

• Segmentation is the process of dividing an image into regions of “related content”

Courtesy Uni Bonn

Grouping and Segmentation

• Both are ill defined problems “Related” or “similar” are “high level” concepts. Hard to apply directly to image data

• How are they used? Are they “early” process that precedes recognizing objects?

• A lot of research, some pretty good algorithms. No single best approach.

Boundaries

Problem: Find best path between two boundary points.

User can help by clicking on endpoints

How do we decide how good a path is? Which of two paths is better?

Discrete Grid• Curve is good if it follows object boundary So it should have:

• Strong gradients along curve (on average)

• Curve directions roughly perpendicular to gradients

(since these is usually true for boundaries)

• Curve should be smooth, not jagged Since:

• Object boundaries typically smooth

• Curves that follow “noise edges” typically jagged

So good curve has:• Low curvature (on average) along curve

• Slow change in gradient direction along curve

Discrete Grid

• How to find the best curve?

• Approach- Define cost function on curves

(Given any curve, rule for computing its cost)

- Define so good curves have lower cost (non-wiggly curves passing mostly through edgels have low cost)

- Search for best curve with lowest cost (minimize cost function over all curves)

Discrete Grid

• How to find the best curve?

• Approach- Define cost function on curves

(Given any curve, rule for computing its cost)

- Define so good curves have lower cost (non-wiggly curves passing mostly through edgels have low cost)

- Search for best curve with lowest cost (minimize cost function over all curves)

• How to compute cost?

- At every pixel in image, define costs for “curve fragments” connecting it to adjoining pixels - Compute curve cost by summing up costs for fragments.

Defining cost: smoothness

• Path: series of pixels , , … , ,

• Good curve has small curvature at each pixel– Small direction changes.– small on average (summing along curve)

• Good curve has small changes in gradient direction

One possible term in cost definition:

11

1

i ii i

i i

I I p pI I

1ip 1, ...ip

ip2p1p

1ip

ip

1ip

1ip

ip

Fragment Fragment Cost

Defining cost: smoothness

• Path: series of pixels , , … , ,

• Good curve has small curvature at each pixel– Small direction changes.– small on average (summing along curve)

• Good curve has small changes in gradient direction

One possible term in cost definition:

11

1

i ii i

i i

I I p pI I

small when gradients at both nearly perpendicular to fragment direction

1,i ip p

1i ip p

1ip 1, ...ip

ip2p1p

1ip

ip

1ip

1ip

ip

Fragment Fragment Cost

Defining cost: smoothness

• Path: series of pixels , , … , ,

• Good curve has small curvature at each pixel– Small direction changes.– small on average (summing along curve)

• Good curve has small changes in gradient direction

One possible term in cost definition:

11

1

i ii i

i i

I I p pI I

small when gradients at both nearly perpendicular to fragment direction

1,i ip p

1i ip p

1ip 1, ...ip

If this small at both i, i+1, then small at i.Low average value of this quantity low average value (good curve)

ip2p1p

1ip

ip

1ip

1ip

ip

Fragment Fragment Cost

Path Cost Function (better path has smaller cost)

• Path: , , , ,

• Total cost: sum cost for each pixel over whole path One possible cost definition:

11 11

1

| | ( )n i ii i i i ii

i i

I Ip p g p p pI I

p ip1 p2

2max

1 ( some constant)( / )i

i

g pI I

1... ip 1, ...ip

Path Cost Function (better path has smaller cost)

• Path: , , , ,

• Total cost: sum cost for each pixel over whole path One possible cost definition:

11 11

1

| | ( )n i ii i i i ii

i i

I Ip p g p p pI I

p ip1 p2

2max

1 ( some constant)( / )i

i

g pI I

1... ip 1, ...ip

Small for high gradient,small direction change

How do we find the best Path? Computer Science…

Remember:

Curve is path through grid.

Cost sums over each step of path.

We want to minimize cost.

Map problem to Graph

Edges connect pixels

Edge weight = cost of fragment connecting the two pixels

Each pixel is a node

Map problem to Graph

Edge weight

Example Cost:

Edge weight = cost of fragment connecting the two pixels

1 111

1

| | ( ) i ii i

ni

ii i i

i

I Ip p g p p pI I

Map problem to Graph

Edge weight

Example Cost:

Edge weight = cost of fragment connecting the two pixels

1 111

1

| | ( ) i ii i

ni

ii i i

i

I Ip p g p p pI I

Note: this is just an example of a cost function. The cost function actually used differs

Algorithm: basic idea

• Two steps:

1) Compute least costs for best paths from start pixel to all other pixels.

2) Given a specific end pixel, use the cost computation to get best path between start and end pixels.

Dijkstra’s shortest path algorithm

0531

33

4 9

2

• Algorithm: compute least costs from start node to all other nodes

Iterate outward from start node, adding one node at a time. Cost estimates start high, nodes gradually “learn” their true cost

Always choose next node so can calculate its correct cost.

link cost

Dijkstra’s shortest path algorithm

4

1 0

5

3

3 2 3

9

531

33

4 9

2

• Algorithm: compute least costs from start node to all other nodes

Iterate outward from start node, adding one node at a time. Cost estimates start high, gradually improve

Always choose next node so can calculate its correct cost.

Example: choose left node L with cost 1. Min cost to L equals 1 since any other path to L gets cost>1 on first step from start node

L

Dijkstra’s shortest path algorithm

• Computes minimum costs from seed to every pixel

– Running time (N pixels): O(N log N) using active priority queue (heap)

fraction of a second for a typical (640x480) image

• Then find best path from seed to any point in O(N).

Real time!

Intelligent Scissors

Results

Voting again

• Recall Hough Transform

– Idea: data votes for best model(eg, best line passing through data points)

– Implementation• Choose type of model in advance (eg, straight lines)• Each data point votes on parameters of model

(eg, location + orientation of line)

-0.2 0 0.2 0.4 0.6 0.8 1 1.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-80 -60 -40 -20 0 20 40 60 80-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Line Direction (degrees)

Dis

tanc

e to

orig

in

-80 -60 -40 -20 0 20 40 60 80-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Line Direction (degrees)

Dis

tanc

e to

orig

in

-0.2 0 0.2 0.4 0.6 0.8 1 1.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Another type of voting

• Popularity contest

– Each data point votes for others that it “likes,” that it feels “most related to”

– “Most popular” points form a distinct group

– (Like identifying social network from cluster of web links)

B A

B “feels related” to A and votes for it.

C

A

C also “feels related” to A and votes for it.

DA

So do D…

A

So do D and E

E

“Popular” points form a group, with “friends”reinforcing each other

feels unconnected to , doesn’t vote for it(or votes against)

also feels unconnected and doesn’t vote for

Only B feels connected and votes for (but it’s not enough)

B

is unpopular and left out of group

Note: even though can connect smoothly to , it doesn’t feel related since it’s too far away

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Grouping:• Start at salient point

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Grouping:• Start at salient point

• Grow curve toward most related salient point

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Grouping:• Start at salient point

• Grow curve toward most related salient point

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Grouping:• Start at salient point

• Grow curve toward most related salient point

Identifying groups by “popularity”

• In vision, technical name for “popularity” is saliency

• Procedure

– Voting: Pixels accumulate votes from their neighbors

– Grouping: Find smooth curves made up of the most salient pixels (the pixels with the most votes)

Grouping:• Start at salient point

• Grow curve toward most related salient point

Affinity• In vision, the amount of “relatedness” between two data points is

called their affinity

Related curve fragments: high affinity

Affinity• In vision, the amount of “relatedness” between two data points is

called their affinity

Related curve fragments: high affinity (Since can be connected by smooth curve)

Affinity• In vision, the amount of “relatedness” between two data points is

called their affinity

Related curve fragments: high affinity (Since can be connected by smooth curve)

Unrelated curve fragments: low affinity

Affinity• In vision, the amount of “relatedness” between two data points is

called their affinity

Related curve fragments: high affinity (Since can be connected by smooth curve)

Unrelated curve fragments: low affinity (Can’t be connected by smooth curve)

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers

1 1,x y

2 2,x y

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers is angle between chord and x-axis

1 1,x y

2 2,x y

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers is angle between chord and x-axis

1 1,x y

2 2,x y

1 1

Angle of first fragment with X axis

1

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers is angle between chord and x-axis

1 1,x y

2 2,x y

1 1

2 2

Angle of second fragment with X axis

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers is angle between chord and x-axis

1 1,x y

2 2,x y

1 1

2 2

2 12Co C

Computing Affinity• Co-circularity: a simple measure of affinity between line fragments

– Two fragments are smoothly related if both lie on same circle.

– Two fragments on the same circle have property that 2

1

1 2

Chord joining line fragment centers is angle between chord and x-axis

1 1,x y

2 2,x y

1 1

2 2

2 12Co C

Co-circularity: Second fragment angle determined byangle of first, and angle of connecting line

Computing Affinity• is the orientation that second fragment should have if it

connects smoothly to first fragment.

• Measure affinity by difference with actual orientation:

2Co C

2 2Co C

2

1 1 1 1,E x y

2 2 2,E x y

2 2

2 2 1 2 1 1 1 21 2 2

, ,, exp

2

Co C Co C

orient

E E E EA E E

2

1 22exp

2 dist

E E

Example (use Gaussians so affinity falls off smoothly):

Edgel locations

Computing Affinity• is the orientation that second fragment should have if it

connects smoothly to first fragment.

• Measure affinity by difference with actual orientation:

2Co C

2 2Co C

2

1 1 1 1,E x y

2 2 2,E x y

2 2

2 2 1 2 1 1 1 21 2 2

, ,, exp

2

Co C Co C

orient

E E E EA E E

Example (use Gaussians so affinity falls off smoothly):

Treat both edge fragments in same way: sum orientation discrepancy for both

2

1 22exp

2 dist

E E

Computing Affinity• is the orientation that second fragment should have if it

connects smoothly to first fragment.

• Measure affinity by difference with actual orientation:

2Co C

2 2Co C

2

1 1 1 1,E x y

2 2 2,E x y

2 2

2 2 1 2 1 1 1 21 2 2

, ,, exp

2

Co C Co C

orient

E E E EA E E

Example (use Gaussians so affinity falls off smoothly):

More distant edges should have less affinity

2

1 22exp

2 dist

E E

Computing Affinity

• Why give smaller affinity to distant fragments?

– Distant fragments likely to come from different objects– Many distant fragments, so some line up just by

random chance

Computing Affinity

• For faster computation, fewer affinities: Set affinities to 0 beyond some distance T:

(Often choose T small: e.g., less than 7 pixels, or just to include the nearest neighbors)

1 2 1 2, 0 for A E E E E T

Eigenvector Grouping

Reminder:Grouping with Affinity Matrix

Task: Partition image elements into groups G1,..,Gn

of similar elements.

Affinities should be:

– high within group:

– low between groups:

, , highai j G Aff i j

, , lowa b ai G j G Aff i j

Eigenvector grouping

• Eigenvector grouping is just improved popularity grouping!

• Idea: – To judge your own popularity, weight the votes from your

friends according to their popularity.

Eigenvector grouping

• Eigenvector grouping is just improved popularity grouping!

• Idea: – To judge your own popularity, weight the votes from your

friends according to their popularity.

– Recompute using better estimate of friends popularity:(use weighted voting for them also)

Eigenvector grouping

• Eigenvector grouping is just improved popularity grouping!

• Idea: – To judge your own popularity, weight the votes from your

friends according to their popularity.

– Recompute using better estimate of friends popularity:(use weighted voting for them also)

– Even better if recompute using this improved estimate for friends

Eigenvector grouping

• Eigenvector grouping is just improved popularity grouping!

• Idea: – To judge your own popularity, weight the votes from your

friends according to their popularity.

– Recompute using better estimate of friends popularity:(use weighted voting for them also)

– Even better if recompute using this improved estimate for friends

Eigenvector grouping

• Eigenvector grouping is just improved popularity grouping!

• Idea: – To judge your own popularity, weight the votes from your

friends according to their popularity.

– Recompute using better estimate of friends popularity:(use weighted voting for them also)

– Even better if recompute using this improved estimate for friends

– Iterating this, “weighted popularity” converges to eigenvector!

• Your popularity = sum of friend’s votes 11

P Aff11

• Your popularity = sum of friend’s votes 11

1P= AffN

11

Usually add normalization so

popularities always add to 1

• Your popularity = sum of friend’s votes

But don’t care about normalization.Important thing is that P is proportional to this

11

P ~ Aff11

• Your popularity = sum of friend’s votes

• Now use new estimate of friend’s popularity to weight their vote

P ~ Aff Pnew old

11

P ~ Aff11

• Your popularity = sum of friend’s votes

• Now use new estimate of friend’s popularity to weight their vote

2

11

P ~ Aff P ~ Aff

1

new old

11

P ~ Aff11

• Your popularity = sum of friend’s votes

• Again use new estimate…

3

11

P ~ Aff P ~ Aff

1

newer new

11

P ~ Aff11

• Your popularity = sum of friend’s votes

• Keep going…

11

P ~ lim Aff

1

Nbest N

11

P ~ Aff11

• Your popularity = sum of friend’s votes

• Keep going…

11

P ~ Aff11

11

P ~ lim Aff

1

Nbest N

An eigenvector of ! Aff

• Your popularity = sum of friend’s votes

• Keep going…

11

P Aff11

11

P ~ lim Aff

1

Nbest N

An eigenvector of !

Why?

For very large N,

(one more time makes little difference)

Aff

N+1Aff ~ Aff NV V

NAff Aff ~ Aff NV V

• Your popularity = sum of friend’s votes

• Keep going…

11

P Aff11

11

P ~ lim Aff

1

Nbest N

An eigenvector of !

Why?

For very large N,

(one more time makes little difference)

Aff

N+1Aff ~ Aff NV V

NAff Aff ~ Aff NV VEigenvector of Aff, by definition

• Your popularity = sum of friend’s votes

• Another view

11

P Aff11

big big

11

P ~ lim Aff ~

1

N Nbest N E

Multiplying by Aff many times,

only the component E with largest |Aff E| remains from (1 1 1 1…1)

• Your popularity = sum of friend’s votes

• Another view

11

P Aff11

big big

11

P ~ lim Aff ~

1

N Nbest N E

Multiplying by Aff many times,

only the component E with largest |Aff E| remains from (1 1 1 1…1)

So this is the eigenvector of Affwith the biggest eigenvalue

Biggest Eigenvector: Math

11

1

i ii

c

E

Eigenvectors of Aff (or any symmetric matrix) give a rotated coordinated system.So we can write

Eigenvector

Biggest Eigenvector: Math

11

1

i ii

c

E

Eigenvectors of Aff (or any symmetric matrix) give a rotated coordinated system.So we can write

Eigenvector

N

11

Aff Aff

1

NNi i i i i

i i

c c

E E

Biggest Eigenvector: Math

11

1

i ii

c

E

Eigenvectors of Aff (or any symmetric matrix) give a rotated coordinated system.So we can write

Eigenvector

N

11

Aff Aff

1

NNi i i i i

i i

c c

E E

Eigenvalue for iE

Biggest Eigenvector: Math

11

1

i ii

c

E

Eigenvectors of Aff (or any symmetric matrix) give a rotated coordinated system.So we can write

Eigenvector

N

11

Aff Aff

1

NNi i i i i

i i

c c

E E

Eigenvalue for iE

Biggest eigenvalue dominates:

NNi j i j

Eigenvector Grouping: Another View

• Toy problem– Single foreground group:

Aff , 1 if , both in foreground groupi j i j

0 otherwise ( or in background)i j

Eigenvector Grouping

• Toy problem– Single foreground group:

Aff , 1 if , both in foreground groupi j i j

0 otherwise ( or in background)i j

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…

Eigenvector Grouping

• Toy problem– Single foreground group:

Aff , 1 if , both in foreground groupi j i j

0 otherwise ( or in background)i j

1 1 ... 1 0 ... 01 1 ... 1 0 ... 0... ... ... 1 0 ... 0

Aff 1 ... ... 1 0 ... 00 ... 0... ... ...0 0

This is what Aff looks like if pixels in foreground group come before background pixels

Grouping

• Toy problem: single foreground group:

• Algorithm to find group– Find unit vector V that makes largest

Affinity matrix:

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…Aff V

Grouping

• Toy problem: single foreground group:

• Algorithm to find group– Find unit vector V that makes largest– Large entry in V indicates group member.

Affinity matrix:

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…Aff V

Grouping

• Toy problem: single foreground group:

• Algorithm to find group– Find unit vector V that makes largest– Answer ~

Affinity matrix:

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…

1 1 0 1 0 ... 0 1 1 .... 1 0

Aff V

(because every nonzero entry of V contributes as much as possible to |Aff V|)

Grouping

• Toy problem: single foreground group:

• Algorithm to find group– Find unit vector V that makes largest– To compute V: find ‘largest’ eigenvector of

Affinity matrix:

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…

Aff(with the largest eigenvalue)

Aff V

Grouping

• Toy problem: single foreground group:

• Algorithm to find group– Find unit vector V that makes largest– To compute V: find ‘largest’ eigenvector of

(This gives largest value for |Aff V|)

Affinity matrix:

1 1 0 1 0 ... 0 1 1 .... 1 01 1 0 1 0 ... 0 1 1 ... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

Aff 1 1 0 1 0 ... 0 1 1 .... 1 0

1 1 0 1 0 ... 0 1 1 .... 1 00 0 0 0 0 ... 0 0 0 ... 0 0

…

AffAff V

Eigenvector Grouping

• Toy problem– Single foreground group:

• (Toy) Algorithm summary:

– Compute largest eigenvector E of Aff– Assign foreground if– Assign background if

i

Aff , 1 if , both in foreground groupi j i j

0 otherwise ( or in background)i j

| | 0iE | | 0iE i

Eigenvector Grouping

Why does largest eigenvector work?

Leading eigenvector E of A has properties:

(by definition)maxAE E•

Eigenvector Grouping

Why does largest eigenvector work?

Leading eigenvector E of A has properties:

(by definition)

Eigenvalue (a number)Should be largest eigenvalue

maxAE E•

Eigenvector Grouping

Why does largest eigenvector work?

Leading eigenvector E of A has properties:

maxAE E (by definition)

max

maxV

AVV

occurs at

(max value )

V E

•

• Fact:

Eigenvector Grouping

Why does largest eigenvector work?

Leading eigenvector E of A has properties:

maxAE E (by definition)

max

maxV

AVV

occurs at

(max value )

V E

•

• Fact:

Interpretation: E is “central direction” of all rows of A (largest dot products with rows of A)

Aside: Math

max

occurs at

(max value )

V E• Fact: 2

2maxV

AVV

(equivalent to maximize the square)

Aside: Math

• Derive from calculus: max is at stationary point

2

2maxV

AV

V occurs for V E

2

20T T

T

AVd d V A AVdV dV V VV

2

22T TT

T T

V V A AVA AVV V V V

Show

Aside: Math

• Derive from calculus: max is at stationary point2

20T T

T

AVd d V A AVdV dV V VV

2

22T TT

T T

V V A AVA AVV V V V

,TA A V VT T

T

V A AVV V

2

2maxV

AV

V occurs for V EShow

Aside: Math

• Derive from calculus: max is at stationary point2

20T T

T

AVd d V A AVdV dV V VV

2

22T TT

T T

V V A AVA AVV V V V

,TA A V VT T

T

V A AVV V

At max V is an eigenvector of TA A

Aside: Math

• Derive from calculus: max is at stationary point2

20T T

T

AVd d V A AVdV dV V VV

2

22T TT

T T

V V A AVA AVV V V V

,TA A V VT T

T

V A AVV V

At max V is an eigenvector of TA A

V an eigenvector of A

Aside: Math

• Derive from calculus: max is at stationary point2

20T T

T

AVd d V A AVdV dV V VV

2

22T TT

T T

V V A AVA AVV V V V

,TA A V VT T

T

V A AVV V

At max V is an eigenvector of TA A

V an eigenvector of A2TA A A

2AE E A AE A E AE E

Why? A symmetric A has same eigenvectors as ,

since

Eigenvector Grouping

• More realistic problem– Single foreground group:

Aff , if , both in foregrolar unge dgroupi j i j

otherwise( or in backgsm ro d)all uni j

... ...

... ...... ... ... ...

Aff ... ... ......

... ... ...

Big Big Big Small Small

Big Big Small SmallSmall Small

Small Small

BIG smallsmall small

Eigenvector Grouping

• More realistic problem– Single foreground group:

Aff , if , both in foregrolar unge dgroupi j i j

otherwise( or in backgsm ro d)all uni j

... ...

... ...... ... ... ...

Aff ... ... ......

... ... ...

Big Big Big Small Small

Big Big Small SmallSmall Small

Small Small

BIG smallsmall small

Eigenvector also has form ;E BIG small

Eigenvector Grouping

• More realistic problem– Single foreground group:

Aff , if , both in foregrolar unge dgroupi j i j

otherwise( or in backgsm ro d)all uni j

... ...

... ...... ... ... ...

Aff ... ... ......

... ... ...

Big Big Big Small Small

Big Big Small SmallSmall Small

Small Small

BIG smallsmall small

Eigenvector also has form ;E BIG small

(E is unit vector. To getlargest product with Aff,more efficient to have bigentries where they multiply big entries of Aff .)

Eigenvector Grouping

• More realistic problem– Single foreground group:

Aff , if , both in foregrolar unge dgroupi j i j

otherwise( or in backgsm ro d)all uni j

... ...

... ...... ... ... ...

Aff ... ... ......

... ... ...

Big Big Big Small Small

Big Big Small SmallSmall Small

Small Small

BIG smallsmall small

Eigenvector also has form ;E BIG small Can identify foreground items by their big entries of E

Eigenvector Grouping

• Real Algorithm summary

– Compute largest eigenvector E of Aff

– Assign foreground if

– Assign background if(T a threshold chosen by you)

i | |iE T

| |iE Ti

Eigenvector Grouping

• Even more realistic problem– Several groups:

– Example: two groups

Aff , if , both inlarge same groupi j i j

s otherwise( and in groups)mall differenti j

Aff'

BIG smallsmall BIG

Group 1 Group 2

Eigenvector Grouping

• Even more realistic problem– Several groups:

– Example: two groups

Aff , if , both inlarge same groupi j i j

s otherwise( and in groups)mall differenti j

Aff'

BIG smallsmall BIG

Group 1 Group 2

Usually, leading eigenvector picks out one of the groups (the one with biggest affinities). In this case, we again expect ;E BIG small

Technical Aside

Why does eigenvector pick out just one group?

• Toy example: 2 x 2 “affinity matrix”

1

2

00B

AB

are “big” values; affinities between “groups” = 01 2B B

Technical Aside

Why does eigenvector pick out just one group?

• Toy example: 2 x 2 “affinity matrix”

1

2

00B

AB

are “big” values; affinities between “groups” = 01 2B B

Leading eigenvector is with eigenvalue 10

E

1B

Intuition: E has to be a unit vector. To be most efficient in getting large dot product , it put all its large values into group with largest affinities

Eigenvector Grouping

• For several groups, leading eigenvector picks out the “leading” (most self similar) group.

(Remember: calculate eigenvector by svd on )

• To identify other groups, can remove points from the leading group and repeat leading eigenvector computation for remaining points

TA A

Eigenvector Grouping

• For several groups, leading eigenvector (or singular vector) picks out the “leading” (most self similar) group.

• To identify other groups, can remove points from the leading group and repeat leading eigenvector computation for remaining points

• Alternative: just use non-leading eigenvectors/eigenvalues to pick out the non-leading groups.

(Alternative) Algorithm Summary

• Choose affinity measures and create affinity matrix Aff

• Compute eigenvalues for Aff. Assign all elements to active list L

• Repeat for each eigenvalue starting from largest:

– Compute corresponding eigenvector

– Choose threshold , assign to new group

– Remove new group elements i from active list L.

– Stop if L empty or new group too small.

• “Clean up” groups (optional)

{ : | | }ii L E T T

Eigenvector Grouping

• Problems

Method picks “strongest” group.

When there are several groups of similar strength, it can get confused

• Eigenvector algorithm described so far not used currently.

Eigenvectors + Graph Cut Methods

Graph Cut Methods

• Image undirected graph G = (V,E)

• Each graph edge has weight value w(E)

• Example– Vertex nodes =edgels i– Graph edges go between edgels i, j– Graph edge has weight w(i,j)=Aff(i,j)

Task: Partition V into V1...Vn, s.t. similarity is high within groups and low between groups

Issues

• What is a good partition ?• How can you compute such a partition efficiently ?

Graph Cut

• G=(V,E)

• Sets A and B are a disjoint partition of V

Graph Cut

• G=(V,E)

• Sets A and B are a disjoint partition of V

Measure of dissimilarity between the two groups:

,

( , ) ( , )u A v B

Cut A B w u v

Graph Cut

Cut

• G=(V,E)

• Sets A and B are a disjoint partition of V

Measure of dissimilarity between the two groups:

,

( , ) ( , )u A v B

Cut A B w u v

Cut

Graph Cut

Cut links that get summedover

• G=(V,E)

• Sets A and B are a disjoint partition of V

Measure of dissimilarity between the two groups:

,

( , ) ( , )u A v B

Cut A B w u v

Cut

The temptation

Cut is a measure of disassociation

Minimizing Cut gives partition with maximum disassociation.

Efficient poly-time algorithms exist

The problem with MinCut

It usually outputs segments that are too small!

The problem with MinCut

It usually outputs segments that are too small!Can get small cut just by having fewer cut links

The Normalized Cut (Shi + Malik)

The Normalized Cut (Shi + Malik)

Given a partition (A,B) of the vertex set V.

Ncut(A,B) measures difference between two groups, normalized by how similar they are within themselves.

If A small, assoc(A) is small and Ncut large (bad partition).

Problem: Find partition minimizing Ncut

,

( , ) ( , )( , )( ) ( )

( ) ( , )u A t V

cut A B cut B ANcut A Bassoc A assoc B

assoc A w u t

Matrix formulation

Definitions:

D is an n x n diagonal matrix with entries

W is an n x n symmetric matrix

D(i,i) w(i, j)j

),(),( jiwjiW

Normalized cuts

• Transform the problem to one which can be approximated by eigenvector methods.

• After some algebra, the Ncut problem becomes

subject to the constraints:

yTD10

1iy for some constant 1

( )minT

y t

y D W yMin Ncuty Dy

Normalized cuts

• Transform the problem to one which can be approximated by eigenvector methods.

• After some algebra, the Ncut problem becomes

subject to the constraints:

yTD10

1iy for some constant 1

( )minT

y t

y D W yMin Ncuty Dy

NP-Complete!

Normalized cuts

• Drop discreteness constraints to make easier, solvable by eigenvectors:

Subject to constraint:

yTD10

( )minT

y t

y D W yApproxMin Ncuty Dy

Normalized cuts

• Drop discreteness constraints to make easier, solvable by eigenvectors:

Subject to constraint:

• Solution

y is second least eigenvector of matrix

yTD10

( )minT

y t

y D W yApproxMin Ncuty Dy

1/ 2 1/ 2( )M D D W D

Normalized cuts

• Drop discreteness constraints to make easier, solvable by eigenvectors:

Subject to constraint:

• Solution

y is second least eigenvector of matrix

yTD10

( )minT

y t

y D W yApproxMin Ncuty Dy

1/ 2 1/ 2( )M D D W D

Easy, sinceD is diagonal

Normalized cuts

• Drop discreteness constraints to make easier, solvable by eigenvectors:

Subject to constraint:

• Solution

y is second least eigenvector of matrix

yTD10

( )minT

y t

y D W yApproxMin Ncuty Dy

Because of constraint! (Means eigenvector for second smallest eigenvalue.)

1/ 2 1/ 2( )M D D W D

Normalized cuts

• Drop discreteness constraints to make easier, solvable by eigenvectors:

Subject to constraint:

• Solution

y is second least eigenvector of matrix

yTD10

( )minT

y t

y D W yApproxMin Ncuty Dy

1/ 2 1/ 2( )M D D W D

MATLAB: [U,S,V] = svd(M); y= V(:,end-1);

Normalized Cuts: algorithm

• Define affinities

• Compute matrices: and

• [U,S,V] = svd(M); y= V(:,end-1);

• Threshold y:

(could also use a different threshold besides the mean of y)

1/ 2 1/ 2( )M D D W D

,i jw1/ 2, ,D W D

i{ : mean y }iA i y

i{ : <mean y }iB i y

Normalized Cuts

• Very powerful algorithm, widely used

• Results shown later