Lecture 19: Solving the Correspondence Problem with Graph Cuts

CAP 5415Fall 2006

Announcements

PS4 Only need to search over a limited number of

disparities

Schedule Changes Project Proposals due next Monday Class canceled on Wednesday Will have class on November 22

Stereo Once we have rectified images, the hard problem is

find the corresponding points Then we can triangulate Known as the correspondence problem When people say “stereo algorithm”, they usually

mean “correspondence algorithm”

A very simple algorithm

To find the point that matches this point

A very simple algorithm

Look at a square window around that point

and compare it to windows in the other image

Could measure sum of squared differences (SSD) or correlation

What's wrong with this algorithm?

This set of correspondences don't bother it

Areas of the image without texture will be a problem for this algorithm.

To do better we need a better model of images

We can make reasonable assumptions about the surfaces in the world

Usually assume that the surfaces are smooth Can pose the problem of finding the

corresponding points as an energy (or cost) minimization

The data term measures how well the local windows match up for different disparities

The smoothness cost

The smoothness is usually implemented by penalizing differences in the disparity

Different penalty functions lead to different assumptions about the surfaces log(1+x^2) x^2 First deriv, Second derivative

Today

Assuming that there a discrete set of possible disparities

How do we optimize this?

Today

I'll be talking about the Graph Cuts algorithm Very popular Used heavily in vision and graphics Paper: “Fast Approximate Energy Minimization Notation for the rest of the lecture

Simple Way to Optimize - ICM

Choose a pixel Fix the disparities at the rest of the pixels Set the pixel to the optimal disparity Iterate

ICM

Advantages Energy always guaranteed to decrease Easy to code

Disadvantages Convergence Can you think of the big on?

Making Bigger Moves

The problem with ICM is that you can only modify one pixel at a time

Get stuck in local minima easily We need a way of moving multiple pixels at

once Boykov, Veksler and Zabih introduced two

types of moves: alpha-beta swap alpha-expansion

Types of Moves

ICM – One pixel moves

(From BKZ-PAMI 01)

alpha-beta swap

Fix all nodes that aren't labeled alpha or beta With remaining nodes, find optimal swap

Some alpha nodes change to beta Vice Versa Some stay the same (From BKZ-PAMI 01)

alpha-expansion

Any node can change to alpha

(From BKZ-PAMI 01)

Basic Algorithm

ICM: Compute one-pixel moves until convergence

Graph Cuts: Compute swaps (or expansions) until convergence

Since the optimal swap is being computed each time, energy always decreases

Important Question

How do we find the swaps? Use min-cut from graphs Slightly different from the min-cut problem that

we discussed in the context of normalized cuts

Terminal NodeTerminal Node

Minimum Cut A cut is a set of edges that we will remove so

that the terminal nodes are separated If a cost is assigned to each edge, the cost of

the cut is the weight assigned to each edge Minimum Cut can be found in polynomial time

(Ford and Fulkerson)

Terminal NodeTerminal Node

Important Rule No proper subset of the cut can also be a cut This is not a minimum cut

Terminal NodeTerminal Node

Using the minimum-cut to find a swap

Consider a 1-D image

Terminal Node

Terminal Node

(From BKZ-PAMI 01)

Can guarantee that every node will have at least one t-link

(From BKZ-PAMI 01)

How to see that

(From BKZ-PAMI 01)

Actually Computing the Min-Cut Could use code to solve general min-cut

problems Boykov and Kolmogorov have released an

algorithm optimized for vision problems

Expansion Move

The difference between the optimal solution and the solution from the expansion move is bounded

What functions can be minimized using graph cuts

V is called a metric if it obeys all 3 Can use Expansion Moves

V is called a semimetric if it obeys the last two Can only use Swap-Moves

Subject of recent research.

Graph-Cuts has been shown to perform very well

From “A Comparative Study of Energy Minimization Methods for

Markov Random Fields”

From “A Comparative Study of Energy Minimization Methods for

Markov Random Fields”

Used for much more than just stereo