Embed Size (px)
Citation preview
Graph-Cuts
Michael BleyerLVA Stereo Vision
What happened last time? (1) We have defined an energy function to measure the quality of a
disparity map D:
where- m(p,dp) computes color dissimilarity for matching pixel p at disparity dp
- N denotes all spatial neighbors in 4-connectivity- s() is the smoothness function. We use the Potts model:
This energy function is important for many computer vision problems.
NqpIpp qpsdpmDE
,),(),()(
),( qps0 if dp = dq P otherwise.
2
What happened last time? (2) Smoothness interactions define a
graph known as 4-connected grid. Computing the energy optimum on the
4-connected grid is an np-complete problem.
We have learned about dynamic programming:• Computes exact energy optimum
• Requires the graph to be a tree=> We had to remove smoothness interactions
4-Connected Grid
3
What is Going to Happen Today?
Just one point on the agenda:
• Graph-Cuts
4
What is Graph-Cuts? Powerful optimization method. Finds strong local minima of our np-complete energy function. Graph-cuts have been around in computer vision for quite some
time (e.g. [Roy,ICCV98]). I will speak about modern graph-cuts, i.e. move making
algorithms
5
Move Making Algorithms We are given a labeled image as
input. (In our case, the image is labeled
with disparity values, i.e. label α can for example mean a disparity of 10 pixels.)
We want to modify the assignment of pixels to labels to obtain a better solution, i.e. one of lower energy.
An operation that changes labels is called a move.
We will learn about 3 types of moves:• αβ-swap
• α-expansion
• fusion move
βγ
α
Current labeling
New labeling (preferably of lower energy than current
labeling)
Move
6
αβ-Swap [Boykov,PAMI01]
Select two labels: α and β. A pixel that is assigned to α in the
current labeling can either:1. switch its label to β or
2. keep its old label α in the new labeling.
Analogously, a pixel that is currently assigned to β can either:1. switch its label to α or
2. keep its old label β in the new labeling.
Simply spoken:• Some pixels that had the label α are now
assigned to β.
• Some pixels that had the label β are now assigned to α.
βγ
α
βγ
α
Current labeling
One possible labeling after αβ-swap 7
α-Expansion [Boykov,PAMI01]
Select one label: α. Any pixel can either
1. switch its label to α or
2. keep its old label.
More global than αβ-swap:• All pixels can change their labels
simultaneously.
In experiments, α-expansion moves typically outperform αβ-swaps.
We will therefore concentrate on α-expansions.
βγ
α
Current labeling
One possible labeling after α-expansion
βγ
α
8
The Key Problem
There is an extremely large number of possible α-expansions. The key challenge is to find the “best” α-expansion, i.e. the one that leads to
the largest decrease of our energy. Good news:
• For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph.
βγ
α
Current labeling
βγ
α
βγ
α
βγ
α
α
α-exp 1
α-ex
p 4
α-ex
p 2
α-exp 3
9
The Key Problem
There is an extremely large number of possible α-expansions. The key challenge is to find the “best” α-expansion, i.e. the one that leads to
the largest decrease of our energy. Good news:
• For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph.
βγ
α
Current labeling
βγ
α
βγ
α
α
α-exp 1
α-ex
p 4
α-ex
p 2
α-exp 3
10
E = 1000
E = 2000
E = 500 E = 750We should take
this one
E = 1000
βγ
α
The Key Problem
There is an extremely large number of possible α-expansions. The key challenge is to find the “best” α-expansion, i.e. the one that leads to
the largest decrease of our energy. Good news:
• For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph.
βγ
α
Current labeling
βγ
α
βγ
α
βγ
α
α
α-exp 1
α-ex
p 4
α-ex
p 2
α-exp 3
11
Iterative Algorithm – α-Expansion
Let us for now assume that we know how to compute the optimal α-expansion.
We can incorporate the α-expansion as follows.
Iterative Algorithm: Start with an arbitrary labeling f. Loop (e.g. 3 times)
• For each allowed label α:- Find f* = argmin E(f’) among f’ within one α-expansion of f- f := f*
Comment:• Note that we compute the optimal α-expansion.
• Therefore, the energy will either decrease after α-expansion or stay the same (not changing the labeling at all is a feasible α-expansion).
• The algorithm will in any case converge to a (strong) local energy optimum. 12
Iterative Algorithm – Example Video
(α-expansions for stereo matching)
13
Computing the Optimal α-Expansion
There are 3 things you have to do to find the optimal α-expansion via graph-cuts:1. Write your energy as a pseudo-boolean function
2. Construct a graph that represents your boolean function
3. Compute the Minimum Cut in this graph
These steps are discussed in the following.
14
Writing the Energy as a Pseudo-Boolean Function (1)
We associate a boolean variable xp with each pixel p where:• xp = 0 means that pixel p keeps its old label after α-expansion
• xp = 1 means that pixel p takes label α after α-expansion For example, if this is the current labeling:
then x =
leads to the label configuration:
after α-expansion. We can represent all possible α-expansions by the boolean
variables x.
β β β γ γ
1 1 0 0 1
α α β γ α
15
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: β β
16
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
NqpIpp qpsdpmDE
,),(),()(
β β
Dissimilarity function
Potts model (Impose penalty P if p and q have
different labels)
17
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
β β
18
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
β β
19
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
β β
20
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
β β
We call these terms unary terms, since they depend on one variable.
We call this term a pairwise term, since it
depends on two variables.21
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
where:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
β β
22
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
where:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
β β
23
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixels are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
where:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
E β β
β α
α β
α α
β β
24
Writing the Energy as a Pseudo-Boolean Function (2)
Let us assume we have two pixels p and q. Both pixel are assigned to label β in the current labeling: Recall our energy function:
We can write our energy as a function of binary variables xp and xq:
where:
NqpIpp qpsdpmDE
,),(),()(
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
E β β
β α
α β
α α
β βWe have to find the settings of binary
variables xp and xq that minimize the energy.
This comes next.
25
The Min-Cut Problem We have two dedicated
nodes, the source and the sink.
source
sink
Example taken from Pushmeet Kohli’s ICCV09 tutorial
p q
2 9
2
1
5 4
26
The Min-Cut Problem We have two dedicated
nodes, the source and the sink.
We partition the graph into two sets S and T where . sink and source TS
source
sink
Example taken from Pushmeet Kohli’s ICCV09 tutorial
p q
2 9
2
1
5 4
S
T
27
The Min-Cut Problem We have two dedicated
nodes, the source and the sink.
We partition the graph into two sets S and T where
The cut consists of all edges that lead from S to T.
. sink and source TS
source
sink
Example taken from Pushmeet Kohli’s ICCV09 tutorial
p q
2 9
2
1
5 4
S
T
28
The Min-Cut Problem We have two dedicated
nodes, the source and the sink.
We partition the graph into two sets S and T where
The cut consists of all edges that lead from S to T.
The costs of a cut are the sum of weights of these edges.
. sink and source TS
source
sink
Example taken from Pushmeet Kohli’s ICCV09 tutorial
p q
2 9
2
1
5 4
S
T
Costs: 5 + 2 + 9 = 16
29
The Min-Cut Problem We have two dedicated
nodes, the source and the sink.
We partition the graph into two sets S and T where
The cut consists of all edges that lead from S to T.
The costs of a cut are the sum of weights of these edges.
The minimum cut is the cut of minimum costs among all possible cuts.
. sink and source TS
source
sink
Example taken from Pushmeet Kohli’s ICCV09 tutorial
p q
2 9
2
1
5 4
S
T
Costs: 2 + 1 + 4 = 7
30
The Min-Cut Problem
The min-cut problem has been extensively studied in graph theory.
There exists a variety of algorithms that1. Can find the exact solution
2. Are computationally very fast.
Side notes:• The min-cut problem and the max-flow problem are dual problems:
=> Solving min-cut also gives the solution for max-flow and vice versa.
• Max-flow and min-cut are therefore often used synonymously.
• If you are interested in algorithms for computing min-cut/max-flow:- Read [Boykov,PAMI04]
31
The Min-Cut Problem
The min-cut problem has been extensively studied in graph theory.
There exists a variety of algorithms that1. Can find the exact solution
2. Are computationally very fast.
Side notes:• The min-cut problem and the max-flow problem are dual problems:
=> Solving min-cut also gives the solution for max-flow and vice versa.
• Max-flow and min-cut are therefore often used synonymously.
• If you are interested in algorithms for computing min-cut/max-flow:- Read [Boykov,PAMI04]
Nice, but does this help us to optimize our pseudo-boolean
function?
32
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel.source
sink
p q
33
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel. If a node p is member of S
after the cut, then xp = 0.
source
sink
p q
S=> xp = 0
34
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel. If a node p is member of S
after the cut, then xp = 0. If p is member of T, then xp = 1
source
sink
p q
S=> xp = 0
T
=> xq = 1
35
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel. If a node p is member of S
after the cut, then xp = 0. If p is member of T, then xp = 1 We adjust the edges so that
the costs of the cut are equal to the energy of our binary variables x.
source
sink
p q
S=> xp = 0
T
=> xq = 1
The costs of this cut have to be equal to the energy
of xp = 0 and xq = 1. 36
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel. If a node p is member of S
after the cut, then xp = 0. If p is member of T, then xp = 1 We adjust the edges so that
the costs of the cut are equal to the energy of our binary variables x.
The minimum cut therefore also represents the minimum of our energy.
source
sink
p q
S=> xp = 0
T
=> xq = 1
37
The costs of this cut have to be equal to the energy
of xp = 0 and xq = 1.
The costs of this cut have to be equal to the energy
of xp = 0 and xq = 1.
Optimization of our Pseudo-Boolean Function We insert a node for each
pixel. If a node p is member of S
after the cut, then xp = 0. If p is member of T, then xp = 1 We adjust the edges so that
the costs of the cut are equal to the energy of our binary variables x.
The minimum cut therefore also represents the minimum of our energy.
source
sink
p q
S=> xp = 0
T
=> xq = 1
How can we do this for our example?
38
Optimization of our Pseudo-Boolean Function Our unary terms:
Our pairwise term:
source
p q
sink
39
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
m(p,β)
m(p,α)
40
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
m(p,β)
m(p,α)
m(q,β)
m(q,α)
41
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
42
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Let us check whether this graph really represents
our energy.
43
S
=> xp = 0, xq = 0
T
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Cut Costs:C = m(p,β)+m(q,β)
Energy: E(0,1) = Ep(0)+Eq(0)+Ep,q(0,0)
= m(p,β)+m(q,β)+0 44
S
T
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Cut Costs:C = m(p,β)+m(q,α)+P
Energy: E(0,1) = Ep(0)+Eq(1)+Ep,q(0,1)
= m(p,β)+m(q,α)+P
=> xp = 0, xq = 1
45
S
T
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Cut Costs:C = m(p,α)+m(q,β)+P
Energy: E(0,1) = Ep(1)+Eq(0)+Ep,q(1,0)
= m(p,α)+m(q,β)+P
=> xp = 1, xq = 0
46
S
=> xp = 1, xq = 1
T
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Cut Costs:C = m(p,α)+m(q,α)
Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1)
= m(p,α)+m(q,α)+0 47
S
=> xp = 1, xq = 1
T
Optimization of our Pseudo-Boolean Function Our unary terms:
•
Our pairwise term:•
source
p q
sink
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
0)1,1(
)0,1(
)1,0(
0)0,0(
,
,
,
,
qp
qp
qp
qp
E
PE
PE
Em(p,β)
m(p,α)
m(q,β)
m(q,α)
P
P
Cut Costs:C = m(p,α)+m(q,α)
Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1)
= m(p,α)+m(q,α)+0
We have shown that the graph represents our
energy.
48
What Energy Function Can be Optimized via Graph-Cuts?
Not every boolean energy function can be represented by a graph! The pairwise terms have to fulfill the following constraint
[Kolmogorov,PAMI04]:
In our example, this has been the case:
If there is at least one pairwise term in the boolean energy function that violates this constraint, the energy is said to be non-submodular.
Otherwise, it is called submodular. Optimizing non-submodular energies is an np-complete problem.
=> Computing the optimal α-expansion becomes very difficult. (but not impossible)
)0,1()1,0()1,1()0,0( ,,,, qpqpqpqp EEEE
PP 00
49
Max-Flow/Min-Cut Library
“I would like to use graph-cuts, but I do not want to mess around with graphs.”
Good news:
• You don’t have to. It is sufficient to define your energy as a pseudo-boolean function. You can then download the Max-Flow/Min-Cut library from
http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software.html The library will:
Construct the graph that represents your boolean function Compute the min-cut Provide you the optimal labeling
See example on next slide.
50
Example Code for the Max-Flow/Min-Cut Library
// Set up graph and add 2 nodes
1. Graph *g = new Graph();
2. int p = g->AddNode();
3. int q = g->AddNode();
// Define boolean energy
4. g->AddUnaryTerm(p, Ep(0), Ep(1));
5. g->AddUnaryTerm(q, Eq(0), Eq(1));
6. g->AddPairwiseTerm(p, q, Ep,q(0,0), Ep,q(0,1), Ep,q(1,0), Ep,q(1,1));
// Construct graph that represents the energy// Compute min-cut
7. g->Solve();
// Write optimal labels
8. printf (“optimal label p %d”, g->GetLabel(p));
9. printf (“optimal label q %d”, g->GetLabel(q));
51
The Fusion Move [Lemptisky,ICCV07]
Two proposals are fused to obtain a new solution of lower energy.
Fusion Move:• Let fp denote pixel p’s label
in proposal 1.
• Let gp denote p’s label in proposal 2.
• After fusion p is either assigned to fp or gp.
α-expansion is a special case of a fusion move where the second proposal contains only a single label.
One possible labeling after fusion of proposals 1 and 2
Proposal 1 Proposal 2
52
Iterative Algorithm – Fusion Moves
Iterative Algorithm: Start with an arbitrary labeling f. For each proposal g:
- Find f* = argmin E(f’) among f’ being one possible fusion of f and g.
- f := f*
53
Iterative Algorithm – Example Video
(Fusion moves for stereo matching)
54
Why Fusion Moves? (1) Parallelization:
• Parallel implementations of Min-Cut algorithms are very difficult to accomplish.
We can do the following parallel implementation:• CPU1 computes α-
expansions for disparities 0-8
• CPU2 computes α-expansions for disparities 9-16
• The results of both CPUs are then fused
Fusion Move
55
Why Fusion Moves? (2) You have two algorithms
that have different failure modes.
Opical flow example:1. Horn-Schunck:
- works well in untextured regions- fails at flow borders
2. Lucas-Kanade:- fails in untextured regions- works well at flow borders
• We run both algorithms and fuse their results
• Fusion move will - pick Horn-Schunk result for
untextured regions- pick Lucas-Kanade result at flow
borders- => much better result
• Can even work in real-time
First frame Ground truth optical flow
Result – Horn-Schunck algo. Result – Lucas-Kanade algo.
Fusion Move
Fusion of both algorithms56
Why Fusion Moves? (3) α-expansions will become intractable if there is a very large or
infinite label set. For example:
• Large resolution stereo:- You might need to test > 1000 disparity labels
• Optical flow:- The space of all possible discrete flow vectors is very large (2 dimensions)
• Assigning pixels to continuous disparity values:- The set of all continuous disparities is of infinite size
• Assigning pixels to 3D surfaces:- There is an infinite amount of 3D surfaces.- You will hear more about surface stereo in a different session.
Fusion moves can handle all of these cases! Probably the most important argument.
57
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
α β γ αProposal 1 Proposal 2
58
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
α β γ αProposal 1 Proposal 2
59
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
As before, we write our energy as a function of binary variables xp and xq:
where:
),()()(),( , qpqpqqppqp xxExExExxE
α β γ αProposal 1 Proposal 2
60
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
As before, we write our energy as a function of binary variables xp and xq:
where:
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
α β γ αProposal 1 Proposal 2
61
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
As before, we write our energy as a function of binary variables xp and xq:
where:
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
α β γ αProposal 1 Proposal 2
62
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
As before, we write our energy as a function of binary variables xp and xq:
where:
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
PE
PE
E
PE
qp
qp
qp
qp
)1,1(
)0,1(
0)1,0(
)0,0(
,
,
,
, α β
α α
γ β
γ α
α β γ αProposal 1 Proposal 2
63
Writing the Boolean Fusion Move Energy Let us assume we have two pixels p and q. Our 2 proposals have the following labeling:
This time xp has a different meaning:• xp = 0, if p takes the label of proposal 1
• xp = 1, if p takes the label of proposal 2
As before, we write our energy as a function of binary variables xp and xq:
where:
),()()(),( , qpqpqqppqp xxExExExxE
),()1(
),()0(
pmE
pmE
p
p
),()1(
),()0(
qmE
qmE
q
q
PE
PE
E
PE
qp
qp
qp
qp
)1,1(
)0,1(
0)1,0(
)0,0(
,
,
,
, α β
α α
γ β
γ α
α β γ αProposal 1 Proposal 2Can you see a problem
here?
64
The Fusion Energy Can Be Non-Submodular
Remember the condition for sub-modularity:
Our example energy is non-submodular:
Finding the optimal fusion move is, in general, an np-complete problem • That is actually the reason why fusion moves have not been used before 2007.
Good news:• Nowadays there exist powerful graph-cut-based optimization algorithms that
can handle non-submodular energies.
• In particular, I mean Quadratic Pseudo Boolean Optimization (QPBO)
)0,1()1,0()1,1()0,0( ,,,, qpqpqpqp EEEE
PPP 0
65
Quadratic Pseudo Boolean Optimization (QPBO) [Kolmogorov,PAMI07]
QPBO can only compute a part of the global optimal solution This means
• Instead of a complete labeling such as- xp = 0, xq = 1, xr = 0
QPBO will in general provide an incomplete labeling such as- xp = 0, xq = ø, xr = ø
where ø means “unknown”.
• Those pixel whose label ≠ ø would also have this label in the “complete” global optimal solution.
Proposal 1 Proposal 2 Fused Result (computed via QPBOI)
Pixels labeled as unknown by QPBO are shown in
black 66
What to do with pixels labeled as unknown?
Autarky property of QPBO:• If you assign all unknown pixels to label 0, the energy is guaranteed to be lower
or equal to the labeling <0,0,0,…,0>.
• In case of a fusion move, this means that assigning unknown pixels to the labels of proposal 1 will lead to a lower or equal energy than that of proposal 1.
• Assigning unknown pixels to label 0 is known as QPBOF.
You can do more [Rother, CVPR07]:• QPBOI (I stands for Improve):
- Tries to improve QPBOF solution.
• QPBOP (P stands for Probe):- Tries to find more pixels of the global optimal solution.
You can download QPBO:• http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software.html
• Also includes QPBOI and QPBOP
• Interface almost identical to MaxFlow Library.
67
Corrected Iterative AlgorithmI have cheated in the definition of the iterative algorithm.
Iterative Algorithm: Start with an arbitrary labeling f. For each proposal g:
- Find f* = argmin E(f’) among f’ being one possible fusion of f and g.
- f := f*
In the general case, we cannot really compute the global optimal fusion move (np-complete problem).
We just find a “good” one. The energy of f* is guaranteed to be equal or lower than that of f.
(autarky property of QPBO). The iterative algorithm will therefore converge to a local energy
minimum.68
Summary Move making algorithms α-expansions:
• Iterative algorithm
• Computing the optimal α-expansion
Sub-modularity condition Fusion moves:
• Handle large label spaces
• Computing a “good” fusion move
QPBO
69
References [Boykov,PAMI01] Y. Boykov, O. Veksler, R. Zabih, Fast Approximate
Energy Minimization via Graph Cuts, PAMI, vol. 23, no. 11, pp. 1222-1239, 2001.
[Boykov,PAMI04] Y. Boykov, V. Kolmogorov, An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision. PAMI, vol. 26, no. 9, pp. 1124-1137, 2004.
[Kolmogorov,PAMI07] V. Kolmogorov, C. Rother, Minimizing Nonsubmodular Functions with Graph Cuts-A Review, PAMI, vol. 29, no. 7, pp. 1274-1279, 2007.
[Lempitsky,ICCV07] V. Lempitsky, C. Rother, A. Blake, LogCut - Efficient Graph Cut Optimization for Markov Random Fields, ICCV 2007.
[Rother,CVPR07] C. Rother, V. Kolmogorov, V. Lempitsky, M. Szummer, Optimizing Binary MRFs Via Extended Roof Duality, CVPR 2007.
[Roy,ICCV98] S. Roy, I. Cox, A Maximum-Flow Formulation of the N-Camera Stereo Correspondence Problem“, ICCV 1998.
70
