View
477
Download
1
Category
Tags:
Preview:
Citation preview
MAP Estimation Algorithms in
M. Pawan Kumar, University of Oxford
Pushmeet Kohli, Microsoft Research
Computer Vision - Part II
Example: Image Segmentation
E(x) = ∑ ci xi + ∑ cij xi(1-xj)
E: {0,1}n → R0 → fg 1 → bg
Image (D)
i i,jn = number of pixels
Example: Image Segmentation
E(x) = ∑ ci xi + ∑ cij xi(1-xj)
E: {0,1}n → R0 → fg 1 → bg
i i,j
Unary Cost (ci)
Dark (negative) Bright (positive)
n = number of pixels
Example: Image Segmentation
E(x) = ∑ ci xi + ∑ cij xi(1-xj)
E: {0,1}n → R0 → fg 1 → bg
i i,j
Discontinuity Cost (cij)
n = number of pixels
Example: Image Segmentation
E(x) = ∑ ci xi + ∑ cij xi(1-xj)
E: {0,1}n → R0 → fg 1 → bg
i i,j
Global Minimum (x*)
x* = arg min
E(x) x
How to minimize E(x)?
n = number of pixels
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Recent Advances and Open Problems
Connection between st-mincut and energy
minimization?
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Connection between st-mincut and energy
minimization?
Recent Advances and Open Problems
The st-Mincut Problem
Source
Sink
v1 v2
2
5
9
42
1
Graph (V, E, C)Vertices V = {v1, v2 ... vn}
Edges E = {(v1, v2) ....}
Costs C = {c(1, 2) ....}
The st-Mincut Problem
Source
Sink
v1 v2
2
5
9
42
1
What is a st-cut?
The st-Mincut Problem
Source
Sink
v1 v2
2
5
9
42
1
What is a st-cut?
An st-cut (S,T) divides the nodes between source and sink.
What is the cost of a st-cut?
Sum of cost of all edges going from S to T
5 + 2 + 9 = 16
The st-Mincut Problem
What is a st-cut?
An st-cut (S,T) divides the nodes between source and sink.
What is the cost of a st-cut?
Sum of cost of all edges going from S to T
What is the st-mincut?
st-cut with the minimum cost
Source
Sink
v1 v2
2
5
9
42
1
2 + 1 + 4 = 7
How to compute the st-mincut?
Source
Sink
v1 v2
2
5
9
42
1
Solve the dual maximum flow problem
In every network, the maximum flow equals the cost of the st-
mincut
Min-cut\Max-flow Theorem
Compute the maximum flow between Source and Sink
Constraints
Edges: Flow < Capacity
Nodes: Flow in = Flow out
Maxflow Algorithms
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Source
Sink
v1 v2
2
5
9
42
1
Algorithms assume non-negative capacity
Flow = 0
Maxflow Algorithms
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Source
Sink
v1 v2
2
5
9
42
1
Algorithms assume non-negative capacity
Flow = 0
Maxflow Algorithms
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Source
Sink
v1 v2
2-2
5-2
9
42
1
Algorithms assume non-negative capacity
Flow = 0 + 2
Maxflow Algorithms
Source
Sink
v1 v2
0
3
9
42
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 2
Maxflow Algorithms
Source
Sink
v1 v2
0
3
9
42
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 2
Maxflow Algorithms
Source
Sink
v1 v2
0
3
9
42
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 2
Maxflow Algorithms
Source
Sink
v1 v2
0
3
5
02
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 2 + 4
Maxflow Algorithms
Source
Sink
v1 v2
0
3
5
02
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 6
Maxflow Algorithms
Source
Sink
v1 v2
0
3
5
02
1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 6
Maxflow Algorithms
Source
Sink
v1 v2
0
2
4
02+1
1-1
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 6 + 1
Maxflow Algorithms
Source
Sink
v1 v2
0
2
4
03
0
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 7
Maxflow Algorithms
Source
Sink
v1 v2
0
2
4
03
0
Augmenting Path Based Algorithms
1. Find path from source to sink with positive capacity
2. Push maximum possible flow through this path
3. Repeat until no path can be found
Algorithms assume non-negative capacity
Flow = 7
History of Maxflow Algorithms
[Slide credit: Andrew Goldberg]
Augmenting Path and Push-Relabel
n: #nodes
m: #edges
U: maximum edge weight
Algorithms assume non-negative edge
weights
History of Maxflow Algorithms
[Slide credit: Andrew Goldberg]
Augmenting Path and Push-Relabel
n: #nodes
m: #edges
U: maximum edge weight
Algorithms assume non-negative edge
weights
Augmenting Path based Algorithms
a1 a2
1000 1
Sink
Source
1000
1000
1000
0
Ford Fulkerson: Choose any augmenting path
a1 a2
1000 1
Sink
Source
1000
1000
1000
0
Augmenting Path based Algorithms
Bad Augmenting
Paths
Ford Fulkerson: Choose any augmenting path
a1 a2
1000 1
Sink
Source
1000
1000
1000
0
Augmenting Path based Algorithms
Bad Augmenting
Path
Ford Fulkerson: Choose any augmenting path
a1 a2
9990
Sink
Source
1000
1000
9991
Augmenting Path based Algorithms
Ford Fulkerson: Choose any augmenting path
a1 a2
9990
Sink
Source
1000
1000
9991
Ford Fulkerson: Choose any augmenting path
n: #nodes
m: #edges
We will have to perform 2000 augmentations!
Worst case complexity: O (m x Total_Flow)
(Pseudo-polynomial bound: depends on flow)
Augmenting Path based Algorithms
Dinic: Choose shortest augmenting path
n: #nodes
m: #edges
Worst case Complexity: O (m n2)
Augmenting Path based Algorithms
a1 a2
1000 1
Sink
Source
1000
1000
1000
0
Maxflow in Computer Vision
• Specialized algorithms for vision problems– Grid graphs – Low connectivity (m ~ O(n))
• Dual search tree augmenting path algorithm[Boykov and Kolmogorov PAMI 2004]• Finds approximate shortest
augmenting paths efficiently• High worst-case time complexity• Empirically outperforms other
algorithms on vision problems
Maxflow in Computer Vision
• Specialized algorithms for vision problems– Grid graphs – Low connectivity (m ~ O(n))
• Dual search tree augmenting path algorithm[Boykov and Kolmogorov PAMI 2004]• Finds approximate shortest
augmenting paths efficiently• High worst-case time complexity• Empirically outperforms other
algorithms on vision problems• Efficient code available on the
webhttp://www.adastral.ucl.ac.uk/~vladkolm/software.html
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Connection between st-mincut and energy
minimization?
Recent Advances and Open Problems
St-mincut and Energy Minimization
T
S st-mincut
E: {0,1}n → R
Minimizing a Qudratic
Pseudoboolean function E(x)
Functions of boolean variables
Pseudoboolean?
Polynomial time st-mincut algorithms require non-negative
edge weights
E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0i,ji
So how does this work?
Construct a graph such that:
1.Any st-cut corresponds to an assignment of x
2.The cost of the cut is equal to the energy of x : E(x)
SolutionT
S st-mincut
E(x)
Graph Construction
Sink (1)
Source (0)
a1 a2
E(a1,a2)
Graph Construction
Sink (1)
Source (0)
a1 a2
E(a1,a2) = 2a1
2
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1
2
5
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2
2
5
9
4
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2
2
5
9
4
2
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1a1 = 1 a2 = 1
E (1,1) = 11
Cost of cut = 11
Sink (1)
Source (0)
Graph Construction
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1
Sink (1)
Source (0)
a1 = 1 a2 = 0
E (1,0) = 8
st-mincut cost = 8
Energy Function Reparameterization
Two functions E1 and E2 are reparameterizations if
E1 (x) = E2 (x) for all x
For instance:
E1 (a1) = 1+ 2a1 + 3ā1
E2 (a1) = 3 + ā1
a1 ā1 1+ 2a1 + 3ā1
3 + ā1
0 1 4 4
1 0 3 3
Flow and Reparametrization
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1
Sink (1)
Source (0)
Flow and Reparametrization
a1 a2
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
2
5
9
4
2
1
Sink (1)
Source (0)
2a1 + 5ā1
= 2(a1+ā1) + 3ā1
= 2 + 3ā1
Flow and Reparametrization
Sink (1)
Source (0)
a1 a2
E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
0
3
9
4
2
12a1 + 5ā1
= 2(a1+ā1) + 3ā1
= 2 + 3ā1
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
0
3
9
4
2
1
Flow and Reparametrization
9a2 + 4ā2
= 4(a2+ā2) + 5ā2
= 4 + 5ā2
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2
0
3
5
0
2
19a2 + 4ā2
= 4(a2+ā2) + 5ā2
= 4 + 5ā2
Flow and Reparametrization
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2
0
3
5
0
2
1
Flow and Reparametrization
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2
0
3
5
0
2
1
Flow and Reparametrization
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2
0
3
5
0
2
1
Flow and Reparametrization
3ā1+ 5a2 + 2a1ā2
= 2(ā1+a2+a1ā2) +ā1+3a2= 2(1+ā1a2) +ā1+3a2
a1 a2 F1 F2
0 0 1 1
0 1 2 2
1 0 1 1
1 1 1 1
F1 = ā1+a2+a1ā2
F2 = 1+ā1a2
Sink (0)
Source (1)
a1 a2
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
0
1
3
0
0
3
Flow and Reparametrization
3ā1+ 5a2 + 2a1ā2
= 2(ā1+a2+a1ā2) +ā1+3a2= 2(1+ā1a2) +ā1+3a2
a1 a2 F1 F2
0 0 1 1
0 1 2 2
1 0 1 1
1 1 1 1
F1 = ā1+a2+a1ā2
F2 = 1+ā1a2
Sink (0)
Source (1)
a1 a2
0
1
3
0
0
3
Flow and Reparametrization
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
No more augmenting paths
possible
Sink (0)
Source (1)
a1 a2
0
1
3
0
0
3
Flow and Reparametrization
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
Total Flow
Residual Graph(positive
coefficients)
bound on the optimal solution
Inference of the optimal solution becomes trivial because the bound is
tight
Sink (0)
Source (1)
a1 a2
0
1
3
0
0
3
Flow and Reparametrization
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
a1 = 1 a2 = 0
E (1,0) = 8
st-mincut cost = 8Total Flow
bound on the optimal solution
Inference of the optimal solution becomes trivial because the bound is
tight
Residual Graph(positive
coefficients)
Example: Image Segmentation
E(x) = ∑ ci xi + ∑ cij xi(1-xj)
E: {0,1}n → R0 → fg 1 → bg
i i,j
Global Minimum (x*)
x* = arg min
E(x) x
How to minimize E(x)?
How does the code look like?
Sink (1)
Source (0)
Graph *g;
For all pixels p
/* Add a node to the graph */nodeID(p) = g->add_node();
/* Set cost of terminal edges */set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost);
end
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
How does the code look like?
Graph *g;
For all pixels p
/* Add a node to the graph */nodeID(p) = g->add_node();
/* Set cost of terminal edges */set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost);
end
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
a1 a2
fgCost(a1)
Sink (1)
Source (0)
fgCost(a2)
bgCost(a1) bgCost(a2)
Graph *g;
For all pixels p
/* Add a node to the graph */nodeID(p) = g->add_node();
/* Set cost of terminal edges */set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost(p,q));
end
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
How does the code look like?
a1 a2
fgCost(a1)
Sink (1)
Source (0)
fgCost(a2)
bgCost(a1) bgCost(a2)
cost(p,q)
Graph *g;
For all pixels p
/* Add a node to the graph */nodeID(p) = g->add_node();
/* Set cost of terminal edges */set_weights(nodeID(p), fgCost(p), bgCost(p));
end
for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost(p,q));
end
g->compute_maxflow();
label_p = g->is_connected_to_source(nodeID(p));
// is the label of pixel p (0 or 1)
How does the code look like?
a1 a2
fgCost(a1)
Sink (1)
Source (0)
fgCost(a2)
bgCost(a1) bgCost(a2)
cost(p,q)
a1 = bg a2 = fg
Image Segmentation in Video
Image Flow Global Optimum
pqw
n-links st-cuts
t
= 0
= 1
E(x) x*
Image Segmentation in Video
Image
FlowGlobal Optimu
m
Dynamic Energy Minimization
EB SBcomputationally
expensive operation
EA SA
minimize
Recycling
Solutions
Can we do better?
Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)
Dynamic Energy Minimization
EB SBcomputationally
expensive operation
EA SA
minimize
cheaperoperation
Simplerenergy
EB*
differencesbetweenA and B
A and Bsimilar
Kohli & Torr (ICCV05, PAMI07)
3 – 100000 time
speedup!
Reparametrization
Reuse flow
Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)
Reparametrized Energy
Dynamic Energy Minimization
Kohli & Torr (ICCV05, PAMI07)
Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
Original Energy
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2
New Energy
New Reparametrized Energy
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Connection between st-mincut and energy
minimization?
Recent Advances and Open Problems
Minimizing Energy Functions
Space of Function Minimization
Problems
Submodular
Functions
NP-Hard
• General Energy Functions– NP-hard to minimize– Only approximate
minimization possible
• Easy energy functions – Solvable in polynomial time– Submodular ~ O(n6)
MAXCUT
Functions defined on trees
Submodular Set Functions
Set function f 2|E| ℝ
2|E| = #subsets of E
Let E= {a1,a2, .... an} be a set
Submodular Set Functions
Set function f 2|E| ℝ is submodular if
E
A B
f(A) + f(B) f(AB) + f(AB) for all A,B E
2|E| = #subsets of E
Let E= {a1,a2, .... an} be a set
Important Property
Sum of two submodular functions is submodular
Minimizing Submodular Functions
• Minimizing general submodular functions– O(n5 Q + n6) where Q is function evaluation time
[Orlin, IPCO 2007]
• Symmetric submodular functions– E (x) = E (1 - x)– O(n3) [Queyranne 1998]
• Quadratic pseudoboolean– Can be transformed to st-mincut– One node per variable [ O(n3) complexity]– Very low empirical running time
Submodular Pseudoboolean Functions
• All functions for one boolean variable (f: {0,1} -> R) are submodular
• A function of two boolean variables (f: {0,1}2 -> R) is submodular if
f(0,1) + f(1,0) f(0,0) + f(1,1)
• A general pseudoboolean function f 2n ℝ is submodular if all its projections fp are submodular i.e.fp(0,1) + fp (1,0) fp (0,0) + fp
(1,1)
Function defined over boolean vectors x = {x1,x2, .... xn}
Definition:
E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji
Quadratic Submodular Pseudoboolean Functions
θij(0,1) + θij (1,0) θij
(0,0) + θij
(1,1)For all ij
E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji
Quadratic Submodular Pseudoboolean Functions
θij(0,1) + θij (1,0) θij
(0,0) + θij
(1,1)For all ij
E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0i,ji
Equivalent (transformable)
i.e. All submodular QPBFs are st-mincut solvable
A B
C D
0 1
0
1
xi
xj
= A +0 0
C-A C-A
0 1
0
1
0 D-C
0 D-C
0 1
0
1
0B+C-A-D
0 0
0 1
0
1
+ +
if x1=1 add
C-A
if x2 = 1 add
D-C
B+C-A-D 0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij
(1,0) D = θij
(1,1)
θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj
A B
C D
0 1
0
1
xi
xj
= A +0 0
C-A C-A
0 1
0
1
0 D-C
0 D-C
0 1
0
1
0B+C-A-D
0 0
0 1
0
1
+ +
if x1=1 add
C-A
if x2 = 1 add
D-C
B+C-A-D 0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij
(1,0) D = θij
(1,1)
θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj
A B
C D
0 1
0
1
xi
xj
= A +0 0
C-A C-A
0 1
0
1
0 D-C
0 D-C
0 1
0
1
0B+C-A-D
0 0
0 1
0
1
+ +
if x1=1 add
C-A
if x2 = 1 add
D-C
B+C-A-D 0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij
(1,0) D = θij
(1,1)
θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj
A B
C D
0 1
0
1
xi
xj
= A +0 0
C-A C-A
0 1
0
1
0 D-C
0 D-C
0 1
0
1
0B+C-A-D
0 0
0 1
0
1
+ +
if x1=1 add
C-A
if x2 = 1 add
D-C
B+C-A-D 0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij
(1,0) D = θij
(1,1)
θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj
A B
C D
0 1
0
1
xi
xj
= A +0 0
C-A C-A
0 1
0
1
0 D-C
0 D-C
0 1
0
1
0B+C-A-D
0 0
0 1
0
1
+ +
if x1=1 add
C-A
if x2 = 1 add
D-C
B+C-A-D 0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0) B = θij(0,1) C = θij
(1,0) D = θij
(1,1)
θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj
E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji
Quadratic Submodular Pseudoboolean Functions
θij(0,1) + θij (1,0) θij
(0,0) + θij
(1,1)For all ij
Equivalent (transformable)
T
Sst-mincut
x in {0,1}n
Minimizing Non-Submodular Functions
• Minimizing general non-submodular functions is NP-hard.
• Commonly used method is to solve a relaxation of the problem
E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji
θij(0,1) + θij (1,0) ≤ θij
(0,0) + θij (1,1) for some ij
[Slide credit: Carsten Rother]
pairwise nonsubmodular
unary
pairwise submodular
Minimization using Roof-dual Relaxation
)0,1()1,0()1,1()0,0( pqpqpqpq
)0,1(~
)1,0(~
)1,1(~
)0,0(~
pqpqpqpq
),(~
),(
)(})({
qppq
qppq
ppp
xx
xx
xxE
[Slide credit: Carsten Rother]
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({'
qppqqppq
qppqqppq
pppppp
xxxx
xxxx
xxxxE
Double number of variables:
),(~
),(
)(})({
qppq
qppq
ppp
xx
xx
xxE
ppp xxx ,
Minimization using Roof-dual Relaxation
)1( pp xx
[Slide credit: Carsten Rother]
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({'
qppqqppq
qppqqppq
pppppp
xxxx
xxxx
xxxxE
Double number of variables:
),(~
),(
)(})({
qppq
qppq
ppp
xx
xx
xxE
ppp xxx ,
Minimization using Roof-dual Relaxation
)1,(~
),( qppqqppq xxxxf
)0,1(~
)1,0(~
)1,1(~
)0,0(~
pqpqpqpq )1,1()0,0()0,1()1,0( pqpqpqpq ffff
Non- submodular Submodular
)1( pp xx
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({'
qppqqppq
qppqqppq
pppppp
xxxx
xxxx
xxxxE
Double number of variables:
),(~
),(
)(})({
qppq
qppq
ppp
xx
xx
xxE
ppp xxx ,
Minimization using Roof-dual Relaxation
)1( pp xx
•
• is submodular ! Ignore (solvable using st-mincut)
Property of the problem:
pp xx 1
pp xx 1 is the optimal labelpx
Property of the solution:
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({'
qppqqppq
qppqqppq
pppppp
xxxx
xxxx
xxxxE
Double number of variables:
),(~
),(
)(})({
qppq
qppq
ppp
xx
xx
xxE
ppp xxx , )1( pp xx
Minimization using Roof-dual Relaxation
Recap
• Exact minimization of Submodular QBFs using graph cuts.
• Obtaining partially optimal solutions of non-submodular QBFs using graph cuts.
But ...
• Need higher order energy functions to model image structure – Field of experts [Roth and Black]
• Many problems in computer vision involve multiple labels
E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)i,ji c
x ϵ Labels L = {l1, l2, … , lk} Clique c V
Transforming problems in QBFs
Multi-label Functions
Pseudoboolean Functions
Higher order Pseudoboolean
Functions
Quadratic Pseudoboolean
Functions
Transforming problems in QBFs
Multi-label Functions
Pseudoboolean Functions
Higher order Pseudoboolean
Functions
Quadratic Pseudoboolean
Functions
Higher order to Quadratic
• Simple Example using Auxiliary variables
{0 if all xi = 0C1 otherwise
f(x) =
min f(x) min C1a + C1 ∑ ā xi
x ϵ L = {0,1}n
x =x,a ϵ {0,1}
Higher Order Submodular
Function
Quadratic Submodular Function
∑xi = 0 a=0 (ā=1) f(x) = 0
∑xi ≥ 1 a=1 (ā=0) f(x) = C1
Higher order to Quadratic
min f(x) min C1a + C1 ∑ ā xix=
x,a ϵ {0,1}
Higher Order Submodular
Function
Quadratic Submodular Function
∑xi
1 2 3
C1
C1∑xi
Higher order to Quadratic
min f(x) min C1a + C1 ∑ ā xix=
x,a ϵ {0,1}
Higher Order Submodular
Function
Quadratic Submodular Function
∑xi
1 2 3
C1
C1∑xi
a=1a=0Lower
envelop of concave
functions is concave
Higher order to Quadratic
min f(x) min f1 (x)a + f2(x)āx
=x,a ϵ {0,1}
Higher Order Submodular
Function
Quadratic Submodular Function
∑xi
1 2 3
a=1
Lower envelop of concave
functions is concave
f2(x)
f1(x)
Higher order to Quadratic
min f(x) min f1 (x)a + f2(x)āx
=x,a ϵ {0,1}
Higher Order Submodular
Function
Quadratic Submodular Function
∑xi
1 2 3
a=1a=0Lower
envelop of concave
functions is concave
f2(x)
f1(x)
Transforming problems in QBFs
Multi-label Functions
Pseudoboolean Functions
Higher order Pseudoboolean
Functions
Quadratic Pseudoboolean
Functions
Multi-label to Pseudo-boolean
So what is the problem?
Eb (x1,x2, ..., xm)Em (y1,y2, ..., yn)
Multi-label Problem
Binary label Problem
yi ϵ L = {l1, l2, … , lk} xi ϵ L = {0,1}
such that:Let Y and X be the set of feasible solutions, then
1.For each binary solution x ϵ X with finite energy there exists exactly one multi-label solution y ϵ Y
-> One-One encoding function T:X->Y
2. arg min Em(y) = T(arg min Eb(x))
Multi-label to Pseudo-boolean
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
Multi-label to Pseudo-boolean
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
Ishikawa’s result:
E(y) = ∑ θi (yi) + ∑ θij (yi,yj)i,ji
y ϵ Labels L = {l1, l2, … , lk}
θij (yi,yj) = g(|yi-yj|)Convex Functio
n
g(|yi-yj|)
|yi-yj|
Multi-label to Pseudo-boolean
• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
Schlesinger & Flach ’06:E(y) = ∑ θi (yi) + ∑ θij (yi,yj)
i,ji
y ϵ Labels L = {l1, l2, … , lk}
θij(li+1,lj) + θij (li,lj+1) θij
(li,lj) + θij
(li+1,lj+1)
Covers all Submodular multi-label functions
More general than Ishikawa
Multi-label to Pseudo-boolean
• Applicability– Only solves restricted class of energy functions– Cannot handle Potts model potentials
• Computational Cost– Very high computational cost– Problem size = |Variables| x |Labels|– Gray level image denoising (1 Mpixel image)
(~2.5 x 108 graph nodes)
Problems
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Connection between st-mincut and energy
minimization?
Recent Advances and Open Problems
St-mincut based Move algorithms
• Commonly used for solving non-submodular multi-label problems
• Extremely efficient and produce good solutions
• Not Exact: Produce local optima
E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji
x ϵ Labels L = {l1, l2, … , lk}
Move Making Algorithms
Solution Space
En
erg
y
Move Making Algorithms
Search Neighbourhood
Current Solution
Optimal Move
Solution Space
En
erg
y
Computing the Optimal Move
Search Neighbourhood
Current Solution
Optimal Move
xc
(t) Key Property
Move Space
Bigger move space
Solution Space
En
erg
y
• Better solutions
• Finding the optimal move hard
Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy
Space of Solutions (x) : LN
Move Space (t) : 2N
Search Neighbourhood
Current Solution
N Number of Variables
L Number of Labels
Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy Current SolutionCurrent Solution
Construct a move function
Construct a move function
Minimize move function to get optimal move
Minimize move function to get optimal move
Move to new
solution
Move to new
solution
How to minimize
move functions?
General Binary Moves
Minimize over move variables t to get the optimal move
x = t x1 + (1- t) x2
New solution
Current Solution
Second solution
Em(t) = E(t x1 + (1- t) x2)
Boykov, Veksler and Zabih, PAMI 2001
Move energy is a submodular QPBF (Exact Minimization
Possible)
Swap Move• Variables labeled α, β can swap their
labels
[Boykov, Veksler, Zabih]
Swap Move
Sky
House
Tree
GroundSwap Sky, House
[Boykov, Veksler, Zabih]
• Variables labeled α, β can swap their labels
Swap Move• Variables labeled α, β can swap their
labels
• Move energy is submodular if:– Unary Potentials: Arbitrary– Pairwise potentials: Semimetric
[Boykov, Veksler, Zabih]
θij (la,lb) ≥ 0
θij (la,lb) = 0 a = b
Examples: Potts model, Truncated Convex
Expansion Move
[Boykov, Veksler, Zabih][Boykov, Veksler, Zabih]
• Variables take label or retain current label
Expansion Move
Sky
House
Tree
Ground
Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky
[Boykov, Veksler, Zabih][Boykov, Veksler, Zabih]
• Variables take label or retain current label
Expansion Move
• Move energy is submodular if:– Unary Potentials: Arbitrary
– Pairwise potentials: Metric
[Boykov, Veksler, Zabih][Boykov, Veksler, Zabih]
θij (la,lb) + θij (lb,lc) ≥ θij (la,lc)
Semi metric+
Triangle Inequality
• Variables take label or retain current label
Examples: Potts model, Truncated linear
Cannot solve truncated quadratic
General Binary Moves
Move Type First Solution
Second Solution
Guarantee
Expansion Old solution All alpha Metric
Fusion Any solution Any solution
Minimize over move variables t
x = t x1 + (1-t) x2
New solution
First solution
Second solution
Move functions can be non-submodular!!
Solving Continuous Problems using Fusion Move
x = t x1 + (1-t) x2
(Lempitsky et al. CVPR08, Woodford et al. CVPR08)
x1, x2 can be continuous
Fx1
x2
x
Optical Flow Example
Final Solutio
n
Solution from
Method 1
Solution from
Method 2
Range Moves • Move variables can be multi-label
• Optimal move found out by using the Ishikawa
• Useful for minimizing energies with truncated convex pairwise potentials
O. Veksler, CVPR 2007
θij (yi,yj) = min(|yi-yj|,T)
|yi-yj|θij (yi,yj)
T
x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk
Move Algorithms for Solving Higher Order Energies
• Higher order functions give rise to higher order move energies
• Move energies for certain classes of higher order energies can be transformed to QPBFs.
E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)i,ji c
x ϵ Labels L = {l1, l2, … , lk} Clique c V
[Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]
Outline of the Tutorial
The st-mincut problem
What problems can we solve using st-mincut?
st-mincut based Move algorithms
Connection between st-mincut and energy
minimization?
Recent Advances and Open Problems
Solving Mixed Programming Problems
x – binary image segmentation (xi ∊ {0,1})
ω – non-local parameter (lives in some large set Ω)
constant unary potentia
ls
pairwise
potentials
E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj)i,ji
≥ 0
Rough Shape Prior
Stickman Model
ωPose
θi (ω, xi) Shape Prior
Open Problems
Submodular
Functions
st-mincut
Equivalent
Characterization of Problems Solvable using st-mincut
• What functions can be transformed to submodular QBFs?
Minimizing General Higher Order Functions
• We saw how simple higher order potentials can be solved
• How more sophisticated higher order potentials can be solved?
Exact Transformation
(global optimum)
Or Relaxed transformation
(partially optimal)
Summary
T
Sst-mincut
Labelling
Problem
Submodular Quadratic Pseudoboolean
Function
Move making algorithms
Sub-problem
Thanks. Questions?
Use of Higher order Potentials
Stereo - Woodford et al. CVPR 2008
P1 P2 P3
5
6
7
8
Pixels
Disparity Labels
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)
θij (xi,xj) =0 if xi=xj
C otherwise{
E(6,6,6) = 0 + 0 = 0
Use of Higher order Potentials
Stereo - Woodford et al. CVPR 2008
P1 P2 P3
5
6
7
8
Pixels
Disparity Labels
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)
θij (xi,xj) =0 if xi=xj
C otherwise{
E(6,6,6) = 0 + 0 = 0
E(6,7,7) = 1 + 0 = 1
Use of Higher order Potentials
Stereo - Woodford et al. CVPR 2008
P1 P2 P3
5
6
7
8
Pixels
Disparity Labels
Pairwise potential penalize slanted planar surfaces
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)
θij (xi,xj) =0 if xi=xj
C otherwise{
E(6,6,6) = 0 + 0 = 0
E(6,7,7) = 1 + 0 = 1
E(6,7,8) = 1 + 1 = 2
Computing the Optimal Move
Search Neighbourhood
Current Solution
Optimal Move
x
E(x)
xcTransformation function
T(t)
T(xc, t) = xn = xc + t
Computing the Optimal Move
Search Neighbourhood
Current Solution
Optimal Move
E(x)
xcTransformation function
T
Em Move Energy
(t)
x
Em(t) = E(T(xc, t))
T(xc, t) = xn = xc + t
Computing the Optimal Move
Search Neighbourhood
Current Solution
Optimal Move
E(x)
xc
Em(t) = E(T(xc, t))
Transformation function
T
Em Move Energy
T(xc, t) = xn = xc + t minimize
t*Optimal Move
(t)
x
Recommended