Joint Optimization of Segmentation and Appearance Modelsvision.stanford.edu/.../slides/sameep_presentation.pdf · Joint Optimization of Segmentation and Appearance Models David Mandle,

Joint Optimization of Segmentation and AppearanceModels

David Mandle, Sameep Tandon

April 29, 2013

David Mandle, Sameep Tandon (Stanford) April 29, 2013 1 / 19

Overview

1 Recap: Image Segmentation2 Optimization Strategy3 Experimental


Recap: Image Segmentation Problem

Segment an image into foreground and background

Figure: Left: Input image. Middle: Segmentation by EM (GrabCut). Right:Segmentation by the method covered today


Recap: Image Segmentation as Energy Optimization

Recall Grid Structured Markov Random Field:

Latent variables xi ∈ {0, 1} corresponding to foreground/backgroundObservations zi . Take to be RGB pixel valuesEdge potentials Φ(xi , zi ), Ψ(xi , xj)



The Graphical Model encodes the following (unnormalized) probabilitydistribution:



Goal: find x to maximize P(x , z) (z is observed)

Taking logs:

E (x , z) =∑i

φ(xi , zi ) +∑i ,j

ψ(xi , xj)

Unary potential φ(xi , zi ) encodes how likely it is for a pixel or patch yito belong to segmentation xi .

Pairwise potential ψ(xi , xj) encodes neighborhood info aboutpixel/patch segmentation labels




Taking logs:

E (x , z) =∑i

φ(xi , zi ) +∑i ,j

ψ(xi , xj)






Taking logs:

E (x , z) =∑i

φ(xi , zi ) +∑i ,j

ψ(xi , xj)






Taking logs:

E (x , z) =∑i

φ(xi , zi ) +∑i ,j

ψ(xi , xj)




Recap: GrabCut Model

Unary Potentials: log of Gaussian Mixture ModelI But to deal with tractability, we assign each xi to component ki

φ(xi , ki , θ|zi ) = − log π(xi , ki ) + logN(zi ;µ(ki ),Σ(ki ))

Pairwise Potentials:

ψ(xi , xj |zi , zj) = [xi 6= xj ] exp(−β−1‖zi − zj‖2)

where β = 2 · avg(‖zi − zj‖2)


Recap: GrabCut Model

Unary Potentials: log of Gaussian Mixture ModelI But to deal with tractability, we assign each xi to component ki

φ(xi , ki , θ|zi ) = − log π(xi , ki ) + logN(zi ;µ(ki ),Σ(ki ))

Pairwise Potentials:

ψ(xi , xj |zi , zj) = [xi 6= xj ] exp(−β−1‖zi − zj‖2)

where β = 2 · avg(‖zi − zj‖2)


Recap: GrabCut Optimization Strategy

GrabCut EM Algorithm

1 Initialize Mixture Models

2 Assign GMM components:

ki = arg minkφ(xi , ki , θ|zi )

3 Get GMM parameters:

θ = arg minθ

∑i

φ(xi , ki , θ|zi )

4 Perform segmentation using reduction to min-cut:

x = arg minx

E (x , z ; k , θ)

5 Iterate from step 2 until converged








θ = arg minθ

∑i



x = arg minx

E (x , z ; k , θ)









θ = arg minθ

∑i



x = arg minx

E (x , z ; k , θ)









θ = arg minθ

∑i



x = arg minx

E (x , z ; k , θ)









θ = arg minθ

∑i



x = arg minx

E (x , z ; k , θ)



New Model

Let’s consider a simpler model. This will be useful soon

Unary terms: Histograms

I K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over

foreground/backgroundI

φ(xi , bi , θ) = − log θxibi

Pairwise Potentialsψ(xi , xj) = wij |xi − xj |

We will define wij later; for now, consider pairwise equal to Grabcut

Total Energy:

E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +

∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


New Model


Unary terms: HistogramsI K bins, bi is bin of pixel zi

I θ0, θ1 ∈ [0, 1]K represent color models (distributions) overforeground/background

I




Total Energy:


∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


New Model


Unary terms: HistogramsI K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over

foreground/background

I




Total Energy:


∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


New Model







Total Energy:


∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


New Model







Total Energy:


∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


New Model







Total Energy:


∑(p,q)∈N

wpq|xp − xq|

P(zp|θxp) = θxpbp


EM under new model

1 Initialize histograms θ0, θ1.

2 Fix θ. Perform segmentation using reduction to min-cut:

x = arg minx

E (x , θ0, θ1)

3 Fix x. Compute θ0, θ1 (via standard parameter fitting).



EM under new model



x = arg minx

E (x , θ0, θ1)




EM under new model



x = arg minx

E (x , θ0, θ1)




EM under new model



x = arg minx

E (x , θ0, θ1)




Optimization

Goal: Minimize Energy

minxE (x)

E (x) =∑k

hk(n1k) +

∑(p,q)∈N

wpq|xp − xq|︸︷︷︸E1(x)

+ h(n1)︸︷︷︸E2(x)

where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

This is hard!

But efficient strategies for optimizing E 1(x) and E 2(x) separately


Optimization


minxE (x)

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


+ h(n1)︸︷︷︸E2(x)

where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

This is hard!



Optimization


minxE (x)

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


+ h(n1)︸︷︷︸E2(x)

where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

This is hard!



Optimization via Dual Decomposition

Consider an optimization of the form

minx f1(x) + f2(x)

where optimizing f (x) = f1(x) + f2(x) is hard

But minx f1(x) and minx f2(x) are easy problems

Dual Decomposition idea: Optimize f1(x) and f2(x) separately andcombine in a principled way.




minx f1(x) + f2(x)







minx f1(x) + f2(x)







minx f1(x) + f2(x)






Original Problem:minx f1(x) + f2(x)

Introduce local variables:

minx1,x2 f1(x1) + f2(x2)

s.t x1 = x2

Equivalent Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0





minx1,x2 f1(x1) + f2(x2)

s.t x1 = x2

Equivalent Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0





minx1,x2 f1(x1) + f2(x2)

s.t x1 = x2

Equivalent Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0



Primal Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0

Lagrangian Dual:

g(y) = minx1,x2

f1(x1) + f2(x2) + yT (x2 − x1)

Decompose Lagrangian Dual:

g(y) = (minx1

f1(x1)− yT x1) + (minx2

f2(x2) + yT x2)



Primal Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0

Lagrangian Dual:

g(y) = minx1,x2

f1(x1) + f2(x2) + yT (x2 − x1)


g(y) = (minx1

f1(x1)− yT x1) + (minx2

f2(x2) + yT x2)



Primal Problem:

minx1,x2 f1(x1) + f2(x2)

s.t x2 − x1 = 0

Lagrangian Dual:

g(y) = minx1,x2

f1(x1) + f2(x2) + yT (x2 − x1)


g(y) = (minx1

f1(x1)− yT x1) + (minx2

f2(x2) + yT x2)



g(y) = (minx1

f1(x1)− yT x1)︸︷︷︸g1(y)

+ (minx2

f2(x2) + yT x2)︸︷︷︸g2(y)

For all y , g(y) is a lower bound on optimal of the primal problem.

Maximize g(y) w.r.t. y to get the tightest bound

Further, g(y) is concave in y . Many techniques for concavemaximization (subgradient ascent, etc).

Ideally have fast optimization strategies for g1(y) and g2(y)



g(y) = (minx1

f1(x1)− yT x1)︸︷︷︸g1(y)

+ (minx2

f2(x2) + yT x2)︸︷︷︸g2(y)







g(y) = (minx1

f1(x1)− yT x1)︸︷︷︸g1(y)

+ (minx2

f2(x2) + yT x2)︸︷︷︸g2(y)







g(y) = (minx1

f1(x1)− yT x1)︸︷︷︸g1(y)

+ (minx2

f2(x2) + yT x2)︸︷︷︸g2(y)







Back to our Segmentation problem

Energy

E (x) =∑k

hk(n1k) +

∑(p,q)∈N

wpq|xp − xq|+ h(n1)

where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

Group terms

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


+ h(n1)︸︷︷︸E2(x)

E (x) = E1(x) + E2(x)




Energy

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

Group terms

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


+ h(n1)︸︷︷︸E2(x)

E (x) = E1(x) + E2(x)




Energy

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


where n1k =

∑p∈Vk

xp and n1 =∑

p∈V xp

Group terms

E (x) =∑k

hk(n1k) +

∑(p,q)∈N


+ h(n1)︸︷︷︸E2(x)

E (x) = E1(x) + E2(x)



Dual Decomposition on E (x) = E 1(x) + E 2(x)

Φ(y) = minx1

(E 1(x1)− yT x1)︸︷︷︸Φ1(y)

+ minx2

(E 2(x2) + yT x2)︸︷︷︸Φ2(y)

Maximize Φ(y) w.r.t. y to get the tightest lower bound.

Φ1(y) can be computed efficiently via reduction to min s-t cut.

Φ2(y) via convex minimization (several strategies)

You could now use your favorite concave maximization strategy forΦ(y)




Φ(y) = minx1

(E 1(x1)− yT x1)︸︷︷︸Φ1(y)

+ minx2

(E 2(x2) + yT x2)︸︷︷︸Φ2(y)








Φ(y) = minx1

(E 1(x1)− yT x1)︸︷︷︸Φ1(y)

+ minx2

(E 2(x2) + yT x2)︸︷︷︸Φ2(y)








Φ(y) = minx1

(E 1(x1)− yT x1)︸︷︷︸Φ1(y)

+ minx2

(E 2(x2) + yT x2)︸︷︷︸Φ2(y)








Φ(y) = minx1

(E 1(x1)− yT x1)︸︷︷︸Φ1(y)

+ minx2

(E 2(x2) + yT x2)︸︷︷︸Φ2(y)







Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.

Sketch:

I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has formy = s1

I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave

F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions

segmentations x per breakpoint

I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)

I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.

I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.




Sketch:

I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has formy = s1










Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form

y = s1











y = s1I Implication: maximize Φ(s1) over all possible s

I Φ1(s1) is piecewise-linear concaveF |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions









y = s1I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave





































References

MRF Review Slides:http://vision.stanford.edu/teaching/cs231b spring1213/slides/segmentation.pdfDual Decomposition Slides: Petter Strandmark and Fredrik Kahl. Paralleland Distributed Graph Cuts by Dual Decomposition.http://www.robots.ox.ac.uk/vgg/rg/slides/parallelgraphcuts.pdf


Documents

Joint Optimization of Segmentation and Appearance Modelsvision.stanford.edu/.../slides/sameep_presentation.pdf · Joint Optimization of Segmentation and Appearance Models David Mandle,