Interactive Graph Cuts

for Optimal Boundary & Region Segmentation of Objects in N-D Images

Yuri Y. Boykov and Marie-Pierre Jolly

1

From [3]2

Vision problems as image labeling

• depth (stereo)

• object index (segmention)

• original intensity (image restoration)

3

Labeling problems can be cast in terms of energy minimization

Labeling of pixels

Penalty for pixel labeling

Interaction between neighboring pixels. Smoothing term.

E(L) =∑

p∈P

Dp(Lp) +∑

p,q∈N

Vp,q(Lp, Lq)

Dp : Lp → "

L : P → Lp

Vp,q : P x P → "

4

Energy minimization can be solved with graph cuts

S terminal S terminal

T terminal T terminal5

• Energy function and graph construction

• Min-cut of graph minimizes energy

• Summar max-flow/min-cut algorithms

• Rest of bone segmentation example

6

Some Notation

L = (L1, . . . , Lp, . . . , L|P |)

P = set of pixels p

N = set of unordered pairs (p, q) of neighbors in P

Binary vector representing a binary segmentation

G = (V, E) graph with nodes, V, and edges, E

B = set of user defined background pixels

O = set of user defined object pixels

7

Labeling problems can be cast in terms of energy minimization

Labeling of pixels

Penalty for pixel labeling

Interaction between neighboring pixels. Smoothing term.

E(L) =∑

p∈P

Dp(Lp) +∑

p,q∈N

Vp,q(Lp, Lq)

Dp : Lp → "

L : P → Lp

Vp,q : P x P → "

8

Their Energy Function

E(L) = λ ·

∑

p∈P

Rp(Lp) +∑

(p,q)∈N

B(p, q) · δ(Lp, Lq)

Dp(Lp) becomes regional term Rp(Lp)

Vp,q becomes boundary term B(p, q) · δ(Lp, Lq)

9

Regional term

E(L) = λ ·

∑

p∈P

Rp(Lp) +∑

(p,q)∈N

B(p, q) · δ(Lp, Lq)

Penalize pixel label based on local properties

Negative log-likelihood of intensity

Rp(obj) = − lnPr(Ip|O)

Rp(bkq) = − lnPr(Ip|B)

10

Boundary term: penalize dissimilar neighbors

E(L) = λ ·

∑

p∈P

Rp(Lp) +∑

(p,q)∈N

B(p, q) · δ(Lp, Lq)

B(p, q) ∝ exp(−(Ip − Iq)2

2σ2) ·

1

dist(p, q)

rp

sq

11

Boundary term: penalize dissimilar neighbors

B(p, q) ∝ exp(−(Ip − Iq)2

2σ2) ·

1

dist(p, q)

rp

sq

E(L) = λ ·∑

p∈P

Rp(Lp) +∑

(p,q)∈N

B(p, q) · |Lp − Lq|

12

Graph construction: cost of n-links

Object

Background

c(p, q) = B(p, q) if (p, q) ∈ N

B(p, q)

T

S

B(p, q) ∝ exp(−(Ip − Iq)2

2σ2) ·

1

dist(p, q)

13

Graph construction: cost of t-link (p, S)

Object

Background

If p ∈ O then c(p, q) = KK

K = 1 + maxp∈P

∑

q:(p,q)∈N

B(p, q)

T

S

14

Object

Background

If p /∈ O ∪ B then c(p, q) = λ · Rp(bkg)

Graph construction: cost of t-link (p, S)

T

S

15

Object

Background

Graph construction: cost of t-link (p, S)

T

S

If p ∈ B then c(p, q) = 0 0

16

Claim: min-cut of graph minimizes energy

• Min-cut on G is a feasible cut

• Each feasible cut has a unique binary segmentation

• Segmentation associated with min-cut that satisfies user defined constraints minimizes the energy function

17

Summary of max-flow/min-cut algorithms

• Augmenting paths (Ford and Fulkerson)

• Push-relabel (Goldberg and Tarjan)

• Their implementation (see [2])

18

From [3]19

Citations

• [1] Kolmogorov, V. and Zabih, R. What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147-159, February 2004.

• [2] Boykov, Y. and Kolmogorov, V. An experimental comparison of min-cut/max-flow algoritms for energy minimization in computer vision. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 9, pp. 1124-1137, September 2004.

• [3] Boykov, Y., Torr, T. and Zabih, R. Tutorial on Discrete Optimization Methods in Computer Vision. ECCV 2004

20