ECCV06 Tutorial PartI Yuri

8/8/2019 ECCV06 Tutorial PartI Yuri

1/64

European Conference on Computer Vision 2006 : Graph Cuts vs. Level Sets, Y. Boykov (UWO), D. Cremers (U. of Bonn), V. Kolmogorov (UCL)

ECCV 2006 tutorial on

Graph Cuts vs. Level Sets

part I

Basics of Graph Cuts

Vladimir Kolmogorov

University College London

Yuri Boykov

University of Western Ontario

Daniel Cremers

University of Bonn


2/64


Graph Cuts versus Level Sets

Part I: Basics ofgraph cuts

Part II: Basics oflevel-sets Part III: Connecting graph cuts and level-sets

Part IV: Global vs. local optimization algorithms


3/64


Graph Cuts versus Level Sets

Part I: Basics ofgraph cuts

Main idea for min-cut/max-flow methods and applications

Implicit and explicit representation of boundaries

Graph cut energy (different views)

submodularity, geometric functionals, posterior energy (MRF)

Extensions to multi-label problems convex and non-convex (robust) interactions, a-expansions


4/64


5/64


Graph cuts for optimal boundary detection

(simple example la Boykov&Jolly, ICCV01)

n-links

s

ta cuthardconstraint

hardconstraint

Minimum cost cut can becomputed in polynomial time

(max-flow/min-cut algorithms)

=22

exp

pq

pqIw

pqI


6/64


Minimum s-tcuts algorithms

Augmenting paths [Ford & Fulkerson, 1962]

Push-relabel [Goldberg-Tarjan, 1986]


7/64


Augmenting Paths

Find a path from S to

T along non-saturatededges

source

A graph with two terminals

S T

sink

Increase flow alongthis path until some

edge saturates


8/64


9/64


Augmenting Paths

Find a path from S to

T along non-saturatededges

source

A graph with two terminals

S T

sink

Increase flow alongthis path until some

edge saturates

Iterate until

all paths from S to T have

at least one saturated edgeMAX FLOW MIN CUT


10/64


Optimal boundary in 2D

max-flow = min-cut


11/64

E C f C t Vi i 2006 G h C t L l S t Y B k (UWO) D C (U f B ) V K l (UCL)


12/64


Graph cuts applied to multi-view

reconstruction

Calibratedimages ofLambertianscene

3D model ofscene

CVPR05 slides from Vogiatzis, Torr, Cippola

European Conference on Computer Vision 2006 : Graph Cuts vs Level Sets Y Boykov (UWO) D Cremers (U of Bonn) V Kolmogorov (UCL)


13/64



reconstruction

(x)


photoconsistency

European Conference on Computer Vision 2006 : Graph Cuts vs Level Sets Y Boykov (UWO) D Cremers (U of Bonn) V Kolmogorov (UCL)


14/64


Estimating photoconsistency

in a narrow bad


Occlusion

1. Get nearest pointon outer surface

2. Use outer surfacefor occlusions

2. Discard occludedviews

European Conference on Computer Vision 2006 : Graph Cuts vs. Level Sets Y. Boykov (UWO) D. Cremers (U. of Bonn) V. Kolmogorov (UCL)


15/64

European Conference on Computer Vision 2006 : Graph Cuts vs. Level Sets , Y. Boykov (UWO), D. Cremers (U. of Bonn), V. Kolmogorov (UCL)


reconstruction


Source

Sink

The cost of the cut integrates

photoconsistency over the whole space



16/64

European Conference on Computer Vision 2006 : Graph Cuts vs. Level Sets , Y. Boykov (UWO), D. Cremers (U. of Bonn), V. Kolmogorov (UCL)


reconstruction


visual hull(silhouettes)

surface of good photoconsistency



17/64

p p p , y ( ), ( ), g ( )

Graph cuts for video textures

Graph-cuts video textures

(Kwatra, Schodl, Essa, Bobick 2003)

Short video clip

21

Long video clip

a cut

3D generalization of image-quilting (Efros & Freeman, 2001)


18/64



19/64

Cuts on directed graphs

Cost of a cut includes only edges from the source to the sink components Cuts cost (on a directed graph) changes if terminals are swapped

Swapping terminals s and tis similar

to switching surface orientation

s

Cut on a directed graphFlux of a vector field throughhypersurface with orientation (A or B)

t



20/64

Cuts on directed graphs



21/64

Segmentation of elongated structures



22/64

Simple shape priorspreferred surface normals

[Funka-Lea et al.06]

blob prior

p

q

))cos(1()( +> cIfw qp

Extra penalty


23/64



24/64

NOTE: hard constrains are not required, in general.EM-style optimization of piece-vice constant Mumford-Shahmodel

Adding regional properties

(another segmentation example la Boykov&Jolly01)

pqw

n-links

s

t a cut)(tDp

)(sDp

t-

lin

k

t-link

expected intensities ofobject and background

can be re-estimated

ts II and ||)( tpp IIconsttD =

||)( spp IIconstsD =



25/64


(another example la Boykov&Jolly, ICCV01)



26/64


(another example la Boykov&Jolly, ICCV01)

a cut )|Pr(ln)(pppp

LILD =

given object and background intensity histograms

)(sDp

)(tDp s

t

I)|Pr( sIp

)|Pr( tIp

pI

More generally, regional bias can be based on any

intensity models of object and background



27/64

Iterative learning of regional models GMMRF cuts (Blake et al., ECCV04)

Grab-cut (Rother et al., SIGGRAPH 04)

parametric regional model Gaussian Mixture (GM)

designed to guarantee convergence



28/64

Shrinking bias Minimization of non-negative boundary costs (sum of

non-negative n-links) gives regularization/smoothing ofsegmentation results

Choosing n-link costs from local image gradients helpsimage-adaptive regularization

May result in over-smoothing or shrinking

Typical for all surface regularization techniques

Graph cuts are no different from snakes or level-sets on that



29/64

Shrinking bias

Optimal cut depending on the size of the hard constrained region



30/64

Shrinking bias

Shrinking (or under-segmentation)

is allowed by intensity gradient ramp

Image intensities on one scan line

ideal segment

actualsegment



31/64

Shrinking bias


Surface regularization approachto multi-view stereo

object protrusion



32/64

Regional term can counter-act shrinking bias

Ballooning force

favouring bigger volumes thatfill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for activecontour models and balloons for 2-d and 3-d images.PAMI,15(11):11311147, November 1993.

object protrusion


All voxels in the center of the scene

are connected to the object terminal



33/64

Shrinking bias




34/64

Uniform ballooning



Regional term based on


35/64

Regional term based on

Laplacian zero-crossings (flux

)

Can be seen as intelligent ballooning

Vasilevsky and Sidiqqi, 2002

R. Kimmel and A. M. Bruckstein 2003

++

- -

Image intensities on a scan line

Laplacian

of intensities

Basic idea


Integrating Laplacian zero-crossings


36/64

Integrating Laplacian zero crossings

into graph cuts (Kolmogorov&Boykov05)

The image is courtesy of David FleetUniversity of Toronto

graph cuts

Smart regional

terms counteract

shrinking bias



37/64

Implicit vs. Explicit graph cuts

Most current graph cuts technique implicitly use surfacesrepresented via binary (interior/exterior) labeling of pixels

0 1

1

1110

0

0

0

0 0



38/64

Implicit and Explicit graph cuts

Except, a recent explicit surfaces representation method- Kirsanov and Gortler, 2004



39/64

Explicit Graph Cuts

for multi-view reconstruction

Lempitsky et al., ECCV 2006 Explicit surface patches allow local estimation

of visibility when computing globally optimal

solution

Compare with Vogiatzis et al., CVPR05

approach for visibility

local orientedvisibility estimate



40/64

Explicit Graph Cuts

Regularization+

uniform

ballooning

some details arestill over-smoothed

Lempitsky et al., ECCV 2006



41/64

Boykov and Lempitsky, 2006

Regularization+

intelligent

ballooning

Low noise and no shrinking

Explicit Graph Cuts



42/64

Space carving

Explicit Graph Cuts

Kutulakos and Seitz, 2000


Three ways to look at


43/64

y

energy of graph cuts

I: Binary submodular energy

II: Approximating continuous surface functionalsIII: Posterior energy (MAP-MRF)



44/64

Simple example of energy

+=Npq

qppq

p

pp LLwLDLE )()()(

},{ tsLp t-links n-links

Boundary termRegional term

binary object

segmentation

pqw

n-links

s

t a cut)(tDp

)(sDp

t-

lin

k

t-link


45/64


Graph cuts for minimization of


46/64

p

continuous surface functionals II

[Kolmogorov&Boykov, ICCV 2005]

[Boykov&Kolmogorov, ICCV 2003]

Tight characterization of energies ofbinary cuts Cas

functionals of continuous surfaces

more in PART III

++=

)(

)(,)()(

CC

x

C

dpxfdsdsgCE vNr

r

Geometric length

any convex,

symmetric metricge.g. Riemannian

Flux

any vector field v

Regional bias

any scalar functionf


Graph cuts for minimization of


47/64

posterior energy III

Greig at. al. [IJRSS, 1989]

Posterior energy (MRF, Ising model)

),()|Pr(ln)(

+=Npq

qppq

p

pp LLVLDLE

},{ tsLp

Example: binary image restoration

Spatial prior(regularization)

Likelihood(data term)



48/64

Graph cuts algorithms can minimize

multi-label energies as well


Multi-scan-line stereo


49/64

with s-tgraph cuts (Roy&Cox98)

x

y


Multi-scan-line stereo


50/64

with s-tgraph cuts (Roy&Cox98)

s

t cut

L(p)

p

cut

x

y

labels

x

yDisparity

labels


s-tgraph-cuts for


51/64

multi-label energy minimization

Ishikawa 1998, 2000, 2003

Modification of construction by Roy&Cox 1998

V(dL)

dL=Lp-Lq

V(dL)

dL=Lp-Lq

Linear interactions Convex interactions

+=Npq

qp

p

pp LLVLDLE ),()()(1

RLp


Pixel interactions V:


52/64

convex vs. discontinuity-preserving

V(dL)

dL=Lp-Lq

Potts

model

Robustdiscontinuity preserving

Interactions V

V(dL)

dL=Lp-Lq

ConvexInteractions V

V(dL)

dL=Lp-Lq

V(dL)

dL=Lp-Lq

linearmodel


Pixel interactions:


53/64

convex vs. discontinuity-preserving

linear V

truncated

linear V



54/64

Robust interactions

NP-hard problem (3 or more labels)

two labels can be solved via s-tcuts (Greig at. al., 1989)

a-expansion approximation algorithm

(Boykov, Veksler, Zabih 1998, 2001)

guaranteed approximation quality (Veksler, thesis 2001) within a factor of 2 from the global minima (Potts model)

applies to a wide class of energies with robust interactions

Potts model (BVZ 1989) Metric interactions (BVZ 2001)

Submodular interactions (e.g Boros and Hummer, 2002, KZ 2004)


i l i h


55/64

a-expansion algorithm

1. Start with any initial solution2. For each label a in any (e.g. random) order

1. Compute optimal a-expansion move (s-t graph cuts)2. Decline the move if there is no energy decrease

3. Stop when no expansion move would decrease energy


i


56/64

other labelsa

a-expansion move

Basic idea: break multi-way cut computationinto a sequence of binary s-tcuts


i


57/64

a-expansion moves

initial solution

-expansion

-expansion

-expansion

-expansion

-expansion

-expansion

-expansion

In each a-expansion a given label a grabs space from other labels

For each move we choose expansion that gives the largest decrease inthe energy: binary optimization problem


a-expansions:

l f t i i t ti


58/64

examples ofmetric interactions

Potts V

noisy diamondnoisy shaded diamond

( ),V

( ) ,V

Truncated linear V


M lti h t


59/64

Multi-way graph cuts

Multi-object Extraction

Obvious generalization of binary object extraction technique

(Boykov, Jolly, Funkalea 2004)


M lti a graph c ts


60/64


Stereo/Motion with slanted surfaces

(Birchfield &Tomasi 1999)

Labels = parameterized surfaces

EM based: E step = compute surface boundaries

M step = re-estimate surface parameters


Multi way graph cuts


61/64


stereo vision

original pair of stereo images

depth map

ground truthBVZ 1998KZ 2002




62/64


Graph-cut textures(Kwatra, Schodl, Essa, Bobick 2003)

similar to image-quilting (Efros & Freeman, 2001)

A

BC D

E F G

H I J

A B

G

DC

F

H I J

E




63/64


Graph-cut textures(Kwatra, Schodl, Essa, Bobick 2003)


a expansions vs simulated annealing


64/64

normalized correlation,

start for annealing, 24.7% err

simulated annealing,

19 hours, 20.3% erra-expansions (BVZ 89,01)

90 seconds, 5.8% err

0

20000

40000

60000

80000

100000

1 10 100 1000 10000 100000

Time in seconds

SmoothnessEnergy

Annealing Our method

a-expansions vs. simulated annealing

Documents

ECCV06 Tutorial PartI Yuri