Deformable/Active Contours

Lecture 06 Computer Vision

• Dr George Stockman

Professor Emeritus, Michigan State University

• Dr David A. Forsyth

Professor, University of Illinois at Urbana- Champaign

• Dr Yuri Boykov

Associate Professor, University of Western Ontario

• Kristen Grauman

Associate Professor, University of Texas at Austin

Material Citations

Suggested Readings

• Chapter 14

Linda G. Shapiro and George Stockman, “Computer Vision”, Upper Saddle River, NJ, Prentice Hall, 2001 .

• Chapter 5

Richard Szeliski , “Computer Vision: Algorithms and Applications”, Springer; 2011.

• Amir A. Amini, Terry E. Weymouth, and Ramesh C. Jain, “Using Dynamic Programming for Solving Variational Problems in Vision”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12, No. 9, September 1990, pp 855-867

Snakes ( energy-minimizing systems): • Is an energy-minimizing spline guided by external constraint

forces and influenced by image forces that pull it toward features such as lines and edges.

• Snakes are active contour models: they lock onto nearby edges, localizing them accurately.

• The name “Snake” comes from the way contours slither while minimizing their energy

Introduction

• Interactive interpretation, in which user-imposed constraint forces

guide the snake near features of interest

• The goal is to find a contour that best approximates the perimeter of an object

What can snakes do?

Deformable contours

• Given: initial contour (model) near desired object

• a.k.a. active contours, snakes

(Single frame)

Fig: Y. Boykov

[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

Deformable contours

• Given: initial contour (model) near desired object • Goal: evolve the contour to fit exact object boundary

(Single frame)

Fig: Y. Boykov [Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

During iterative process the shape and location of the contour evolves

Deformable contours

initial intermediate final

Fig: Y. Boykov

• Initialize near contour of interest

• Iteratively refine: elastic band is adjusted so as to

- Be near image positions with high gradients, and

- Satisfy shape “preferences” or contour priors

• Non-rigid, deformable objects can change their shape over time, e.g. lips, hands.

Figure from Kass et al. 1987

Why do we want to fit deformable shapes?

Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG

Figure from Shapiro & Stockman

Deformable contours: intuition

initial intermediate final

How is the current contour adjusted to find the new contour at each iteration? • Define a cost function (“energy” function) that says how good a possible

configuration is

• Seek next configuration that minimizes that cost function.

Deformable contours

Measures the appropriateness of the contour

• The total energy (cost) of the current snake is defined as:

externalinternaltotal EEE

A good fit between the current deformable contour and the target shape in the image will yield a low value for this cost function.

• Internal energy: encourage prior shape preferences: e.g., smoothness, elasticity, particular known shape.

• External energy (“image” energy): encourage contour to fit on places where image structures exist, e.g., edges.

Snakes energy function

10))(),(()( ssysxs

Parametric curve representation (continuous case)

• The snake is a contour represented parametrically as

• x(s) and y(s) are the coordinates along the contour

• Measure how well the curve matches the image data

• Attract the curve toward different image features

– Edges, lines, etc.

External energy: intuition

Magnitude of gradient

- (Magnitude of gradient)

22 )()( IGIG yx

22 )()( IGIG yx

How do edges affect “snap” of rubber band? Think of external energy from image as gravitational pull towards areas of high contrast

External image energy

• Image I(x,y)

• Gradient images and

• External energy at a point v(s) on the curve is

• External energy for the whole curve:

),( yxGx),( yxGy

)|))((||))((|())(( 22 sGsGsE yxexternal

1

0

))(( dssEE externalexternal

External image energy

Internal energy: intuition

http://www3.imperial.ac.uk/pls/portallive/docs/1/52679.JPG

A priori, we want to favor smooth shapes, contours with low curvature, contours similar to a known shape, etc. to balance what is actually observed (i.e., in the gradient image).

Internal energy: intuition

For a continuous curve, a common internal energy term is the “bending energy”.

At some point v(s) on the curve, it is given as:

The more the curve bends the larger this energy value is The weights α and β

dictate how much influence each component has Elasticity,

Tension Stiffness, Curvature

sd

d

ds

dsEinternal 2

2

))((

22

1

0

))(( dssEE internalinternal Internal energy for whole curve:

Internal energy

Bending energy

Total edge strength under curve

externalinternaltotal EEE

1

0

internal ))(( dssEEinternal

1

0

))(( dssEE externalexternal

Total energy (continuous form)

• Represent the curve with a set of n points

10),( niyx iii …

Parametric curve representation (discrete form)

21

0

2 |),(||),(| iiy

n

i

iixexternal yxGyxGE

Discrete image gradients

),( yxGx ),( yxGy

• If the curve is represented by n points

Discrete energy function: external term

)|))((||))((|())(( 22 sGsGsE yxexternal

• If the curve is represented by n points

Elasticity,

Tension

Stiffness

Curvature

10),( niyx iii

i1ivds

d

11112

2

2)()( iiiiiiids

d

1

0

2

11

2

1 2n

i

iiiiiinternalE

…

Discrete energy function: internal term

Continuity term: First derivative is approximated by a finite difference

Smoothness term: Enforce smoothness and avoid oscillations of the snake by penalizing high contour curvatures

sd

d

ds

dsEinternal 2

2

))((

22

• Current elastic energy definition uses a discrete estimate of the derivative, and can be re-written as:

Possible problem with this definition? This encourages a closed curve to shrink to a cluster

1

0

2n

i

iinternal LE 2

1

1

0

2

1 )()( ii

n

i

ii yyxx

Penalizing elasticity

21

0

2

1

2

1 )()(

n

i

iiiiinternal dyyxxE

• To stop the curve from shrinking to a cluster of points, we can adjust the energy function to be:

• This encourages chains of equally spaced points

Average distance between pairs of points – updated at each iteration

Penalizing elasticity

large small medium

• weight controls the penalty for internal elasticity

1

0

2n

i

iinternal LE 2

1

1

0

2

1 )()( ii

n

i

ii yyxx

Fig from Y. Boykov

Function of the weights

• If object is some smooth variation on a known shape, we can use a term that will penalize deviation from that shape:

where are the points of the known shape

1

0

2)ˆ(n

i

iiinternalE

}ˆ{ i

Fig from Y. Boykov

Optional: specify shape prior

• A simple elastic snake is defined by:

– A set of n points,

– An internal elastic energy term

– An external edge based energy term

• To use this to locate the outline of an object

– Initialize in the vicinity of the object

– Modify the points to minimize the total energy

Summary: Elastic Snake

• Several algorithms proposed to fit deformable contours, including methods based on

– Greedy search

– Dynamic programming (for 2d snakes)

– Gradient descent

Energy minimization

• For each point, search window around it and move to where energy function is minimal

– Typical window size, e.g., 5 x 5 pixels

• Stop when predefined number of points have not changed in last iteration, or after max number of iterations

• Note

– Convergence not guaranteed

– Need decent initialization

Energy minimization: greedy

A greedy algorithm makes locally optimal choices, hoping that the final solution will be globally optimum

1

0

110 ),(),,(n

i

iiintotal EE

• Often snake energy can be rewritten as a sum of pair-wise interaction potentials

• Or sum of triple-interaction potentials.

1

0

1110 ),,(),,(n

i

iiiintotal EE

Energy minimization: dynamic programming

1v

2v3v

4v6v

5v

With this form of the energy function, we can minimize using dynamic programming, with the Viterbi algorithm. Iterate until optimal position for each point is the center of the box, i.e., the snake is optimal in the local search space constrained by boxes.

[Amini, Weymouth, Jain, 1990]

Fig from Y. Boykov

Energy minimization: dynamic programming

),( 44 nvvE),( 433 vvE

)3(3E

)(3 mE )(4 mE

)3(4E

)2(4E

)1(4E

)(mEn

)3(nE

)2(nE

)1(nE

)2(3E

)1(3E

)(2 mE

)3(2E

),(...),(),( 11322211 nnntotal vvEvvEvvEE

),( 322 vvE

)1(2E

)2(2E

),( 211 vvE

0)1(1 E

0)2(1 E

0)3(1 E

0)(1 mE

Main idea: determine optimal position (state) of predecessor, for each possible position of self. Then backtrack from best state for last vertex.

This example: considering first-order interactions, one iteration.

states

1

2

…

m

vert

ices

1v 2v 3v 4v nv

)( 2nmOComplexity: vs. brute force search Example adapted from Y. Boykov

Energy minimization: dynamic programming

1v

2v3v

4v6v

5v

With this form of the energy function, we can minimize using dynamic programming, with the Viterbi algorithm. Iterate until optimal position for each point is the center of the box, i.e., the snake is optimal in the local search space constrained by boxes.

[Amini, Weymouth, Jain, 1990]

Fig from Y. Boykov

Energy minimization: dynamic programming

),(...),(),( 11322211 nnn vvEvvEvvE

DP can be applied to optimize an open ended snake

For a closed snake, a “loop” is introduced into the total energy.

1n

),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn

1

n

2

1n

3 4

Work around:

1) Fix v1 and solve for rest .

2) Fix an intermediate node at its position found in (1), solve for rest.

Energy minimization: dynamic programming

• Use final contour/model extracted at frame t as an initial solution for frame t+1

• Evolve initial contour to fit exact object boundary at frame t+1

• Repeat, initializing with most recent frame

Tracking via deformable models

Tracking Heart Ventricles (multiple frames)

Deformable contours

Applications: • Traffic monitoring • Human-computer interaction • Animation • Surveillance • Computer Assisted Diagnosis in medical imaging

Tracking via deformable models

• May over-smooth the boundary

• Cannot follow topological changes of objects

Limitations

• External energy: snake does not really “see” object boundaries in the image unless it gets very close to it.

image gradients

are large only directly on the boundary

I

Limitations

Interactive forces

• An energy function can be altered online based on user input – use the cursor to push or pull the initial snake away from a point.

• Modify external energy term to include:

1

02

2

||

n

i i

pushp

rE

Nearby points get pushed hardest

Interactive forces

Pros:

• Useful to track and fit non-rigid shapes

• Contour remains connected

• Possible to fill in “subjective” contours

• Flexibility in how energy function is defined, weighted

Cons:

• Must have decent initialization near true boundary, may get stuck in local minimum

• Parameters of energy function must be set well based on prior information

Snakes: pros and cons

• Deformable shapes and active contours are useful for

– Segmentation: fit or “settle” to boundary in image

– Tracking: previous frame’s estimate serves to initialize the next

• Optimization for snakes: general idea of minimizing a cost/energy function

– Can define terms to encourage certain shapes, smoothness, low curvature, push/pulls, …

– And can use weights to control relative influence of each component cost term.

• Edges / optima in gradients can act as “attraction” force for interactive segmentation methods.

• Distance transform definition: efficient map of distances to nearest feature of interest.

Summary: main points

• Demo by: Chris Bregler and Malcolm Slaney

http://www.slaney.org/malcolm/pubs.html

Snakes: Demo