Active Contours / Planes
Sebastian Thrun, Gary Bradski, Daniel RussakoffStanford CS223B Computer Vision
http://robots.stanford.edu/cs223b
Some slides taken from: Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Outline
Snakes Planes
Sebastian Thrun Stanford University CS223B Computer Vision
Introduction (1)
The active contour model, or snake, is defined as an energy-minimizing spline.
Active contours results from work of Kass et.al. in 1987.
Active contour models may be used in image segmentation and understanding.
The snake’s energy depends on its shape and location within the image.
Snakes can be closed or open
Sebastian Thrun Stanford University CS223B Computer Vision
Example
Sebastian Thrun Stanford University CS223B Computer Vision
Introduction (2)
Aorta segmentation using active contours
Sebastian Thrun Stanford University CS223B Computer Vision
Introduction (3)
First an initial spline (snake) is placed on the image, and then its energy is minimized.
Local minima of this energy correspond to desired image properties.
Unlike most other image models, the snake is active, always minimizing its energy functional, therefore exhibiting dynamic behavior.
Also suitable for analysis of dynamic data or 3D image data.
Sebastian Thrun Stanford University CS223B Computer Vision
Kass Algorithm
The snake is defined parametrically as v(s)=[x(s),y(s)], where s[0,1] is the normalized arc length along the contour. The energy functional to be minimized may be written as
Econt = snake continuity
Ecurv = snake curvature
Eimage = image forces (e.g., edge attraction)
10
10
10
10
*
))((
))(())((
))((
dssvE
dssvEdssvE
dssvEE
image
curvcont
snakesnake
Sebastian Thrun Stanford University CS223B Computer Vision
Internal Energy
Continuity:
Curvature:
2
))((ds
dvsvEcont
2
2
2
))((ds
vdsvEcurv
Sebastian Thrun Stanford University CS223B Computer Vision
Image Forces
Dark/Bright Lines
Edges
))(())(( svIsvEimage
))(())(( svIsvEimage
Sebastian Thrun Stanford University CS223B Computer Vision
Trade-offs
determine trade-off
10
10
10
10
*
))((
))(())((
))((
dssvE
dssvEdssvE
dssvEE
image
curvcont
snakesnake
Sebastian Thrun Stanford University CS223B Computer Vision
Numerical Algorithm
Select N initial locations p1,…, pN Update until convergence
|)(|
|2|
||
1.0 211
21
k
kkk
kk
kiii
pI
ppp
pp
ppp
|)(||2|||2
112
1*
kk
kkkkksnake pIpppppE
Sebastian Thrun Stanford University CS223B Computer Vision
Snakes, Done Right
Define Spline over p1,…, pN Optimize criterion for all points on spline Allow for corners
(optimization becomes tricky, fills entire literature)
Sebastian Thrun Stanford University CS223B Computer Vision
Applications
Main Applications are: Segmentation Tracking Registration
http://www.markschulze.net/snakes/
Sebastian Thrun Stanford University CS223B Computer Vision
Examples (1)
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Examples (2)
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Examples (3)
Heart Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Examples (4)
3D Segmentation of the Hippocampus
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Examples (5)
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Example (6)
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Example (7)
Julien Jomier
Sebastian Thrun Stanford University CS223B Computer Vision
Problems with snakes
Snakes sometimes degenerate in shape by shrinking and flattening.
Stability and convergence of the contour deformation process unpredictable.
Solution: Add some constraints
Initialization is not straightforward.
Solution: Manual, Learned, Exhaustive
Sebastian Thrun Stanford University CS223B Computer Vision
References
M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. In Proc. 1st ICCV, pages 259-268, June 1987. London, UK.
Yongjik Kim. A summary of Implicit Snake Formulation.
Jorgen Ahlberg. Active Contours in Three Dimensions.
M. Bertalmio, G. Sapiro and G. Randall. Morphing Active Contours. IEEE PAMI, Vol 22, No 7, July 2000
Sebastian Thrun Stanford University CS223B Computer Vision
Outline
Snakes Planes
Sebastian Thrun Stanford University CS223B Computer Vision
Finding Planes
Lines in Image (e.g., Hough transform) Planes in 3D space (e.g., stereo reconstruction,
SFM) Problems:
– Number of planes– Parameters of planes– Data association: which point belongs to which plane– Outlier removal (noise, non-flat surfaces)
Sebastian Thrun Stanford University CS223B Computer Vision
Range Data in Multi-Planar Environment
Taken with Laser Range Finder, but similar for
Sebastian Thrun Stanford University CS223B Computer Vision
Basic Idea
******
Sebastian Thrun Stanford University CS223B Computer Vision
Mathematical Model: Expectation Maximization
3D Model: },,,{ 21 J
3, jjj Planar surface in 3D
ijij zz ),dist(
Distance point-surfacesurface
surface normal
y
x
z
displacement
Sebastian Thrun Stanford University CS223B Computer Vision
Mixture Measurement Model
Case 1: Measurement zi caused by plane j
2
2)(
2
1
22
1)|(
jij z
ji ezp
2
2max
2ln
2
1
2max
*2
11)|(
z
i ez
zp
Case 2: Measurement zi caused by something else
Sebastian Thrun Stanford University CS223B Computer Vision
Measurement Model with Correspondences
J
j
jijj
zc
zc
Ji eccczp 12
2
2
2max
*
)(
2ln
2
1
2*12
1),,,,|(
correspondence variables C:
}
J
jj
j
cc
cc
1*
*
1
}1,0{,
1
)(
2ln
2
1
2
12
2
2
2max
*
2
1),|(
i
zc
zc
J
j
jijiji
eCZp
Sebastian Thrun Stanford University CS223B Computer Vision
Expected Log-Likelihood Function
1
)(
2ln
2
1
2
12
2
2
2max
*
2
1),|(
i
zc
zc
J
j
jijiji
eCZp
i
J
j
jijij
ic
zcE
zcE
J
CZpE
12
2
2
2max
*
2
)(][
2
1
2ln][
2
1
2)1(
1ln
)|,(ln
…after some simple math
mapping with known data association
probabilistic data association
Sebastian Thrun Stanford University CS223B Computer Vision
E-Step
Calculate expectations with fixed model :
(normalize so that )
2
2
2
2max
)(
2
1
2ln
2
1
*
][
][
jij z
ij
z
i
ecE
ecE
J
jiji cEcE
1* 1][][
Sebastian Thrun Stanford University CS223B Computer Vision
M-Step
Maximize
i
jijij
ic
zcE
zcE
J
CZpE
2
2
2
2max
*
2
)(][
2
1
2ln][
2
1
2)1(
1ln
)|,(ln
1 jj subject to
Is equivalent to minimizing
i
J
jjijij zcE
1
2)]([ 1 jj subject to
Sebastian Thrun Stanford University CS223B Computer Vision
M-Step: Solve via Lagrange Multipliers
i
J
jjijij zcE
1
2 min)]([ 1: jjj
i
J
j
J
jjjjjijij zcEL
1 1
2)]([
Define
0)]([!
jijijj
zcEL
0)]([!
jjijijijj
zzcEL
1 jj
And observe that
Sebastian Thrun Stanford University CS223B Computer Vision
M-Step: Solve via Lagrange Multipliers
Solve for :
Substitute back:
Is of the linear form:
Solution: two Eigenvectors of A with smallest Eigenvalues.
kjkj
kkjkj
j cE
zcE
][
][0)]([
!
jijijj
zcEL
0)][
][]([
!
jji
kjkj
kkjkj
ijijj
zcE
zcEzcE
L
0)]([!
jjijijijj
zzcEL
jjjA
Sebastian Thrun Stanford University CS223B Computer Vision
Determining Number of Surfaces
J =1
First model component
*
*
J =1
E-Step
*
*
J =1
M-Step
*
*
J =3
Add model components
J =3
E-Step
J =3
M-step
J =1
Prune model
J =3
Add model components
J =3
E/M Steps
*J =2
Prune model
Sebastian Thrun Stanford University CS223B Computer Vision
Choosing the “Right” Number of Planes: AIC
J=2 J=3 J=5J=0 J=1 J=4
increased data likelihood
increased prior probability
)(log)|(log)|(log JpJdpconstdJp
Sebastian Thrun Stanford University CS223B Computer Vision
Model Selection
Approximately every 20 iterations of EM: Start new surfaces
– Near any set of collinear measurements Terminate unsupported surfaces
– If not supported by enough measurements– If density of measurements too low– If two planes are too close to each other
Sebastian Thrun Stanford University CS223B Computer Vision
Results
Sebastian Thrun Stanford University CS223B Computer Vision
Online Robotic Mapping @ CMU
Sebastian Thrun Stanford University CS223B Computer Vision
Online Robotic Mapping @ Stanford