Upload
opal-adele-stevenson
View
218
Download
1
Embed Size (px)
Citation preview
1
Tal Nir
Alfred M. Bruckstein
Ron Kimmel
Over-Parameterized Variational Optical Flow
Technion, Israel institute of technology
Haifa 32000
ISRAEL
2
What is optic flow?
• Optic flow relates to the perception of motion.• Optic flow – the apparent motion of objects in the scene
as seen on the 2D image plane.
3
An image
4
Warped image
5
The corresponding optical flow
6
Applications of optic flow
An important pre-processing for many visual tasks• Tracking.• Segmentation.• Compression.• Super-resolution – requires high accuracy.• 3D reconstruction (structure from motion).
7
Basic equations
, , 1 ( , , )I x u y v t I x y t
u,v are the optic flow components between frame t and t+1
0I u I v Ix y t
Linearized brightness constancy equation
Brightness constancy equation
8
The aperture problem
0x y t
x y t
t
I u I v I
uI I I
v
uI I
v
��������������
Linearized brightness constancy equation
From an algebraic point of view this is an ill-posed problem
An image with N pixels: N equations with 2N unknowns.
I�������������� ,s u v
Only the flow component in the gradient direction can be determined (normal flow).
9
Going around the aperture problem
Looking for locations where the image has • “Multiple” gradient directions,• Discontinuous first image derivatives,• “Corners”.
I��������������
I��������������
10
The Lucas-Kanade method
0 0
2
( , ) ( , )
Assume constant motion in a region
( , , 1) ( , , )x y N x y
J I x u y v t I x y t
2),(),( 00
),,()1,,(
yxNyx
kk tyxItdvvyduuxIJ
0 0
2
( , ) ( , )x y z
x y N x y
J I du I dv I
dvvv
duuu
kk
kk
1
1
B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” Proc. DARPA Image Understanding Workshop, April, 1981.
11
Lucas-Kanade continued
2
2
x zx x y
y zx y y
I II I I du
I II I I dv
0 0
2
( , ) ( , )
( , , 1)
( , , 1)
( , , 1) ( , , )
x y zx y N x y
x k k
y k k
z k k
J I du I dv I
I I x u y v tx
I I x u y v ty
I I x u y v t I x y t
Solve the linear 2x2 system of equations
1. The “aperture problem” can occur in certain regions (zero eigenvalue).
2. Typically, the aperture problem does not appear in an exact sense.
3. Method may yield a sparse flow field estimate.
12
Neighborhood based methods
• The flow in the patch can be described by a constant, affine, or other model.
M. Irani, B. Rousso, S. Peleg, “Recovery of Ego-Motion Using Region Alignment” . IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 19, No. 3, pp. 268--272, March 1997
• The smoothness within the patch is inherently enforced.• Discontinuities of the model within the patch may cause
inaccuracies.• The resulting problem is over-constrained.
13
Optical flow estimation – an ill posed
problem
Motion in a patch –Over constrained
solution (Lucas-Kanade)
Over-parameterized Variational
Our work
14
The variational approachB. K. P. Horn and B. G. Schunck, "Determining optical flow,"
Artificial Intelligence, vol. 17, pp. 185--203, 1981.
( , )min ( , )
u vE u v
2
2 2 2 2
( , ) ( , ) ( , )
( , )
( , )
D S
D x y t
S x y x y
E u v E u v E u v
E u v I u I v I dxdy
E u v u u v v dxdy
Find the flow which minimizes the functional
Composed of a data and smoothness (regularization) term
The resulting Euler-Lagrange equations
0)(
0)(
yyxxytyx
yyxxxtyx
vvIIvIuI
uuIIvIuI
15
Variational approach. Cont.’
• Dense optical flow field (i.e. a vector at each pixel).• The smoothness (regularization) term enables the
completion of the flow in locations with insufficient information.
• Global solution – incorporates all the available information.
• The best results are achieved by modern variational approaches.
16
T. Brox, A. Bruhn, N. Papenberg, J. Weickert“High Accuracy Optical Flow Estimation Based on a Theory
for Warping”, ECCV 2004.
22( , ) (x+w) (x) (x+w) (x) x
x ( , , )
( , ,1)
DE u v I I I I d
x y t
w u v
2 2( , ) xSE u v u v d
2 2 2s s
• L1 non-linear data term with a gradient constancy term
• L1 smoothness term in x,y,t space (3D)
2 22' div ' 0z x zI I I u u u
( , ) ( , ) ( , )D SE u v E u v E u v
Euler-Lagrange equation for u (Γ=0)
17
Brox et. al. “High Accuracy Optical Flow Estimation”. Cont.’
Three loops of iteration• Outer loop k.• Inner loop fixed point
iteration in order to deal with the nonlinearity in Ψ.
• Gauss-Seidel iterations are used in order to solve the resulting sparse linear system of equations.
1
1 1
(x ) (x)
;
(x )
(x )
(x ) ( , , )
k k kx y t
k k k k k k
kx
ky
kt
I w I I du I dv I
u u du v v dv
I I wx
I I wy
I I w I x y t
18
Brox et. al. “High Accuracy Optical Flow Estimation”. Advantages
• Solution in Multi-scale helps to avoid being trapped in local minima – large motion (reduction factor of 0.95).
• The 3D smoothness term solves the problem in the volume in contrast to the 2D (two frames) solution.
• The gradient constancy term reduces the sensitivity to brightness changes.
• Choosing Ψ as an approximately L1 function:• In the smoothness term it allows discontinuities in the
flow field.• In the data term it reduces the sensitivity to outliers.• The addition of ε is for numerical reasons.
19
Results – Brox et al.
20
Our motivation
Our motivation stems from the smoothness term
Penalty for changes in the optical flow
Penalty for changes from an optical flow model
Weighted spatio-temporal gradient
21
The proposed over-parameterization model
The different roles of the coefficients and basis functions• The basis functions are selected a-priori, the
coefficients are estimated.• The regularization is applied only to the coefficients.
1
( , , ) ( , , )n
i ii
u A x y t x y t
• Basis functions of the flow model
• Space and time varying coefficients
1
( , , ) ( , , )n
i ii
v A x y t x y t
The optical flow is now estimated via the coefficients
22
Over-parameterization - one frame
1
u
v*
*
u
*
*+v
Coefficients
Basis functions
Basis functions
Conventional representation
Over-parameterized representation
n
1
1A
nA1
n
+
23
Over-parameterized functional2
1 1
, , 1 ( , , ) xn n
D i i i ii i
E I x A y A t I x y t d
The new regularization term penalizes for changes in the model parameters.
2
1
xn
S ii
E A d
24
Euler-Lagrange equations
22
1
ˆ' div ' 0n
z z x q y q i qi
I I I I A A
The Euler-Lagange equation for the coefficient Aq
25
Over-Parameterization models Constant motion model
• This case reduces to the regular variational approach of solving directly for u and v.
1 2
1 2
1 ; 0
0 ; 1
11
( , , ) ( , , )n
i ii
u A x y t x y t A
21
( , , ) ( , , )n
i ii
v A x y t x y t A
The number of coefficients is n=2
26
Affine over-parameterization model• Six basis functions
is a relative weight constant.
1
( , , ) ( , , )n
i ii
u A x y t x y t
1
( , , ) ( , , )n
i ii
v A x y t x y t
1 2 3
4 5 6
ˆ ˆ1; ;
0; 0; 0
x y
1 2 3
4 5 6
0; 0; 0
ˆ ˆ1; ;x y
0
0
0
0
ˆ
ˆ
x xx
x
y yy
y
2
1
xn
S ii
E A d
27
Rigid motion over-parameterization model
• The optic flow of a rigid body
1 1 2 2 3 3 4 1 5 2 6 3; ; ; ; ;A A A A A A
21 2 3 4 5 6ˆ ˆˆ ˆ ˆ1; 0; ; ; 1 ;x xy x y
21 2 3 4 5 6ˆ ˆ ˆˆ ˆ0; 1; ; 1 ; ;y y xy x
21 3 1 2 3
22 3 1 2 3
ˆ ˆˆ ˆ ˆ1
ˆ ˆ ˆˆ ˆ1
u x xy x y
v y y xy x
1 2 3, ,T is the translation vector divided by the depth (z)
1 2 3, ,T is the rotation vector
28
Rigid motion, cont…’
• In a seminal paper
• The optical flow calculation is a pre-processing followed by motion and structure estimation.
• In our formulation, the rigid motion model is used directly in the optical flow estimation process.
29
Pure translation over-parameterization model
• Rigid motion with pure translation
1 3
2 3
ˆ
ˆ
u x
v y
Use only the first three coefficients and basis functions of the general rigid motion model.
30
Numerical scheme
• Multi-resolution necessary to deal with large displacements.
• At each resolution, three loops of iterations are applied.
We adopt parts of the numerical scheme from T. Brox, A. Bruhn, N. Papenberg, and J. Weickert,“High Accuracy Optical Flow Estimation Based on a Theory for Warping,” ECCV 2004.to our over-parameterization model
31
Outer loop k
Euler-Lagrange equations, q=1...n
Insert first order Taylor approximation to the brightness constancy equation
21 1
21 1
1
'
ˆdiv ' 0
k kk kz z x q y q
nk ki q
i
I I I I
A A
32
Inner loop – fixed point iteration l
Solves the nonlinearity of the convex function Ψ
At each pixel we have n linear equations with n unknowns: the increments of the coefficients - dAi
33
Experimental results
The parameters were set experimentally to the following values
34
Synthetic piecewise affine flow example
2 1Brightness constancy equation: ( , , 1) ( , ) ( , ) ( , , )I x u y v t I x u y v I x y I x y t
35
Synthetic piecewise affine flow – ground truth
36
Results
1
( )
( )
, ,1 , ,1cos
, ,1 , ,1
( , )
( , )
AAE Average Angular Error mean
STD std
u v u v
u v u v
u v Estimated flow
u v Ground truth flow
����������������������������
����������������������������
Our method is better in the AAE by 68%
37
Piecewise affine test case
The estimated affine parameters are approximately piecewise constant
38
Ground truthOur method - affine
model
39
Yosemite without clouds sequence
40
41
42
43
44
45
46
The End
47
+35%
+15%+16%
+39%
48
Yosemite without clouds – ground truth
49
Images of the angular error
50
Histogram of the angular error
Our method – pure translation model
Brox et. al.
51
Yosemite - Solution of the affine parameters
1A 2A3A
4A 5A 6A
52
Noise sensitivity results
53
“Variational Joint optic-flow Computation and Video Restoration”
T. Nir, A.M. Bruckstein, R. Kimmel
Errors in the data term appear for two reasons:• Errors in the computed flow.• Errors in the image data – noise, blur, interlacing,
lossy compression, …
The proposed functional
2( , , ) (x , , 1) ( ) xDE u v I I u y v t I x d
0( , , ) ( , , ) ( , ) ( , )D S FE u v I E u v I E u v E I I
2 2( , ) xSE u v u v d
2
0Fidelity term : ( ) xFE I I I d
54
Variational Joint optic-flow Computation and Video Restoration. Cont’.
• Minimization is performed with respect to the optical flow u,v and the image sequence I.
• The fidelity term requires that the minimization would not deviate too far from the measured sequence, thus avoiding trivial solutions.
• If the expected noise is large, a lower choice of λ is appropriate, allowing larger deviations from the measured sequence.
• For , the sequence is constrained to be equal to the measurement, resulting in a regular optic flow scheme.
55
Solution strategy
Optic flowcalculation
Denoising
Iterations between optic flow and denoising.
1. Initialization: zero optic flow and initial sequence.
2. Solve for the optic flow.
3. Perform denoising.
4. Iterate steps 2,3 until convergence.
56
The Denoising step
• For the denoising step we use the discrete approximation with bilinear interpolation:
• Minimize with respect to I1,I2,I3,I4 and I is performed by gradient descent (A,B,C,D are constant – frozen flow).
• The denoising step performs smoothing along the optical flow trajectories.
• Remark: Smoothing by total variation is not good for optic flow calculation.
2 2
0
2 2
1 2 3 4 0
( , , 1) ( , , )I x u y v t I x y t I I dxdydt
AI BI CI DI I I I
57
Office sequence – Frame 7
58
Office sequence – Frame 8
59
Office sequence – Frame 9
60
Office sequence – Frame 10
61
Office sequence – Optic flow at frame 9
62
Experimental results - Office sequence
63
Office sequence results - Cont’.
64
A. Borzi, K. Ito, K. Kunisch: “Optimal control formulation for determining optical flow”, SIAM J.
Sci. Comp. 24(3), 818-847, (2002)
• Minimize with respect to u,v,I
• Subject to the constraints
65
Comparison with Borzi
Our methodBorzi
Symmetric - all the sequence is denoised.
Constrain the first image to equal the measurement.
Non-linear brightness constancy penalty.
Linearized brightness constancy equation as a constraint
Comparison on sequences run by the best results available from the literature.
Comparative results reported on simple synthetic examples.
First to suggest the idea of changing the images together with the flow.
66
What is the actual gap between L1 and L2?
No clouds
AAE
No clouds
STD
Cloudy
AAE
Cloudy
STD
HS~26.14~19.1632.4330.28
HS modified. σ= 1.5
11.2616.41
Multiscale + re-linearization
σ=0.82.382.386.259.14
+Smoothness 3D
1.861.965.909.01
L1 – Brox0.981.171.946.02
67
Summary
• Over-parameterized representation of the optic flow introduces better regularization.
• The per pixel model allows the functional minimization to decide on the locations of discontinuities in the higher dimensional space.
• Significant improvement for both the 2D and 3D cases.• Coupling with our joint optic flow and denoising scheme
gives excellent results under heavy noise.• Future: The improved accuracy of the method has the
potential to improve motion segmentation, video compression, super-resolution…