1
CSE152, Spring 2019 Intro Computer Vision
Motion Field and Optical Flow
Introduction to Computer Vision CSE 152
Lecture 15
CSE152, Spring 2019 Intro Computer Vision
• Reading on optical flow from Trucco & Verri is on ereserves
• HW3 assigned • Midterm graded
2
CSE152, Spring 2019 Intro Computer Vision
Midterm • Maximum78 (93%) • Median 56.2 (67%) • Minimum 29.6 (35%) • Score Distribution
• Recap of answers.
CSE152, Spring 2019 Intro Computer Vision
Lambertian Photometric Stereo Videos this time
Reconsstruction with Albedo Map
3
CSE152, Spring 2019 Intro Computer Vision
Constant albedo map
CSE152, Spring 2019 Intro Computer Vision
Another person
4
CSE152, Spring 2019 Intro Computer Vision
Constant Albedo map
CSE152, Spring 2019 Intro Computer Vision
Structure-from-Motion (SFM) Given two or more images or video w/o any
information on camera position/motion as input, estimate camera motion and 3-D structure of a scene.
Two Approaches
1. Discrete motion (wide baseline) 2. Continuous (Infinitesimal) motion usually
from video
5
CSE152, Spring 2019 Intro Computer Vision
Motion field • The motion field is the projection of the
3D scene motion into the image
CSE152, Spring 2019 Intro Computer Vision
Rigid Motion: General Case
Position and orientation of a rigid body Rotation Matrix: R Translation vector: t
Rigid Motion: Velocity vector: T
Angular velocity vector: ω
P
pTp ×+= ω!
p!ω
||ω||
I know, using T for velocity is poor notation, but it’s consistent with Trucco and Verri textbook
6
CSE152, Spring 2019 Intro Computer Vision
General Motion
Let (x,y,z) be functions of time (x(t), y(t), z(t)):
Substitute where p=(x,y,z)T
CSE152, Spring 2019 Intro Computer Vision
Motion Field Equation
• Image • (u,v): Image point coordinates • : Image point velocity
• Camera • T: Components of 3-D linear motion • ω: Angular velocity vector • f: focal length
• Scene • z: depth
!u =Tzu−Tx f
z−ω y f +ω zv +
ωxuvf
−ω yu
2
f
!v =Tzv −Ty f
z+ωx f −ω zu−
ω yuv
f−ωxv
2
f
( !u, !v)
7
CSE152, Spring 2019 Intro Computer Vision
fv
fuv
ufZ
fTvTv
fu
fuvvf
ZfTuTu
xyzx
yz
yxzy
xz
2
2
ωωωω
ωωωω
−−−+−
=
−++−−
=
!
!
Pure Translation
ω = 0
• is inversely proportional to Z (remember Euclid) • Focus of expansion is located at (u,v) where !u, !v
!u = !v = 0
If camera is just translating with velocity (Tx, Ty, Tz), there’s no rotation:
CSE152, Spring 2019 Intro Computer Vision
Forward Translation & Focus of Expansion [Gibson, 1950]
8
CSE152, Spring 2019 Intro Computer Vision
Focus of Expansion (FOE)
• Focus of expansion is located at (u,v) where
• Solve for u,v
Tzu−Tx f = 0Tzv −Ty f = 0
!u = !v = 0
!u =Tzu−Tx fZ
!v =Tzv −Ty f
Z
u = fTxT z
v = fTyT z
Insight: The FOE is the perspective projection of the linear velocity vector (Tx, Ty, Tz).
CSE152, Spring 2019 Intro Computer Vision
Pure Translation
Radial about FOE
Parallel ( FOE point at infinity)
TZ = 0 Motion parallel to image plane
9
CSE152, Spring 2019 Intro Computer Vision
Sideways Translation [Gibson, 1950]
Parallel ( FOE point at infinity)
TZ = 0 Motion parallel to image plane
CSE152, Spring 2019 Intro Computer Vision
Pure Rotation: T=0
• Independent of Tx Ty Tz • Independent of Z • Only function of image plane position (u,v), f and ω
fv
fuv
ufZ
fTvTv
fu
fuvvf
ZfTuTu
xyzx
yz
yxzy
xz
2
2
ωωωω
ωωωω
−−−+−
=
−++−−
=
!
!
10
CSE152, Spring 2019 Intro Computer Vision
Rotational MOTION FIELD The “instantaneous” velocity of points in an image
PURE ROTATION
ω = (0,0,1)T
CSE152, Spring 2019 Intro Computer Vision
Motion Field Equation: Depth Estimation
If T, ω, and f are known or measured, then for each image point (u,v), one can solve for the depth Z given measured motion at (u,v).
€
˙ u =Tzu −Tx f
Z−ω y f +ω zv +
ω xuvf
−ω yu
2
f
˙ v =Tzv −Ty f
Z+ω x f −ω zu −
ω yuvf
−ω xv
2
f
€
Z =Tzu −Tx f
˙ u +ω y f −ω zv −ω xuv
f+ω yu
2
f
!u, !v
Inversely proportional to image velocity .
11
CSE152, Spring 2019 Intro Computer Vision
Measuring Motion
CSE152, Spring 2019 Intro Computer Vision
Estimating the motion field from images
1. Feature-based (Sect. 8.4.2 of Trucco & Verri)
1. Detect Features (corners) in an image 2. Search for the same features nearby (Feature
tracking).
2. Differential techniques (Sect. 8.4.1)
12
CSE152, Spring 2019 Intro Computer Vision
Optical Flow • Optical flow is the pattern of apparent
motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene
• Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image. i.e., the motion field.
• As we’ll see, it’s not but more on that later.
CSE152, Spring 2019 Intro Computer Vision
Optical Flow
• How to estimate pixel motion from image H to image I? – Find pixel correspondences
• Given a pixel in H, look for nearby pixels of the same color in I
• Key assumptions – color constancy: a point in H looks “the same” in image I
• For grayscale images, this is brightness constancy – small motion: points do not move very far
13
CSE152, Spring 2019 Intro Computer Vision
Warning • Notation shift: • (x,y) will be image positions • (u,v) will be image velocities
• Why? Notation in Trucco & Verri
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Constraint Equation
– Assume brightness of patch remains same in both images:
– Assume small motion: (Taylor series expansion of LHS about (x,y,t) and retain first order terms)
),( yx
(x +u δt, y + v δt)
ttime tttime δ+),( yx
Optical Flow: Velocities ),( vuDisplacement:
€
I(x + u δt,y + v δt, t +δt) = I(x,y,t)
€
I(x,y,t) +δx ∂I∂x
+δy∂I∂y
+δt∂I∂t
= I(x,y,t)
(δx,δ y) = (u δt,v δt)
14
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Constraint Equation
– Assume small interval, this becomes:
),( yx
),( tvytux δδ ++
ttime tttime δ+),( yx
Optical Flow: Velocities ),( vuDisplacement: (δx,δ y) = (u δt,v δt)
€
δxδt
∂I∂x
+δyδt
∂I∂y
+∂I∂t
= 0
– Subtracting I(x,y,t) from both sides and dividing by
€
δt
€
dxdt
∂I∂x
+dydt
∂I∂y
+∂I∂t
= 0
CSE152, Spring 2019 Intro Computer Vision
Solving for flow
• We can measure
§ Convolve image with [-1, 0, 1]
§ Convolve image with [-1, 0, 1]T
§ Consider stacking 3 images at (t-1, t, t+1). Convolve with [-1,0,1] over time
• We want to solve for • One equation, two unknowns à Can’t solve it
Optical flow constraint equation :
dtdy
dtdx,
∂I∂x,∂I∂y,∂I∂t
∂I∂t
∂I∂y
∂I∂x€
dxdt
∂I∂x
+dydt
∂I∂y
+∂I∂t
= 0
15
CSE152, Spring 2019 Intro Computer Vision
Ix =∂I∂x
,
, ∂∂
=yII y
tIIt ∂∂
=
Measurements
u = dxdt
,
dtdyv =
Flow vector
The component of the optical flow in the direction of the image gradient.
CSE152, Spring 2019 Intro Computer Vision
Barber Pole Illusion
http://www.opticalillusion.net/optical-illusions/the-barber-pole-illusion/
Optical Flow
Motion Field
Optical flow field isn’t always the same as the motion field
16
CSE152, Spring 2019 Intro Computer Vision
Optical Flow Motion Field ≠
Motion field exists but no optical flow No motion field but shading changes
CSE152, Spring 2019 Intro Computer Vision
Apparently an aperture problem
17
CSE152, Spring 2019 Intro Computer Vision
Assume a single velocity (u,v) for pixels within an image patch Ω
dE(u,v)du
= 2Ix Ixu+ I yv + It( ) = 0∑dE(u,v)dv
= 2I y Ixu+ I yv + It( ) = 0∑
E(u,v) is minimized when partial derivatives equal zero.
Rewrite as
CSE152, Spring 2019 Intro Computer Vision
Lukas-Kanade (cont.)
• So, the optical flow U = (u,v) can be written as
MU=b • And optical is just U=M-1b