lec15 - motion field optical flow · 2019. 5. 22. · 1 CSE152, Spring 2019 Intro Computer Vision Motion Field and Optical Flow Introduction to Computer Vision CSE 152 Lecture 15

1

CSE152, Spring 2019 Intro Computer Vision

Motion Field and Optical Flow

Introduction to Computer Vision CSE 152

Lecture 15


•  Reading on optical flow from Trucco & Verri is on ereserves

•  HW3 assigned •  Midterm graded

2


Midterm •  Maximum78 (93%) •  Median 56.2 (67%) •  Minimum 29.6 (35%) •  Score Distribution

•  Recap of answers.


Lambertian Photometric Stereo Videos this time

Reconsstruction with Albedo Map

3


Constant albedo map


Another person

4


Constant Albedo map


Structure-from-Motion (SFM) Given two or more images or video w/o any

information on camera position/motion as input, estimate camera motion and 3-D structure of a scene.

Two Approaches

1.  Discrete motion (wide baseline) 2.  Continuous (Infinitesimal) motion usually

from video

5


Motion field •  The motion field is the projection of the

3D scene motion into the image


Rigid Motion: General Case

Position and orientation of a rigid body Rotation Matrix: R Translation vector: t

Rigid Motion: Velocity vector: T

Angular velocity vector: ω

P

pTp ×+= ω!

p!ω

||ω||

I know, using T for velocity is poor notation, but it’s consistent with Trucco and Verri textbook

6


General Motion

Let (x,y,z) be functions of time (x(t), y(t), z(t)):

Substitute where p=(x,y,z)T


Motion Field Equation

•  Image •  (u,v): Image point coordinates •  : Image point velocity

•  Camera •  T: Components of 3-D linear motion •  ω: Angular velocity vector •  f: focal length

•  Scene •  z: depth

!u =Tzu−Tx f

z−ω y f +ω zv +

ωxuvf

−ω yu

2

f

!v =Tzv −Ty f

z+ωx f −ω zu−

ω yuv

f−ωxv

2

f

( !u, !v)

7


fv

fuv

ufZ

fTvTv

fu

fuvvf

ZfTuTu

xyzx

yz

yxzy

xz

2

2

ωωωω

ωωωω

−−−+−

=

−++−−

=

!

!

Pure Translation

ω = 0

•  is inversely proportional to Z (remember Euclid) •  Focus of expansion is located at (u,v) where !u, !v

!u = !v = 0

If camera is just translating with velocity (Tx, Ty, Tz), there’s no rotation:


Forward Translation & Focus of Expansion [Gibson, 1950]

8


Focus of Expansion (FOE)

•  Focus of expansion is located at (u,v) where

•  Solve for u,v

Tzu−Tx f = 0Tzv −Ty f = 0

!u = !v = 0

!u =Tzu−Tx fZ

!v =Tzv −Ty f

Z

u = fTxT z

v = fTyT z

Insight: The FOE is the perspective projection of the linear velocity vector (Tx, Ty, Tz).


Pure Translation

Radial about FOE

Parallel ( FOE point at infinity)

TZ = 0 Motion parallel to image plane

9


Sideways Translation [Gibson, 1950]

Parallel ( FOE point at infinity)

TZ = 0 Motion parallel to image plane


Pure Rotation: T=0

•  Independent of Tx Ty Tz •  Independent of Z •  Only function of image plane position (u,v), f and ω

fv

fuv

ufZ

fTvTv

fu

fuvvf

ZfTuTu

xyzx

yz

yxzy

xz

2

2

ωωωω

ωωωω

−−−+−

=

−++−−

=

!

!

10


Rotational MOTION FIELD The “instantaneous” velocity of points in an image

PURE ROTATION

ω = (0,0,1)T


Motion Field Equation: Depth Estimation

If T, ω, and f are known or measured, then for each image point (u,v), one can solve for the depth Z given measured motion at (u,v).

€

˙ u =Tzu −Tx f

Z−ω y f +ω zv +

ω xuvf

−ω yu

2

f

˙ v =Tzv −Ty f

Z+ω x f −ω zu −

ω yuvf

−ω xv

2

f

€

Z =Tzu −Tx f

˙ u +ω y f −ω zv −ω xuv

f+ω yu

2

f

!u, !v

Inversely proportional to image velocity .

11


Measuring Motion


Estimating the motion field from images

1.  Feature-based (Sect. 8.4.2 of Trucco & Verri)

1.  Detect Features (corners) in an image 2.  Search for the same features nearby (Feature

tracking).

2.  Differential techniques (Sect. 8.4.1)

12


Optical Flow •  Optical flow is the pattern of apparent

motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene

•  Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image. i.e., the motion field.

•  As we’ll see, it’s not but more on that later.


Optical Flow

•  How to estimate pixel motion from image H to image I? –  Find pixel correspondences

•  Given a pixel in H, look for nearby pixels of the same color in I

•  Key assumptions –  color constancy: a point in H looks “the same” in image I

•  For grayscale images, this is brightness constancy –  small motion: points do not move very far

13


Warning •  Notation shift: •  (x,y) will be image positions •  (u,v) will be image velocities

•  Why? Notation in Trucco & Verri


Optical Flow Constraint Equation

–  Assume brightness of patch remains same in both images:

–  Assume small motion: (Taylor series expansion of LHS about (x,y,t) and retain first order terms)

),( yx

(x +u δt, y + v δt)

ttime tttime δ+),( yx

Optical Flow: Velocities ),( vuDisplacement:

€

I(x + u δt,y + v δt, t +δt) = I(x,y,t)

€

I(x,y,t) +δx ∂I∂x

+δy∂I∂y

+δt∂I∂t

= I(x,y,t)

(δx,δ y) = (u δt,v δt)

14


Optical Flow Constraint Equation

–  Assume small interval, this becomes:

),( yx

),( tvytux δδ ++

ttime tttime δ+),( yx

Optical Flow: Velocities ),( vuDisplacement: (δx,δ y) = (u δt,v δt)

€

δxδt

∂I∂x

+δyδt

∂I∂y

+∂I∂t

= 0

–  Subtracting I(x,y,t) from both sides and dividing by

€

δt

€

dxdt

∂I∂x

+dydt

∂I∂y

+∂I∂t

= 0


Solving for flow

•  We can measure

§  Convolve image with [-1, 0, 1]

§  Convolve image with [-1, 0, 1]T

§  Consider stacking 3 images at (t-1, t, t+1). Convolve with [-1,0,1] over time

•  We want to solve for •  One equation, two unknowns à Can’t solve it

Optical flow constraint equation :

dtdy

dtdx,

∂I∂x,∂I∂y,∂I∂t

∂I∂t

∂I∂y

∂I∂x€

dxdt

∂I∂x

+dydt

∂I∂y

+∂I∂t

= 0

15


Ix =∂I∂x

,

, ∂∂

=yII y

tIIt ∂∂

=

Measurements

u = dxdt

,

dtdyv =

Flow vector

The component of the optical flow in the direction of the image gradient.


Barber Pole Illusion

http://www.opticalillusion.net/optical-illusions/the-barber-pole-illusion/

Optical Flow

Motion Field

Optical flow field isn’t always the same as the motion field

16


Optical Flow Motion Field ≠

Motion field exists but no optical flow No motion field but shading changes


Apparently an aperture problem

17


Assume a single velocity (u,v) for pixels within an image patch Ω

dE(u,v)du

= 2Ix Ixu+ I yv + It( ) = 0∑dE(u,v)dv

= 2I y Ixu+ I yv + It( ) = 0∑

E(u,v) is minimized when partial derivatives equal zero.

Rewrite as


Lukas-Kanade (cont.)

•  So, the optical flow U = (u,v) can be written as

MU=b •  And optical is just U=M-1b

Documents

lec15 - motion field optical flow · 2019. 5. 22. · 1 CSE152, Spring 2019 Intro Computer Vision Motion Field and Optical Flow Introduction to Computer Vision CSE 152 Lecture 15