Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I

Motion & Tracking

Motion is the only Cue

G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.

Motion and perceptual organization

• Even “impoverished” motion data can evoke a

strong percept

Motion Estimation & Tracking

Sequence of images contains information about the scene,We want to estimate motion

5

Applications

• video compression

• 3D reconstruction

• segmentation

• object detection

• activity detection

• key frame extraction

• interpolation in time

Tracking

Procerus Technologies

http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies

Video Stabilization

Structure From Motion

Structure From Motion

Motion Field & Optical Flow

Optical center

2D motion field

Projection on the image

plane of the 3D velocity

of the scene

3D motion field

Image intensity

I1

I2

Motion vector - ?

Motion Field & Optical Flow

• Motion field

– The true 2d projection of the 3d motion

• Optical flow

– The measured 2d flow

What we are able to perceive is just an apparent motion, called

Optical flow

(motion, observable only through intensity variations)

Intensity remains constant –

no motion is perceived

No object motion, moving light source

produces intensity variations

Optical Flow ≠ Motion Field

Optical Flow vs. Motion Field

• Optical flow: the measured flow

• Motion field: the real motion

The Barberpole Illusion

Examples of Motion Fields

(a) Motion field of a pilot looking straight ahead while approaching a fixed

point on a landing strip. (b) Pilot is looking to the right in level flight.

(a) (b)

Global Motion

Examples of Motion Fields

(a) (b)

(c) (d)

(a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background.

)1( +tI

What is Optical Flow?

Optical Flow1p

2p

3p

4p

}{),( iptI

1vr

2vr

3vr

4vr

}{ ivr

Velocity vectors

Source: Stanford CS223B Computer Vision, Winter 2006, lecture notes

Optical Flow Based Search

…

From Darya Frolova and Denis Simakov

Optical flow-methods

Correlation-based techniques - compare

parts of the first image with parts of the

second in terms of the similarity in brightness

patterns in order to determine the motion

vectors

Feature-based methods - compute and

analyze Optical Flow at small number of

well-defined image features

Gradient-based methods - use

spatiotemporal partial derivatives to

estimate flow at each point

Optical Flow - Assumptions

• Constant brightness assumption

The apparent brightness of moving objects remains

constant

• Small motion

Points do not move very far

• Spatial coherence

– Neighboring points in the scene typically belong to the

same surface and hence typically have similar motions

• Temporal coherence

– The image motion of a surface patch changes

gradually over time

Estimating Optical Flow

• Constant brightness assumption

( )tyxI ,, ( )dttdyydxxI +++ ,,=

Time = t Time = t+dt

* From Lihi Zelnik-Manor

(x,y) (x+dx,y+dy)

( ) tdt

dIv

dy

dIu

dx

dItyxIttvyuxI ∆+++=∆+++ ),,(,,

)tt,vy,ux(I)t,y,x(I ∆+++=

vIuII0 vxt ++=

Take the Taylor series expansion of I :

using brightness assumption:

Constant Brightness Assumption - 2D Case:

Optical Flow Equation

vIuII0 vxt ++=

x

( )t,xI ( )tt,xI ∆+

u

tI

Optical Flow Equation- Intuition

uII xt =−

The change in value It at a pixel P is dependent on:

The distance moved (u).

x

II x ∆

∆=

x∆I∆

tyx vu III −=+


tIvuI −=⋅∇ ],[

Only the component of the flow in the gradient direction can be determined

The component of the flow parallel to an edge is unknown


Intuition - The Barber Pole Illusion.

true motion

vector

perp.

component

parallel.

component

The Aperture Problem

• Different motions – classified as similar

source: Ran Eshel

The Aperture Problem

• Similar motions – classified as different

source: Ran Eshel

Frame 1

flow (1): true motionflow (2)

From Darya Frolova and Denis Simakov

Ambiguity of Optical Flow

tyx vu III −=+


Shoot! One equation, two velocity unknowns (u,v) ...

Solving for u,v:

Spatial coherence

• Impose additional constraints

– Assume the pixel’s neighbors have the same (u,v)

( ) ( )( ) ( )

( ) ( )

( )( )

( )

−=

−=+

Nt

t

t

NyNx

yx

yx

tyx

v

u

vu

p

p

p

pp

pp

pp

2

1

22

11

I

I

I

II

II

II

III

MMM

A

N×2

x

2×1b

N×1

p1

pN

bAx =

( ) bAAAxtt 1−

=

bAAxAtt =

p2

Lukas Kanade Scheme

Equivalent to Solving least squares:

ATA ATb

bAx)AA( TT =

• The summations are over all pixels in the K x K window

• This technique was first proposed by Lukas & Kanade (1981)

x

−=

∑∑

∑∑∑∑

ty

tx

yyx

yxx

v

u

II

II

III

III2

2

Lucas-Kanade Equation

I R IxIy It

22×

AAt

12×

x

12×

bAt

When can we solve LK Eq ?Optimal (u, v) satisfies Lucas-Kanade equation

• ATA should be invertible

• The eigenvalues of ATA should not be too small (noise)

• ATA should be well-conditioned:

λ1/ λ2 should not be too large (λ1 = larger eigenvalue)

ATA is invertible when there is no aperture problem

Hessian Matrix

Ix = 0

Iy = 0

M =

M =0 0

0 0Non Invertable

Ix = 0

Iy = kM =

0 0

0 k2Non Invertable

Ix = k

Iy = 0M =

k2 0

0 0Non Invertable

Ix = k1

Iy = k2RM =

k2 0

0 0Non Invertable

k1, k2 correlated(R = rotation)

Ix = k1

Iy = k2M =

k12 0

0 k22

Invertable

k1 * k2 = 0

Classification of AtA

50 100 150 200

20

40

60

80

100

120

140

160

180

200

220

2 4 6 8 10 12

2

4

6

8

10

12

large λ1, small λ2

2 4 6 8 10 12

2

4

6

8

10

12

large λ1, large λ2

2 4 6 8 10 12

2

4

6

8

10

12

small λ1, small λ2

=

∑∑∑∑

2

2

yyx

yxxt

III

IIIΑA

The Hessian Matrix

Optical Flow: Iterative Estimation

Optical flow is first order approximation (first order Taylor).

What if first order is not good approx of the area around a pixel?

( ) tdt

dIv

dy

dIu

dx

dItyxIttvyuxI ∆+++=∆+++ ),,(,,


xx0

Estimate:

u0= 0

u1= u0+ û

estimate

update

û

f2(x)f1(x)

Initial guess:

xx0

u1

u2= u1+ û

f1(x-u1) f2(x)

Estimate:

Initial guess: estimate

update

û


xx0

f2(x)f1(x-u2)

u2

u3= u2+ ûEstimate:

Initial guess: estimate

update

û


xx0

f1(x-u3) ≈ f2(x)


( ) ( )∑∈

++=Ayx

tyxvuvu

vuvuSSD,

2

,,min,min III

0=++ tyx vu III

Iterative Refinement

Iterative Lukas-Kanade Algorithm

1. Estimate velocity at each pixel by solving Lucas-Kanade

equations:

2. Warp I1 towards I2 using the estimated flow field

3. Repeat until convergence

Find


• Some Implementation Issues:

– Warp one image, take derivatives of the other

so you don’t need to re-compute the gradient

after each iteration.

– Often useful to low-pass filter the images

before motion estimation (for better derivative

estimation, and linear approximations to

image intensity)

Optical Flow

http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/


Optical Flow

Revisiting the small motion assumption

• Is this motion small enough?

– Probably not—it’s much larger than one pixel (2nd order terms dominate)

– How might we solve this problem?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Aliasing in large motions

Temporal aliasing causes ambiguities in optical flow because

images can have many pixels with the same intensity.

I.e., how do we know which ‘correspondence’ is correct?

nearest match is

correct (no aliasing)

nearest match is

incorrect (aliasing)

To overcome aliasing: coarse-to-fine estimation.

actual shift

estimated shift

Reduce the resolution!


image It-1 image I

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fine optical flow

estimation

image Iimage J

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1

Coarse-to-fine optical flow

estimation

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

Optical Flow Results


Optical Flow Results


Optical Flow for Affine Transformation

+

=

f

c

y

x

ed

ba

v

u

tyx feydxcbyax III −=++⋅+++⋅ )()(

( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

( )

( )

−=

Nt

t

NyNyNyNxNxNx

yyyxxx

f

e

d

c

b

a

yxyx

yxyx

p

p

pppppp

pppppp 1111111

I

I

IIIIII

IIIIII

MM

Optical Flow for Affine Transformation

1616

66

222

222

22222

22222

22222

22222

×××

−=

∑∑∑∑∑∑

∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑

bAx

AA

II

II

II

II

II

II

IIIIIIIII

IIIIIIIII

IIIIIIIII

IIIIIIIII

IIIIIIIII

IIIIIIIII

tt

ty

tx

ty

tx

ty

tx

yyyyxyxyx

yxyxyxxxx

yyyyxyxyx

yxyxyxxxx

yyyyxyxyx

yxyxyxxxx

x

y

y

x

f

e

d

c

b

a

yxyx

yxyx

xxyxxxyx

yyxyyyxy

yyxyyyxy

xxyxxxyx

Errors in Lukas-Kanade

Suppose ATA is easily invertible

Suppose there is minimal noise in the image.

When our assumptions are violated:

– Brightness constancy is not satisfied

– The motion is not small

– A point does not move like its neighbors

• window size is too large

• what is the ideal window size?

What are the potential causes of errors in the LK approach?

Horn and Schunck (1981)

• Two criteria:

– Small error in optical flow constraint equation, Fh(u,v)

– Optical flow is smooth, Fs(u,v)

• Minimize a combined error functional

Fc(u,v) = Fh(u,v) + λ Fs(u,v)

λ is a weighting parameter

Horn & Schunck Optical Flow - Extension

Horn & Schunck Optical Flow - Extension

Fc(u,v) = Fh(u,v) + λ Fs(u,v)

Solve by means of calculus of variation (iteratively).

( ) ( ) smoothness from departure :dydxvvuue2

y2

x

D

2y

2xs +++= ∫∫

( ) equation flow opticalin error :2

∫∫ ++=D

tyxc dydxIvIuIe

( ) ( ) min2

2

2

2

2 →∇+∇++⋅∇∫∫ dydxvuII t λu

Fs(u,v) =

Fh(u,v) =

Smoothness term

(regularization term)Data term

Horn and Schunck

• Assumptions

– brightness constancy

– neighboring velocities are nearly identical

• Properties

+ incorporates global information

+ image first derivatives only

- iterative

- smoothes across motion boundaries

1 iteration 4 iteration

16 iteration 64 iteration

Horn & Schunck - Example

1 frame of sequence

Discontinuities

But smoothing term does not allow to save discontinuities

Discontinuities near edges are lost

Synthetic example

(method of Horn and Schunck)

Use edge preserving approaches

Reconsidering Smoothness Term

( ) ( ) min2

2

2

2

2 →∇+∇++⋅∇∫∫ dydxvuII t λu

Smoothness term

(regularization term)Data term

Use different

regularization in

smoothness term

Homogeneous propagation

∫ ∇+∇video

vu22

min- flow in the x direction

- flow in the y direction

- gradient∇

),,( tyxu),,( tyxv

[Horn&Schunck 1981]

This constraint is not correct on motion boundaries => over-smoothing of the resulting flow

Robustness to flow discontinuities

(also known as isotropic flow-driven regularization)

∫ ∇+∇video

vu )||||(min 22φ

:φ

[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]High accuracy optical flow estimation based on a theory for warping

22 ε+x

ε

64

Brightness is not always constant

Rotating

cylinder

Brightness constancy

does not always hold

),,()1,,( tyxItvyuxI ≠+++

inte

nsity

position

),,()1,,( tyxItvyuxI ∇=+++∇

Gradient constancy holds

inte

nsity d

erivative

position

http://cbia.fi.muni.cz/user_dirs/xulman/gtgen/gt_application.png

Horn & Schunck

https://www.youtube.com/watch?v=Ox8oI7nzSPw&nohtml5=False

Navigation

http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies

Navigation

Car Tracking Using Optical Flow

Automatic Plane landing

Using Horizon and Optical Flow

https://www.youtube.com/watch?v=U9iy1B5QG-0

Tracking

Some slides are from Sebastian Thrun, Rick Szeliski, Hendrik Dahlkamp, Wolfram

Burgard, Kim Chule Hwon, D. Forsyth, M. Isard, T. Darrell Also From Yuri Rapaport

Examples

Tracking applications

• Tracking missiles

• Tracking heads/hands/drumsticks

• Extracting lip motion from video

• Lots of computer vision applications

• Economics

• Navigation

SIGGRAPH 2001

Detect Foreground objects –

Background subtraction

Compare the current frame to a reference image and label all the different pixels as motion.

Issues:

Detect Foreground objects –

Background subtraction

Examples

Tracking: First Idea!

estimateinitial position

x

y

x

y

state dynamics

x

y

measurement

x

y

Starting point Possible new positions Combine Measurement

and possibilities to obtain

estimate.

Tracking

• Bayes Filtering

• Kalman Filtering

• Particle Filtering

On-line posterior density estimation algorithms that

estimate the posterior density of the state-space by

directly implementing the Bayesian recursion equations.

Particle Filters Condensation Algorithm

Sequential Monte Carlo (SMC)

HUH???

Partially Observable Chains

z2 z3 z4Measurements z1

state x4state x3state x2state x1

)( 0xp

( )tt zzxp ,,1 LWe have to estimate:

Assumptions

• The system follows a Markovian process

• The observations are independent

( ) ( )111 ,, −− = tttt xxpxxxp L

( ) ( )tttt xzpxxzp =,,1 L

System Dynamics

t-1 t t+1 time

zt-1 zt zt+1

xt-1 xt xt+1

Measurements(observed)

States(to be estimated)( )1−tt xxp

( )tt xzp

82

Bayes Filters

System state dynamics (uncertain)

Observation dynamics (measurement error)

We are interested in: posterior probability

Estimating system state from noisy observations

( )tt zzxp ,,1 L

( )1−tt xxp

( )tt xzp

Recursive Bayes Filters

• Sequential update of previous estimate

• Allow on-line processing of data

• Rapid adaptation to changing signals

characteristics

• Consists of two steps:

( ) ( )1:11:11 −−− → tttt zxpzxp

( ) ( )ttttt zxpzzxp :11:1 , →−

– Prediction step:

– Update step:

84

Bayes Formula

evidence

prior likelihood

)(

)()|()(

)()|()()|(),(

⋅==

⇒

==

yP

xPxyPyxP

xPxyPyPyxPyxP

85

1:( 1) 1:( 1)( | , ) ( | ) ( | )t t t t t t t tp x z z p z x p x zα− −=

1:( 1) 1 1 1:( 1) 1( | ) ( | ) ( | )t t t t t t tp x z p x x p x z dx− − − − −= ∫

1( | )

( | )

t t

t t

p x x

p z x

−Motion Model

Observation Model

Start from: 0 00 0 0

0

( | )( | ) ( )

( )

p z xp x z p x

p z=

Predict:

Update:

Recursive Bayes Filters

Bayes Filters

Prior P(x)

Measurement

evidence

P(z|x)

Posterior

P(x|z)

)()|()|( xpxzpzxp ∝

dxxpxxpxp )()|'()'( ∫=

Update

Predict

Predict the new prior p(x’)

On-line posterior density estimation algorithms that

estimate the posterior density of the state-space by

directly implementing the Bayesian recursion equations.

Particle Filters Condensation Algorithm

Sequential Monte Carlo (SMC)

Uses Particles (many of them) to represent the distributions.

Want to estimate x position of plane flying across a Fiord

Demo – Particle Filtering

Particle Filter Explained without Equations

Andreas Svensson, Uppsala Univ.


Want to estimate x position of plane flying across a Fiord


Andreas Svensson, Uppsala Univ.


Zt = elevationt - heightt∈ [0; 1].

Xt ∈ [0; 5000].

Xt can be estimated from Xt-1.

Zt can be calculated from Xt .

Velocity distributes uniformly from 800 to 1000 km/h.

Assume have map of elevation.

Zt

Xt



Andreas Svensson, Uppsala Univ.https://www.youtube.com/watch?v=aUkBa1zMKv4

Particle Filtering

Occlusion

Challenges

Resolving Occlusion

• Types of occlusions:

– Self occlusion

– Inter-object occlusion

– Occlusion by the background scene structure

“Detection and Tracking of Occluded People”

Siyu Tang, Mykhaylo Andriluka, Bernt Schiele

Use double person detector

“Detection and Tracking of Occluded People”

Siyu Tang, Mykhaylo Andriluka, Bernt Schiele

Shadows

Multiple Cameras

• Challenge : How to integrate all the detections from the different cameras

source: Ran Eshel

Documents

Motion & Tracking - University of Haifacs.haifa.ac.il/hagit/courses/CV/Lectures/CV05_Motion.pdf · * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003. image I t-1 image I