1

Image Processing:4. Optical Flow4. Optical Flow

Aleix M. Martinezaleix@ece.osuedu

Motion estimation•Optical flow is used to compute the motion of the

pixels of an image sequence. It provides a dense(point to point) pixel correspondance.•Correspondence problem: determine where the

pixels of an image at time t are in the image attime t+1.•Large number of applications.

Two important definitions•Motion field:“the 2-D projection of a 3-Dmotion onto the image plane.”•Optical flow:“the apparent motion of the brightness pattern in an image sequence.”

The method ofThe method ofHorn and SchunckHorn and Schunck

•This is the most fundamental optical flow algorithm.•As you will see, it has several important flaws thatmakes its use inappropriate in a large number ofapplications.•Most of the other algorithms proposed to date arebased on the formulation advanced by Horn andSchunck.

2

•If the brightness is assumed to be constant fromframe to frame, then the motion associated toeach pixel (x,y) of an image I can be modeled as:

This is known as the data conservationconstraint.•The 1st-order Taylor expansion

),,(),,( tttvytuxIzyxI

0 tyx IvIuI

R

tyxD dxdyIvIuIE 2)(

),,(),,( tttvytuxIzyxI

tI

tyI

yxI

xtyxItyxI ),,(),,(

0t 0

tI

yI

dtdy

xI

dtdx

0dtdI

,dtdx

u ,dtdy

v ,xI

I x

,yI

I y

.tI

I t

0 tyx IvIuI

•Derivations of previous result:

•The method is underdetermined.•We can add an additional constraint known

as: spatial coherence

•The solution can be obtained by minimizingthe functional:

R

yxyxS dxdyvvuuE ))()(( 2222

R

S dxdyvuE 2),(

SD EE

Minimization

dxdyvvuuvuF yxyx ),,,,,(

To minimize the above integral, we can use calculus of variations.The Euler equations are:

0

yx uuu F

yF

xF

0

yx vvv F

yF

xF

22222tyxyxyx IvIuIvvuuF

And we want to minimize the expression:

xtyx IIvIuIu 2

ytyx IIvIuIv 2

Note that:

2

2

2

22

yx

(Details in: Horn, “Robot Vision”, MIT Press, 1986.)

The discrete case

We can estimate the derivatives, Ix andIy, by using the following discreetapproximation:

1,1,11,,1,1,1,,141

kjikjikjikjix IIIII

1,1,1,,,1,,,41

kjikjikjikji IIII

3

Example

Example

1

Limitations

•The assumptions embedded in Horn &Schunck formulation are generallyinappropriate.•E.g. both constraints are violated at motion

boundaries–known as motiondiscontinuities.•The aperture problem: to gain accuracy R

needs to be large, however, the larger R is themost probable that our assumptions becomeinvalid.

Robust Statistics:Robust Statistics:Black & AnandanBlack & Anandan

•It is possible to regard motion discontinuities asoutliers.•We can discard outliers only if we can detectthem.•Outliers normally deviate largely from the meanmotion.

Same constraints as in Horn & Schunck

R

tyxD dxdyIvIuIE )(

R

S dxdyvuE )),((

2

21

1log),(

x

x

New estimator, e.g.:

4

Examples

1

Illumination Changes:Illumination Changes:NegahdaripourNegahdaripour

•The methods seen above assume nochanges in the illumination of a scene fromframe to frame.•This is not realistic; even if theillumination source(s) is (are) not moving.•It is possible to modify the constraint usedby the two preceding method.

The illumination problem Illumination Changes

•Illumination changes also violate the assumptionsof ED and ES.

•B&A approach cannot handle large variations inlighting. Its formulation does not take this intoaccount.•We can easily incorporate this information in the

form of a multiplier and an offset:

),,(),,(),,(),,( tyxCtyxItyxMtttvytuxI

5

Gennert & Negahdaripour

R

tttyxD dxdycmIIvIuIE )(

R

S dxdyvuE 2

2),(

R

tM dxdymE2

2

R

tC dxdycE2

2

CCMMSSDD EEEEE

Closed solutionNegahdaripour has proposed the following closed-form solution:

W

t

t

ty

tx

W

y

x

y

x

y

y

y

yx

x

x

yx

I

II

II

II

cmyx

I

I

I

II

II

II

I

II

I

II

I

II

II

Ix

1

2

2

2

Images:

Horn & Schunck:

Ground-truth:

Images:

Black & Anandan:

Ground-truth:

Images:

Gennert & Negahdaripour:

Ground-truth:

More Examples

Images:

6

Horn & Schunck Gennert & Negahdaripour

Images Horn & Schunck

Black & Anandan Gennert & Negahdaripour

7

Error estimationError estimation

v

n

igtestimate

vel n

VVErr

v

ii

1

o

n

i gtestimate

gtestimate

ang n

VV

VV

Err

o

ii

ii

1

arccos

The flower sequence

Gaussian

Flower sequence

Vel. Err. Stdv. Density Angl. Err. Stdv. DensityH&S 1.388 0.662 100% 78.144 65.407 39.96%L&K 1.268 0.782 100.00% 73.061 64.264 62.43%N&Y 0.562 0.313 100% 18.652 21.086 99.32%B&A 0.531 0.386 100% 73.264 57.76 97.82%G&N 0.821 0.283 100% 14.696 16.226 98.97%

The Yosemite sequence

Yosemite sequence

Vel. Err. Stdv. Density Angl. Err. Stdv. DensityH&S 0.935 1.35 100% 45.342 85.14 98.58%L&K 0.746 1.159 99.88% 30.54 73.761 98.40%N&Y 0.636 0.726 99.97% 33.877 76.764 99.01%B&A 0.509 0.725 100% 18.775 57.809 99.13%G&N 1.547 1.786 100% 41.999 77.642 98.59%

Some Examples

8

9